topic Re: If Dataflows Can Do Everything a Copy Activity Can (and More), Why Use Copy Activity? in Pipelines

If Dataflows Can Do Everything a Copy Activity Can (and More), Why Use Copy Activity?

HamidBee — Fri, 01 Nov 2024 10:59:39 GMT

Hi everyone,

I’m working with data ingestion and transformation in Microsoft Fabric and noticed that Dataflows seem to offer a lot more flexibility in terms of connecting to various data sources and performing transformations. Given that Dataflows can connect to a broader range of sources and seem capable of more complex transformations, is there any advantage to using Copy Activity instead?

Is Copy Activity preferable in certain scenarios, such as faster data transfer performance, or are there specific use cases where it might be better suited than Dataflows? I’d love to understand why and when to choose one over the other.

Thanks for your insights.

Re: If Dataflows Can Do Everything a Copy Activity Can (and More), Why Use Copy Activity?

mattiasdesmet — Fri, 01 Nov 2024 12:12:36 GMT

Copy Activity is recommended for ingestion of data as it is much more performant than Dataflows. If you don't need to do any manipulations, Copy Activity is recommended. Once the data is in your data platform and you need to transform it, Dataflows, Notebooks etc are the options to work with data at scale.

Re: If Dataflows Can Do Everything a Copy Activity Can (and More), Why Use Copy Activity?

mattiasdesmet — Fri, 01 Nov 2024 12:23:15 GMT

Fabric decision guide - copy activity, dataflow, or Spark - Microsoft Fabric | Microsoft Learn

Re: If Dataflows Can Do Everything a Copy Activity Can (and More), Why Use Copy Activity?

OktayPamuk80 — Fri, 01 Nov 2024 20:21:14 GMT

The answer is always, it depends. With millions of rows you want to have a performant and compute unit efficient process. Dataflows does support fast copy for some sources, but it will require more compute units by comparison to copy activity or notebooks.

Did I answer your question? Then please mark my post as the solution.

If I helped you, click on the Thumbs Up to give Kudos.

Re: If Dataflows Can Do Everything a Copy Activity Can (and More), Why Use Copy Activity?

NandanHegde — Sat, 02 Nov 2024 06:27:36 GMT

Copy activity needs to be used in case if one needs to transfer data as is between source and sink with no transformations and is much more quicker than dataflow.

Also copy activity supports multiple sink types whereas dataflow is restricted to only specific sinks

Re: If Dataflows Can Do Everything a Copy Activity Can (and More), Why Use Copy Activity?

HamidBee — Mon, 04 Nov 2024 15:34:59 GMT

Thanks for sharing this information.