topic Fast Copy DataFlowGen2 in Dataflow

Fast Copy DataFlowGen2

Anonymous — Tue, 29 Oct 2024 13:51:27 GMT

Hi Team,

Im trying to move files from azure data lake gen2 to lakehouse in ms fabric through dataflowgen2 using fast copy.But every time i do it im getting following error.

is this an existing issue ? if not can you help me with resolving this issue?

Re: Fast Copy DataFlowGen2

Expiscornovus — Tue, 29 Oct 2024 14:40:59 GMT

Hi Anonymous,

The known limitations sections of the fast copy documentation does mention the below:

Writing data into an existing table in Lakehouse isn't supported.

https://learn.microsoft.com/en-us/fabric/data-factory/dataflows-gen2-fast-copy#known-limitations

Re: Fast Copy DataFlowGen2

Anonymous — Tue, 29 Oct 2024 15:44:08 GMT

Expiscornovus

we are not trying to write the data into existing table. I have gone throught the document.

We are trying to Move the files from azuredatalake gen 2 to fabric lakehouse . what will be the best appraoch using fast copy dataglow gen 2 feature.

Regards,

Shiva

Re: Fast Copy DataFlowGen2

SofBL — Fri, 31 Oct 2025 10:06:47 GMT

Same kind of issue here,
Using the Azure SQL DB connector I use a native sql query to load data into a new Lakehouse table. and then it states, We encountered an error while trying to validate the query for Fast Copy. I proceeded, publish and the refresh failed stating
Fast Copy is not supported with the current set of transformations on the query

but every requirement seems to be approved, the connector is supported
Lakehouse -> new table (no fixed schema), Replace data, Dynamic Schema
no transformations outside the native query.
there is just a link back to the https://learn.microsoft.com/en-us/fabric/data-factory/dataflows-gen2-fast-copy site but hey after reading that several times I am confused. This definitly requires a more detailed error message to help determine what goes wrong with the set up of fast copy.

Re: Fast Copy DataFlowGen2

srinidhiprabhu — Fri, 31 Oct 2025 17:39:45 GMT

SofBL What options does your M query to connect to Sql.Database contain? For Fast copy, we only allow the CommandTimeout and Query options. Options like ConnectionTimeout and EnableCrossDatabaseFolding are not supported. All options are documented here, out of which only the Query and CommandTimeout options are currently supported

Re: Fast Copy DataFlowGen2

srinidhiprabhu — Fri, 31 Oct 2025 17:32:05 GMT

Anonymous Are you still running into this problem?

Re: Fast Copy DataFlowGen2

SofBL — Fri, 31 Oct 2025 20:56:38 GMT

settings above are default and result in this M query

let
Source = Sql.Database("Redacted", "BC_Lakehouse", [Query = "SELECT [Document_No] #(lf)FROM [REDACTED]", CreateNavigationProperties = false])
in
Source

I removed the , CreateNavigationProperties = false option to bring this back to
let
Source = Sql.Database("Redacted", "BC_Lakehouse", [Query = "SELECT [Document_No] #(lf)FROM [REDACTED]"])
in
Source

but it makes no difference.

Re: Fast Copy DataFlowGen2

srinidhiprabhu — Fri, 31 Oct 2025 21:34:19 GMT

The CreateNavigationProperties option is why you would be failing to use Fast copy previously. Have you tried saving your dataflow after removing the option to check if you still hit the same "Fast Copy is not supported with the current set of transformations on the query" error? While the error in the indicator does suggest that Fast copy may not end up being used, it is a check at design-time, and runtime behavior can be different

Re: Fast Copy DataFlowGen2

SofBL — Mon, 03 Nov 2025 09:31:03 GMT

Although I tested this on the 31th of october. I decided to give it a new try with a brandnew dataflow gen2. Doing exactly the same steps as prior it now does state that fast copy will be used.
To explain it in more detail, I create the connection to the database and then edit the M query manual to remove the section stating

, CreateNavigationProperties = false

after doing that it worked. I am happy.

Re: Fast Copy DataFlowGen2

srinidhiprabhu — Mon, 03 Nov 2025 17:23:59 GMT

SofBL Glad to see it working for you.