Fast Copy DataFlowGen2

[shiva_fabric]

Hi Team,

Im trying to move files from azure data lake gen2 to lakehouse in ms fabric through dataflowgen2 using fast copy.But every time i do it im getting following error.

is this an existing issue ? if not can you help me with resolving this issue?

[srinidhiprabhu]

@shiva_fabric Are you still running into this problem? 

[Expiscornovus]

Hi @shiva_fabric,

The known limitations sections of the fast copy documentation does mention the below:

Writing data into an existing table in Lakehouse isn't supported.

https://learn.microsoft.com/en-us/fabric/data-factory/dataflows-gen2-fast-copy#known-limitations

[shiva_fabric]

HI 

Expiscornovus

we are not trying to write the data into existing table. I have gone throught the document.

We are trying to Move the files from azuredatalake gen 2 to fabric lakehouse . what will be the best appraoch using fast copy dataglow gen 2 feature.

Regards,

Shiva

[SofBL]

Same kind of issue here,
Using the Azure SQL DB connector I use a native sql query to load data into a new Lakehouse table. and then it states, We encountered an error while trying to validate the query for Fast Copy. I proceeded, publish and the refresh failed stating
Fast Copy is not supported with the current set of transformations on the query

but every requirement seems to be approved, the connector is supported 
Lakehouse -> new table (no fixed schema), Replace data, Dynamic Schema 
no transformations outside the native query. 
there is just a link back to the https://learn.microsoft.com/en-us/fabric/data-factory/dataflows-gen2-fast-copy site but hey after reading that several times I am confused. This definitly requires a more detailed error message to help determine what goes wrong with the set up of fast copy. 

[srinidhiprabhu]

@SofBL What options does your M query to connect to Sql.Database contain? For Fast copy, we only allow the CommandTimeout and Query options. Options like ConnectionTimeout and EnableCrossDatabaseFolding are not supported. All options are documented here, out of which only the Query and CommandTimeout options are currently supported

[SofBL]

settings above are default and result in this M query 

let
Source = Sql.Database("Redacted", "BC_Lakehouse", [Query = "SELECT [Document_No] #(lf)FROM [REDACTED]", CreateNavigationProperties = false])
in
Source

I removed the , CreateNavigationProperties = false option to bring this back to
let
Source = Sql.Database("Redacted", "BC_Lakehouse", [Query = "SELECT [Document_No] #(lf)FROM [REDACTED]"])
in
Source

but it makes no difference. 

[srinidhiprabhu]

The CreateNavigationProperties option is why you would be failing to use Fast copy previously. Have you tried saving your dataflow after removing the option to check if you still hit the same "Fast Copy is not supported with the current set of transformations on the query" error? While the error in the indicator does suggest that Fast copy may not end up being used, it is a check at design-time, and runtime behavior can be different

[SofBL]

Although I tested this on the 31th of october. I decided to give it a new try with a brandnew dataflow gen2. Doing exactly the same steps as prior it now does state that fast copy will be used. 
To explain it in more detail, I create the connection to the database and then edit the M query manual to remove the section stating 

, CreateNavigationProperties = false

after doing that it worked.  I am happy. 

[srinidhiprabhu]

@SofBL Glad to see it working for you.