Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago

Fast Copy DataFlowGen2

Hi Team, Im trying to move files from azure data lake gen2 to lakehouse in ms fabric through dataflowgen2 using fast copy.But every time i do it im getting following error.       is this...
Dataflow
Need Help
Anonymous's avatar
Anonymous
Not applicable
1 year ago

HI 

Expiscornovus

we are not trying to write the data into existing table. I have gone throught the document.

 

We are trying to Move the files from azuredatalake gen 2 to fabric lakehouse . what will be the best appraoch using fast copy dataglow gen 2 feature.

 

Regards,

Shiva

 

 

SofBL's avatar
SofBL
Advocate II
9 months ago

Same kind of issue here,
Using the Azure SQL DB connector I use a native sql query to load data into a new Lakehouse table. and then it states, We encountered an error while trying to validate the query for Fast Copy. I proceeded, publish and the refresh failed stating
Fast Copy is not supported with the current set of transformations on the query

but every requirement seems to be approved, the connector is supported 
Lakehouse -> new table (no fixed schema), Replace data, Dynamic Schema 
no transformations outside the native query. 
there is just a link back to the https://learn.microsoft.com/en-us/fabric/data-factory/dataflows-gen2-fast-copy site but hey after reading that several times I am confused. This definitly requires a more detailed error message to help determine what goes wrong with the set up of fast copy. 

  • srinidhiprabhu's avatar
    srinidhiprabhu
    Microsoft Employee
    9 months ago

    SofBL What options does your M query to connect to Sql.Database contain? For Fast copy, we only allow the CommandTimeout and Query options. Options like ConnectionTimeout and EnableCrossDatabaseFolding are not supported. All options are documented here, out of which only the Query and CommandTimeout options are currently supported

    • SofBL's avatar
      SofBL
      Advocate II
      9 months ago

      settings above are default and result in this M query 

      let
      Source = Sql.Database("Redacted", "BC_Lakehouse", [Query = "SELECT [Document_No] #(lf)FROM [REDACTED]", CreateNavigationProperties = false])
      in
      Source

      I removed the , CreateNavigationProperties = false option to bring this back to
      let
      Source = Sql.Database("Redacted", "BC_Lakehouse", [Query = "SELECT [Document_No] #(lf)FROM [REDACTED]"])
      in
      Source

       

      but it makes no difference. 

       

      • srinidhiprabhu's avatar
        srinidhiprabhu
        Microsoft Employee
        9 months ago

        The CreateNavigationProperties option is why you would be failing to use Fast copy previously. Have you tried saving your dataflow after removing the option to check if you still hit the same "Fast Copy is not supported with the current set of transformations on the query" error? While the error in the indicator does suggest that Fast copy may not end up being used, it is a check at design-time, and runtime behavior can be different