Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Dataflow gen 2 doesn't transfer data between lakehouse tables?

Hi community! I have two different dataflows, the first one I connect to the source (Business Central), transform some columns to text and then have a data destination to dump it all in a lakehous...
Dataflows
General Question
Need Help
Show and Tell
  • Anonymous's avatar
    Anonymous
    2 years ago

    After reading the link you sent it seems that there is a delay to transfer data, so I implemented the Wait activity and put the wait to 3 minutes and it seems to work. So when bronze is done it'll wait 3 min before executing the silver dataflow gen 2 and it seems to work.

    Thanks for your help, very appriciated!

     

frithjof_v's avatar
frithjof_v
Community Champion
2 years ago

Are the two Dataflows run on a scheduled refresh?

 

Have you checked that the first Dataflow has finished refreshing before the second Dataflow starts refreshing?

 

Or are they run inside a Data Pipeline? If so, how are the activities connected?

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    I'm trying both to see what happens.
    The refresh is after one another with 30 min margin, and the bronze DF takes about 10 min so it should be fine but sometimes it still doesn't work as I said earlier.

    The pipeline is a couple of hours later just to see if it's also working and connected so when bronze is done do the silver one, it worked yesterday so will see later on today if it's the same result aka working.

      • frithjof_v's avatar
        frithjof_v
        Community Champion
        2 years ago

        One theory:

         

        Your second dataflow is using Lakehouse.Contents() M function (you can verify this in Advanced Editor in your dataflow) to get the content from the Lakehouse.

         

        The Lakehouse.Contents() function uses the SQL Analytics Endpoint of the Lakehouse. At least this is true when using Lakehouse.Contents() in Power BI Desktop. I'm guessing it's the same also when using Lakehouse.Contents() in Dataflows Gen2.

         

        So if there is a long delay to sync data between your Lakehouse and SQL Analytics Endpoint, this might explain the behaviour you're seeing, because your second dataflow would be querying the SQL Analytics Endpoint.

         

        https://www.reddit.com/r/MicrosoftFabric/s/leYLOX5uRK

         

        https://www.reddit.com/r/MicrosoftFabric/s/nexpMsKFfx

         

        https://learn.microsoft.com/en-us/fabric/data-warehouse/sql-analytics-endpoint-performance

         

        To verify whether the Dataflow is actually querying the SQL Analytics Endpoint, you could go to the SQL Analytics Endpoint -> queryinsights -> exec_requests_history, and verify if you can find that your Dataflow queries appear here.