Forum Discussion
Dataflow gen 2 doesn't transfer data between lakehouse tables?
After reading the link you sent it seems that there is a delay to transfer data, so I implemented the Wait activity and put the wait to 3 minutes and it seems to work. So when bronze is done it'll wait 3 min before executing the silver dataflow gen 2 and it seems to work.
Thanks for your help, very appriciated!
I'm trying both to see what happens.
The refresh is after one another with 30 min margin, and the bronze DF takes about 10 min so it should be fine but sometimes it still doesn't work as I said earlier.
The pipeline is a couple of hours later just to see if it's also working and connected so when bronze is done do the silver one, it worked yesterday so will see later on today if it's the same result aka working.
If it still doesn't work, you can also consider to create a support ticket: https://support.fabric.microsoft.com/en/support/
One theory:
Your second dataflow is using Lakehouse.Contents() M function (you can verify this in Advanced Editor in your dataflow) to get the content from the Lakehouse.
The Lakehouse.Contents() function uses the SQL Analytics Endpoint of the Lakehouse. At least this is true when using Lakehouse.Contents() in Power BI Desktop. I'm guessing it's the same also when using Lakehouse.Contents() in Dataflows Gen2.
So if there is a long delay to sync data between your Lakehouse and SQL Analytics Endpoint, this might explain the behaviour you're seeing, because your second dataflow would be querying the SQL Analytics Endpoint.
https://www.reddit.com/r/MicrosoftFabric/s/leYLOX5uRK
https://www.reddit.com/r/MicrosoftFabric/s/nexpMsKFfx
https://learn.microsoft.com/en-us/fabric/data-warehouse/sql-analytics-endpoint-performance
To verify whether the Dataflow is actually querying the SQL Analytics Endpoint, you could go to the SQL Analytics Endpoint -> queryinsights -> exec_requests_history, and verify if you can find that your Dataflow queries appear here.
Thanks very much for that!
It seems like more people are saying the same thing, so I'll try to set at wait timer between the dataflows and see what the sweetspot is for it to transfer from bronze to silver and take all the rows with it.After reading the link you sent it seems that there is a delay to transfer data, so I implemented the Wait activity and put the wait to 3 minutes and it seems to work. So when bronze is done it'll wait 3 min before executing the silver dataflow gen 2 and it seems to work.
Thanks for your help, very appriciated!