Forum Discussion
Dataflow gen 2 doesn't transfer data between lakehouse tables?
After reading the link you sent it seems that there is a delay to transfer data, so I implemented the Wait activity and put the wait to 3 minutes and it seems to work. So when bronze is done it'll wait 3 min before executing the silver dataflow gen 2 and it seems to work.
Thanks for your help, very appriciated!
One theory:
Your second dataflow is using Lakehouse.Contents() M function (you can verify this in Advanced Editor in your dataflow) to get the content from the Lakehouse.
The Lakehouse.Contents() function uses the SQL Analytics Endpoint of the Lakehouse. At least this is true when using Lakehouse.Contents() in Power BI Desktop. I'm guessing it's the same also when using Lakehouse.Contents() in Dataflows Gen2.
So if there is a long delay to sync data between your Lakehouse and SQL Analytics Endpoint, this might explain the behaviour you're seeing, because your second dataflow would be querying the SQL Analytics Endpoint.
https://www.reddit.com/r/MicrosoftFabric/s/leYLOX5uRK
https://www.reddit.com/r/MicrosoftFabric/s/nexpMsKFfx
https://learn.microsoft.com/en-us/fabric/data-warehouse/sql-analytics-endpoint-performance
To verify whether the Dataflow is actually querying the SQL Analytics Endpoint, you could go to the SQL Analytics Endpoint -> queryinsights -> exec_requests_history, and verify if you can find that your Dataflow queries appear here.
After reading the link you sent it seems that there is a delay to transfer data, so I implemented the Wait activity and put the wait to 3 minutes and it seems to work. So when bronze is done it'll wait 3 min before executing the silver dataflow gen 2 and it seems to work.
Thanks for your help, very appriciated!