Forum Discussion
Dataflow gen 2 doesn't transfer data between lakehouse tables?
After reading the link you sent it seems that there is a delay to transfer data, so I implemented the Wait activity and put the wait to 3 minutes and it seems to work. So when bronze is done it'll wait 3 min before executing the silver dataflow gen 2 and it seems to work.
Thanks for your help, very appriciated!
Do you get the same result if you query the SQL Analytics Endpoint in silver lakehouse, or if you query the silver lakehouse from Power BI desktop (use "let Source = Lakehouse.Contents() in Source" in Advanced Editor in Power BI Desktop).
Or if you connect a Dataflow to the silver lakehouse and count the rows in the table.
Just for clarification it's the same lakehouse, just that i named the table silver_*tablename* :).
I get the same result in notebook and sql query. The dataflow gives me no error, it all seems to work. But it kinda works 50/50, because sometimes when I refresh the dataflow manually it works and sometimes it doesn't.
I tried to wait to se if it's a delay or but it seems that it's only working every now and then and that it's very wierd.
When it doesn't work: is the data then unchanged in the silver table after refreshing the dataflow? Meaning it is still exactly the same old data there as before the dataflow run?
When it works: does it sometimes work after it has not worked? So sometimes it is not working and you just see old data, and then sometimes it works and then you get the new data?
Could you run this code in a Notebook cell:
%%sql
DESCRIBE HISTORY Prod_LH.silver_glEntries;
Then you get a list of all the versions of the table. Then you could check if it creates a new version at the time of the "failed" dataflow runs, or if nothing happens to the table at the time of the "failed" dataflow runs.
Thanks for your answer!
Well the operation column after I typed that is saying update on every row, so I guess it's updating the table but sometimes just don't take all the new rows from bronze to silver.
It's just wierd that the schedule for the dataflow just works sometimes, for instance yesterday it worked but today it only updated the bronze and not the silver table, but after a manually refresh I had the same total of rows.On top of that the dataflow for silver didn't say fail, just a regular pass.