Forum Discussion
Dataflow Gen2 not reading latest data after Copy Job
Hi Stefanve - Have you tried triggering the SQL endpoint refresh before the bronze to silver dataflow execution?
I had a similar issue (instead of dataflow gen2 I had notebooks), where I added an extra activity between bronze and silver in the pipeline, to call a notebook with this API that triggers the SQL Endpoint refresh of the source lakehouse.
Thank you for your answers,
tayloramy The Dataflow Gen2 source is connected to the bronze lakehouse and not to the SQL endpoint.
Gpop13 Thanks for the links; I will test the refresh of the SQL endpoint. However, I still don’t understand why this is necessary since I’m not using the SQL endpoint.
The interesting part is that, as a test, I added a notebook with a query to check whether the notebook would pick up the latest changes. This step is placed before the 20-minute delay, and now, on our latest refresh the dataflow did picked up the most recent data.