Forum Discussion
Dataflow Gen2 not reading latest data after Copy Job
Hi Stefanve - Have you tried triggering the SQL endpoint refresh before the bronze to silver dataflow execution?
I had a similar issue (instead of dataflow gen2 I had notebooks), where I added an extra activity between bronze and silver in the pipeline, to call a notebook with this API that triggers the SQL Endpoint refresh of the source lakehouse.
Hi Stefanve,
Gpop13 is on the right track. Are you using the SQL endpoint in the dataflow?
THe SQL Endpoints can take a few minutes to sync themselves, so you should force a metadata sync after loading data via the copy job, and before reading data from the dataflow to ensure that everything is up to date.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.