Forum Discussion

Stefanve's avatar
Stefanve
Frequent Visitor
8 months ago
Solved

Dataflow Gen2 not reading latest data after Copy Job

We are experiencing an identical issue with two clients where a Dataflow Gen2 in Fabric (Silver layer) does not copy recent data from the Bronze layer during the execution of a nightly pipeline, even...
data engineering
data pipeline
lakehouse
Gpop13's avatar
Gpop13
Advocate IV
8 months ago

Hi Stefanve - Have you tried triggering the SQL endpoint refresh before the bronze to silver dataflow execution?

 

https://blog.fabric.microsoft.com/en-us/blog/refresh-sql-analytics-endpoint-metadata-rest-api-now-in-public-preview?ft=All

I had a similar issue (instead of dataflow gen2 I had notebooks), where I added an extra activity between bronze and silver in the pipeline, to call a notebook with this API that triggers the SQL Endpoint refresh of the source lakehouse.

 

https://github.com/microsoft/fabric-toolbox/blob/main/samples/notebook-refresh-tables-in-sql-endpoint/MDSyncNewRESTAPI.ipynb