Power BI is turning 10, and we’re marking the occasion with a special community challenge. Use your creativity to tell a story, uncover trends, or highlight something unexpected.
Get startedJoin us at FabCon Vienna from September 15-18, 2025, for the ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM. Get registered
We use a medallion architecture based on Lakehouses (LH) and Gen2 dataflows to process and prepare data for productive use in a Power BI report.
Data is processed sequentially through multiple dataflows and intermediate results are stored in staging LH, bronze LH, etc.
The execution of the dataflows is orchestrated by a pipeline with a generic structure like this:
dataflow1 ==> LH1 == dataflow2 ==> LH2
We have noticed that sometimes dataflow2 does not use the data written by dataflow1 in the n-th pass of the pipeline. Instead, the data from the (n-1) run of the pipeline was read by dataflow2.
To investigate this behavior, I created a simplified test pipeline:
"Init Write" writes a table with 2 million rows into LH and WH.
After a delay of 2 minutes, two dataflows "LH Rewrite" and "WH Rewrite" are executed 10 times in a row.
Each dataflow reads the table from LH/WH finds the highest Index and adds another row (current date/time and Index+1) to it and writes the whole table back to LH/WH ("replace" is used instead of "append").
After 10 iterations, the resulting tables are as follows.
Lakehouse:
Warehouse:
Apparently "LH Rewrite" sometimes gets outdated data from the LH and the read/write process does not seem to be managed properly. After 10 consecutive executions, the Index column in LH is corrupted.
In the WH, each read operation seems to wait until the write operation is finished and I get the 10 updates of the Index column.
I observed that this LH behavior is independent of enabled/disabled staging in the "LH Rewrite" dataflow.
How should we set up the dataflow or Lakehouse to ensure proper subsequent write and read behavior of Lakehouses?
------------
Footnotes:
Hi @Michal_Izak ,
Some steps that can be taken for the problem you describe:
Best Regards,
Adamk Kong
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi Adamk Kong,
thank you for your feedback on my problem.
Regarding your suggestions:
This is your chance to engage directly with the engineering team behind Fabric and Power BI. Share your experiences and shape the future.
Check out the June 2025 Fabric update to learn about new features.