Dataflow Gen2 Copying Old Data

[Russell_DC]

I have a dataflow that is transforming data and then moving it to a staging table in a Silver Lakehouse. I keep having issues where the number of rows moved to that lakehouse doesnt match what I am seeing in the dataflow preview or what it should be. A simple example of this is that this should have read 2 and written 2 rows but instead it read and wrote 3 rows. Upon investigating this was the 3 rows from the previos time it ran, suggesting a caching issue! I have others where I have 578 new rows in my table within the dataflow and then in the Staging table it has 212 rows. I'm very confused as to what is going on and concerned I may need to convert to notebooks for my transformations?

[tayloramy · Accepted solution]

Hi @Russell_DC, 

If you have the skills to move this from dataflow gen 2 to notebooks, I would do that. Notebooks are far more powerful and flexible than dataflow gen 2, and as long as things are configured properly on the spark side, or if you use python instead of spark, can use quite a bit less CU than dataflow gen 2 does. 

That being said, if you look at the run history from the workspace instead of from inside the dataflow, do the numbers look better to you? 
I've found that going to recent runs from clicking the ... on the item in the workspace instead of opening the item has been more reliable. 

[Parchitect]

I would not move this to notebooks yet. First I would try to prove whether the mismatch is coming from the saved dataflow run or from the Lakehouse destination mapping.

The most useful next step is to check the exact refresh run:

1. Make a small harmless change in the dataflow.
2. Use Save & run, not just preview/refresh in the editor.
3. Open Recent runs for that exact run.
4. Download the detailed refresh logs.
5. Check which query/activity actually wrote to the Lakehouse and how many rows it wrote.

The preview and the refresh result are not always the same thing to troubleshoot. The preview shows what you are seeing in the editor, but the refresh writes from the saved dataflow definition and from the query that has the data destination attached.

If the detailed logs show that the dataflow wrote 3 rows, then the issue is before or inside the saved dataflow execution. If the logs show 2 rows but the Lakehouse table ends up with 3, then I would look at the destination settings/mapping, especially whether the destination is appending instead of replacing.

So I would start with the detailed refresh logs and the exact run ID before redesigning this as a notebook.

Reference: View refresh history and monitor Dataflow Gen2 refreshes

Best regards,
Solutions Architect · Microsoft Fabric - Parchitect

💡Did my response help you? Clicking Kudos is a small gesture that goes a long way, it encourages contributors and helps the community thrive!
✔️Did I answer your question? Please mark my post as a Solution, it helps others find the answer faster.