Forum Discussion

Russell_DC's avatar
Russell_DC
Regular Visitor
1 month ago
Solved

Dataflow Gen2 Copying Old Data

I have a dataflow that is transforming data and then moving it to a staging table in a Silver Lakehouse. I keep having issues where the number of rows moved to that lakehouse doesnt match what I am s...
Dataflow
Need Help
  • tayloramy's avatar
    tayloramy
    1 month ago

    Hi Russell_DC, 

     

    If you have the skills to move this from dataflow gen 2 to notebooks, I would do that. Notebooks are far more powerful and flexible than dataflow gen 2, and as long as things are configured properly on the spark side, or if you use python instead of spark, can use quite a bit less CU than dataflow gen 2 does. 

     

    That being said, if you look at the run history from the workspace instead of from inside the dataflow, do the numbers look better to you? 
    I've found that going to recent runs from clicking the ... on the item in the workspace instead of opening the item has been more reliable. 

Parchitect's avatar
Parchitect
Solution Sage
1 month ago

I would not move this to notebooks yet. First I would try to prove whether the mismatch is coming from the saved dataflow run or from the Lakehouse destination mapping.

The most useful next step is to check the exact refresh run:

  1. Make a small harmless change in the dataflow.
  2. Use Save & run, not just preview/refresh in the editor.
  3. Open Recent runs for that exact run.
  4. Download the detailed refresh logs.
  5. Check which query/activity actually wrote to the Lakehouse and how many rows it wrote.

The preview and the refresh result are not always the same thing to troubleshoot. The preview shows what you are seeing in the editor, but the refresh writes from the saved dataflow definition and from the query that has the data destination attached.

 

If the detailed logs show that the dataflow wrote 3 rows, then the issue is before or inside the saved dataflow execution. If the logs show 2 rows but the Lakehouse table ends up with 3, then I would look at the destination settings/mapping, especially whether the destination is appending instead of replacing.

 

So I would start with the detailed refresh logs and the exact run ID before redesigning this as a notebook.

Reference: View refresh history and monitor Dataflow Gen2 refreshes

 

Best regards,
Solutions Architect · Microsoft Fabric - Parchitect

💡Did my response help you? Clicking Kudos is a small gesture that goes a long way, it encourages contributors and helps the community thrive!
✔️Did I answer your question? Please mark my post as a Solution, it helps others find the answer faster.