Forum Discussion

Russell_DC's avatar
Russell_DC
Regular Visitor
3 months ago
Solved

Dataflow Gen2 Copying Old Data

I have a dataflow that is transforming data and then moving it to a staging table in a Silver Lakehouse. I keep having issues where the number of rows moved to that lakehouse doesnt match what I am s...
dataflow
Need Help
  • tayloramy's avatar
    tayloramy
    3 months ago

    Hi Russell_DC, 

     

    If you have the skills to move this from dataflow gen 2 to notebooks, I would do that. Notebooks are far more powerful and flexible than dataflow gen 2, and as long as things are configured properly on the spark side, or if you use python instead of spark, can use quite a bit less CU than dataflow gen 2 does. 

     

    That being said, if you look at the run history from the workspace instead of from inside the dataflow, do the numbers look better to you? 
    I've found that going to recent runs from clicking the ... on the item in the workspace instead of opening the item has been more reliable. 

tayloramy's avatar
tayloramy
Icon for Super User rankSuper User
3 months ago

Hi Russell_DC, 

 

If you have the skills to move this from dataflow gen 2 to notebooks, I would do that. Notebooks are far more powerful and flexible than dataflow gen 2, and as long as things are configured properly on the spark side, or if you use python instead of spark, can use quite a bit less CU than dataflow gen 2 does. 

 

That being said, if you look at the run history from the workspace instead of from inside the dataflow, do the numbers look better to you? 
I've found that going to recent runs from clicking the ... on the item in the workspace instead of opening the item has been more reliable.