Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Gen2 DataFlows

I recently created a data flow and connected it to the lakhouse. As I create more reports I am having to put additional tables in my data flow. I am now noticing that my data flow is taking a while t...
Need Help
  • ToddChitt's avatar
    ToddChitt
    1 year ago

    I think there is a maximum of 50 tables per Data Flow. If you have that many and are worried about performance, I suggest you break up the transformations into a second step, like in a stored procedure that moves data from the lakehouse to a warehouse. This frees up the Data Flow to be a straight-foward 'move data as fast as you can' operation. 

    Look into the 'medallion architecture' philosphy.

ToddChitt's avatar
ToddChitt
Icon for Super User rankSuper User
1 year ago

I think there is a maximum of 50 tables per Data Flow. If you have that many and are worried about performance, I suggest you break up the transformations into a second step, like in a stored procedure that moves data from the lakehouse to a warehouse. This frees up the Data Flow to be a straight-foward 'move data as fast as you can' operation. 

Look into the 'medallion architecture' philosphy.