Forum Discussion
Dataflows Gen 2 vs Lakehouse for storing data in staging / transform phase
I learned that Dataflows Gen 2 use an internal Lakehouse to store data when you enable staging on a query in the dataflow. You could then re-use that data in another dataflow through an powerplatforms-connector. What would be the benefit of the other route: writing the query output to a Lakehouse and re-using that data in another dataflow through the OneHub/Lakehouse connector? It is more work to store the data in the Lakehouse (for every query you have to manually set the output destination table, checking datatypes etc.) so now I am thinking using the Dataflows Gen 2 internal staging Lakehouse is better, am I right or do I overlook imported implications?
Hi Anonymous ,
First of all if you are dealing with large amounts of data or complex data transformations, then Writing to a Lakehouse has more extra control and scalability.
If your main goal is to integrate data flows within the Power Platform ecosystem, then Using Internal Lakehouse Staging may be more efficient.
So in summary if simplicity and fast integration are your top priorities, Internal Lakehouse Staging may be a better choice. However, if you need more control and scalability, then writing to Lakehouse and using the OneHub/Lakehouse connector may be worth the extra effort.
If you want to learn more about this, I think you can read this document: Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn
Best Regards
Yilong Zhou
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
1 Reply
Hi Anonymous ,
First of all if you are dealing with large amounts of data or complex data transformations, then Writing to a Lakehouse has more extra control and scalability.
If your main goal is to integrate data flows within the Power Platform ecosystem, then Using Internal Lakehouse Staging may be more efficient.
So in summary if simplicity and fast integration are your top priorities, Internal Lakehouse Staging may be a better choice. However, if you need more control and scalability, then writing to Lakehouse and using the OneHub/Lakehouse connector may be worth the extra effort.
If you want to learn more about this, I think you can read this document: Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn
Best Regards
Yilong Zhou
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.