Forum Discussion
Data Flow Gen2 Best Practices
Hi wojciech ,
Thanks for your response.
No, they don’t have to be combined into a single dataflow. You can still maintain separate staging and transformation layers, but the transformation dataflow must explicitly connect to the staging storage (Lakehouse/Warehouse).
Regards,
Yugandhar.
Hi wojciech ,
Thank you for engaging with the Microsoft Fabric Community.
The staging and transformation pattern is still valid in Dataflows Gen2, but it works differently due to the removal of linked entities. In Gen2, staging is done using Lakehouse or Warehouse storage instead of linked entities. This approach maintains the separation between raw data extraction and transformation layers.
Thanks for your response Akash_Varuna .
Regards,
Yugandhar.
Thank you V-yubandi-msft ,
Would it mean that the downstream "transformation" dataflow should connect to a staging lakehouse/warehouse or staging and transformation queries must now be a part of a single dataflow?
Hi wojciech ,
Thanks for your response.
No, they don’t have to be combined into a single dataflow. You can still maintain separate staging and transformation layers, but the transformation dataflow must explicitly connect to the staging storage (Lakehouse/Warehouse).
Regards,
Yugandhar.
Thanks, V-yubandi-msft ,
I did some testing yesterday, and it looks like with staging enabled - since it provisions a Lakehouse/Warehouse behind the scenes - you can implement incremental refresh end-to-end. First, you set the policies on the staging dataflow, and then set them again on the transformation dataflow.
Appreciate your response!