Data Flow Gen2 Best Practices

[wojciech]

Hi all,

There are many great resources on Microsoft Learn regarding best practices for Dataflows Gen1:
https://learn.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-best-practices

While some of these practices still apply to Gen2, it seems there is a gap in guidance for more complex patterns with Dataflows Gen2:
https://learn.microsoft.com/en-us/power-query/dataflows/best-practices-developing-complex-dataflows

According to the article above, we should separate dataflows into staging and transformation layers. That approach makes sense and I’ve been following it for quite some time.

However, in Gen2, there doesn’t appear to be a concept of linked entities as there was in Gen1. Based on that, I’m wondering whether the following pattern is still valid:

Staging Dataflow – exports raw data with staging turned on and no destination set
Transformation Dataflow – connects to the upstream staging dataflow with staging turned off and optionally sets a destination

Would this pattern still work as expected in Gen2?

I’d really appreciate if anyone could share some insights or point me to relevant resources, as I haven’t been able to find a definitive answer.

Thanks,

WJ

[V-yubandi-msft]

Hi @wojciech ,

Thank you for engaging with the Microsoft Fabric Community.

The staging and transformation pattern is still valid in Dataflows Gen2, but it works differently due to the removal of linked entities. In Gen2, staging is done using Lakehouse or Warehouse storage instead of linked entities. This approach maintains the separation between raw data extraction and transformation layers.

Thanks for your response @Akash_Varuna .

Regards,

Yugandhar.

[wojciech]

Thank you @V-yubandi-msft ,

Would it mean that the downstream "transformation" dataflow should connect to a staging lakehouse/warehouse or staging and transformation queries must now be a part of a single dataflow?

[V-yubandi-msft · Accepted solution]

Hi @wojciech ,

Thanks for your response. 

No, they don’t have to be combined into a single dataflow. You can still maintain separate staging and transformation layers, but the transformation dataflow must explicitly connect to the staging storage (Lakehouse/Warehouse).

Regards,

Yugandhar.

[wojciech]

Thanks, @V-yubandi-msft ,

I did some testing yesterday, and it looks like with staging enabled - since it provisions a Lakehouse/Warehouse behind the scenes - you can implement incremental refresh end-to-end. First, you set the policies on the staging dataflow, and then set them again on the transformation dataflow.

Appreciate your response!

[Akash_Varuna]

Hi @wojciech  Dataflow Gen2 removes linked entities but keeps the staging-transformation pattern viable with adjustments. Use Staging Dataflows to export raw data with staging enabled, and connect Transformation Dataflows directly to the storage account tied to the staging output. Ensure correct storage setup and configurations are done properly.

[wojciech]

Hey @Akash_Varuna,

Thanks for coming back to me, much appreciated!

Could you explain what do you mean by connecting directly to the storage account tied to the staging output. Do you mean connecting to:

- staging dataflow through dataflow connector

- staging warehouse/lakehouse directly

- actual ADLS account for staging lakehouse/warehouse - not sure that's even possible

"Ensure correct storage setup and configurations are done properly" - is there a documantation explaining what proper configuration looks like by any chance?

Many Thanks,

WJ