Power BI dataflows

[BharathS1307]

Hi All,
I am trying to update one of my dataflows 
1) There is a step for filtered rows, i.e I am filtering some of the values from a field and then pivoting them then merging them with a dim table to get the required data for a ID column, previously I had only 12 values filtered in that column and now I need to add 3 more values in the OR condition of the filtered rows step, after inputting thatmthe flow is loading for more time. I waited for more time and went to the last step of that query and gave save and close and teh query validation window pops up and it validates all the query. For the newly edited query it is validating till 11-12Gb then its throwing a message 

"Failed to analyze issues in the query"

So, I tried duplicatiing the same query and in the filtered row step added only 3 new values insted of all 15, plan is have these 3 values data separately and already prepared 12 values pivot table in other query and in the end I planned to append both, but the newly duplicated wuery with the 3 values in filtered row step its taking so much time to load the step(data).

what will be the best approach to solve this?

[tayloramy · Accepted solution]

Hi @BharathS1307, 

How much data are you working with? 

THis type of error generally means that you're trying to process too much data at once. 

Is there any filtering or removing of columns that you can do before the pivot? 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

[v-pnaroju-msft]

Hi DevikaDayal,

Thank you for the followup.

Based on my understanding, incremental refresh with Azure Databricks is supported in Microsoft Fabric when using Dataflows Gen2. Most issues do not stem from platform reliability. Rather, they arise from very large historical datasets combined with complex transformations such as filters, pivots, and merges being executed inside Dataflows.

Dataflows do not independently manage large-scale partitioning. Therefore, incremental logic should be pushed to the source by using a date or timestamp watermark and applying the filters directly in the Databricks SQL query. Dataflows Gen2 should be used primarily for light shaping, standardisation, and reuse, while Fabric Data Pipelines are recommended for orchestration, retries, and dependency management. Fabric Data Pipelines integrate well with Dataflows and provide a Fabric native alternative to Azure Synapse Pipelines.

Compared to Gen1, Dataflows Gen2 are Fabric native and offer better scalability, improved pipeline integration, and enhanced monitoring, making them the recommended choice for enterprise scenarios.

For more information, please refer to:
Differences between Dataflow Gen1 and Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
What is Data Factory - Microsoft Fabric | Microsoft Learn

We hope the above information helps resolve the issue. If you have any further queries, please feel free to contact the Microsoft Fabric Community.

Thank you.

[v-pnaroju-msft]

Hi DevikaDayal,

Thank you for contacting the Microsoft Fabric Community Forum.

Based on my understanding, Microsoft Fabric Dataflows can be reliable at scale when designed appropriately. The earlier issue does not indicate a Fabric reliability problem, rather, it stems from very large data volumes combined with computationally intensive transformations (for example, OR based filters, pivots, and merges) performed within Dataflows. These operations can cause high memory consumption and validation failures, because Dataflows are not optimised for large and complex transformations on raw data.

Please consider the following approach that may help resolve the issue:

1. Push heavy filtering, joins and pivots to the source system (for example, SQL, Databricks, Synapse, etc.).
2. Use Dataflows primarily for light shaping, standardisation and reuse, rather than for large scale transformations.
3. Design modular or layered Dataflows instead of monolithic flows.
4. Use Fabric Data Pipelines for orchestration, retries and dependency management.
5. Monitor health using Dataflow refresh history and the Capacity Metrics app to detect resource pressure proactively.

Additionally, please refer to the links below for further information:
Differences between Dataflow Gen1 and Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
Data refresh in Power BI - Power BI | Microsoft Learn
What is the Microsoft Fabric Capacity Metrics app? - Microsoft Fabric | Microsoft Learn

We hope the above information helps to resolve the issue. If you have any further queries, please feel free to contact the Microsoft Fabric Community.

Thank you.

[DevikaDayal]

Hello Everyone,

I am trying to implement incremental refresh using dataflows in my Fabric Environment. 

My datasource is Azure Databricks.

Can someone please share their experience for a similar setup? I want to understand what are the limitations you have faced after switiching to dataflows? 

what challenges from a data handling perspective have been faced?

How smoothly does dataflow integrate with Azure Synapse Pipelines and any challenges faced in that?

Please if possible , share the working level differences you have experienced between dataflow gen 1 and gen 2.

Please share resource links if any.

Thanks

Devika

[v-pnaroju-msft]

Hi BharathS1307,

Thank you for the update. Please continue to use the Microsoft Fabric community for any further queries.

Thank you.

[v-pnaroju-msft]

Hi BharathS1307,

We are following up to see if what we shared solved your issue. If you need more support, please reach out to the Microsoft Fabric community.

Thank you.

[v-pnaroju-msft]

Thankyou, @tayloramy for your response.

Hi BharathS1307,

We appreciate your inquiry through the Microsoft Fabric Community Forum.

We would like to inquire whether have you got the chance to check the solution provided by @tayloramy to resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community.

Thank you.

[tayloramy · Accepted solution]

Hi @BharathS1307, 

How much data are you working with? 

THis type of error generally means that you're trying to process too much data at once. 

Is there any filtering or removing of columns that you can do before the pivot? 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

[BharathS1307]

Hi @tayloramy , Thanks for the response
Yes before adding filtering step I am removing most of the columns and I am filtering with only 3 columns, adding filtering in one of the columns and I am receiving this error

The dataload is high only, but we need this filtering to be part of it, Can you suggest some steps which can make this filteration and merge work?

[BharathS1307]

Hi @tayloramy, 
As it contains huge data, and as I mentioned we are filtering 15 values and making it as pivot, Can I create duplicate of the main query and do filtering for 3 values separately like 3*5, then total it would be 5 queries and after loading the data in Power BI. I can trying appending them or keep it as separate tables for use?

please let me know will this step works!
Thanks

[tayloramy · Accepted solution]

Hi @BharathS1307, 

GIve it a shot and see what happens. That is the best way to learn. 

Without a clearer understanding of what you're trying to achieve and the structure of the data you're working with, I can't give any specific recommendations. Whatever you do try to filter data early and limit it to only what you need so that the power query engine isn't trying to process all of it at once. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[BharathS1307]

Tried most of the methods in the dataflow, but still faced the same data volume issue,So we are currently working in the source to get the data as per our convinence with required filters.

Thanks for the help!!

[DevikaDayal]

@BharathS1307 : How are you maintaining and monitoring large number of dataflows in Microsoft Fabric? I am trying to setup Dataflows ecosystem in my org but I am skeptical about dataflows reliability and how to maintaing them?