Forum Discussion

BharathS1307's avatar
BharathS1307
Frequent Visitor
8 months ago
Solved

Power BI dataflows

Hi All, I am trying to update one of my dataflows  1) There is a step for filtered rows, i.e I am filtering some of the values from a field and then pivoting them then merging them with a dim table...
Dataflow
Need Help
  • tayloramy's avatar
    tayloramy
    8 months ago

    Hi BharathS1307, 

     

    How much data are you working with? 

    THis type of error generally means that you're trying to process too much data at once. 

     

    Is there any filtering or removing of columns that you can do before the pivot? 

     

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

  • tayloramy's avatar
    tayloramy
    8 months ago

    Hi BharathS1307, 

     

    GIve it a shot and see what happens. That is the best way to learn. 

     

    Without a clearer understanding of what you're trying to achieve and the structure of the data you're working with, I can't give any specific recommendations. Whatever you do try to filter data early and limit it to only what you need so that the power query engine isn't trying to process all of it at once. 

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

v-pnaroju-msft's avatar
v-pnaroju-msft
Community Support
7 months ago

Hi DevikaDayal,

Thank you for contacting the Microsoft Fabric Community Forum.

Based on my understanding, Microsoft Fabric Dataflows can be reliable at scale when designed appropriately. The earlier issue does not indicate a Fabric reliability problem, rather, it stems from very large data volumes combined with computationally intensive transformations (for example, OR based filters, pivots, and merges) performed within Dataflows. These operations can cause high memory consumption and validation failures, because Dataflows are not optimised for large and complex transformations on raw data.

Please consider the following approach that may help resolve the issue:

  1. Push heavy filtering, joins and pivots to the source system (for example, SQL, Databricks, Synapse, etc.).
  2. Use Dataflows primarily for light shaping, standardisation and reuse, rather than for large scale transformations.
  3. Design modular or layered Dataflows instead of monolithic flows.
  4. Use Fabric Data Pipelines for orchestration, retries and dependency management.
  5. Monitor health using Dataflow refresh history and the Capacity Metrics app to detect resource pressure proactively.

Additionally, please refer to the links below for further information:
Differences between Dataflow Gen1 and Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
Data refresh in Power BI - Power BI | Microsoft Learn
What is the Microsoft Fabric Capacity Metrics app? - Microsoft Fabric | Microsoft Learn

We hope the above information helps to resolve the issue. If you have any further queries, please feel free to contact the Microsoft Fabric Community.

Thank you.

DevikaDayal's avatar
DevikaDayal
Frequent Visitor
7 months ago

Hello Everyone,

I am trying to implement incremental refresh using dataflows in my Fabric Environment. 

 

My datasource is Azure Databricks.

 

Can someone please share their experience for a similar setup? I want to understand what are the limitations you have faced after switiching to dataflows? 

 

what challenges from a data handling perspective have been faced?

 

How smoothly does dataflow integrate with Azure Synapse Pipelines and any challenges faced in that?

 

Please if possible , share the working level differences you have experienced between dataflow gen 1 and gen 2.

 

Please share resource links if any.

 

Thanks

Devika