Forum Discussion

BharathS1307's avatar
BharathS1307
Frequent Visitor
8 months ago
Solved

Power BI dataflows

Hi All, I am trying to update one of my dataflows  1) There is a step for filtered rows, i.e I am filtering some of the values from a field and then pivoting them then merging them with a dim table...
Dataflow
Need Help
  • tayloramy's avatar
    tayloramy
    8 months ago

    Hi BharathS1307, 

     

    How much data are you working with? 

    THis type of error generally means that you're trying to process too much data at once. 

     

    Is there any filtering or removing of columns that you can do before the pivot? 

     

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

  • tayloramy's avatar
    tayloramy
    8 months ago

    Hi BharathS1307, 

     

    GIve it a shot and see what happens. That is the best way to learn. 

     

    Without a clearer understanding of what you're trying to achieve and the structure of the data you're working with, I can't give any specific recommendations. Whatever you do try to filter data early and limit it to only what you need so that the power query engine isn't trying to process all of it at once. 

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

v-pnaroju-msft's avatar
v-pnaroju-msft
Community Support
7 months ago

Hi DevikaDayal,

Thank you for the followup.

Based on my understanding, incremental refresh with Azure Databricks is supported in Microsoft Fabric when using Dataflows Gen2. Most issues do not stem from platform reliability. Rather, they arise from very large historical datasets combined with complex transformations such as filters, pivots, and merges being executed inside Dataflows.

Dataflows do not independently manage large-scale partitioning. Therefore, incremental logic should be pushed to the source by using a date or timestamp watermark and applying the filters directly in the Databricks SQL query. Dataflows Gen2 should be used primarily for light shaping, standardisation, and reuse, while Fabric Data Pipelines are recommended for orchestration, retries, and dependency management. Fabric Data Pipelines integrate well with Dataflows and provide a Fabric native alternative to Azure Synapse Pipelines.

Compared to Gen1, Dataflows Gen2 are Fabric native and offer better scalability, improved pipeline integration, and enhanced monitoring, making them the recommended choice for enterprise scenarios.

For more information, please refer to:
Differences between Dataflow Gen1 and Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
What is Data Factory - Microsoft Fabric | Microsoft Learn

We hope the above information helps resolve the issue. If you have any further queries, please feel free to contact the Microsoft Fabric Community.

Thank you.