Forum Discussion

Hussein_charif's avatar
7 months ago
Solved

Capacity Throttling

we have an F16, that has 3 pipelines that run incremental refresh, getting incremental data from dataverse and pushing them into a staging table, then updating the Fact tables with the data. the data refreshes near real-time, and every day there is on average around 1.5 million new records.

 

we have around 13 users, and around 6 reports. users create and extract paginated reports alot, which is causing Throttling to go up over 100% sometimes, and sometimes stays between 40-70%. how can this issue be solved?

  • cengizhanarslan's avatar
    cengizhanarslan
    7 months ago

    In Fabric/Power BI, a workspace is assigned to exactly one capacity at a time. So “two capacities” really means: two sets of workspaces, each mapped to a different capacity.

     

    1) Buy / provision a second capacity

    You’ll need:

    • Capacity A (existing): e.g., F16

    • Capacity B (new): e.g., F4/F8 (often enough for pipelines)

     

    2) Split by workspace (the only supported way)

    Create two workspace groups:

    A) “Data Engineering” workspace(s) → Capacity B

    Put pipelines / ingestion / staging here:

    • Data pipelines that load from Dataverse

    • Lakehouses / Warehouses used for staging and transformations

    • Scheduled jobs

    Assign these workspaces to Capacity B.

    B) “BI / Consumption” workspace(s) → Capacity A (F16)

    Put user workloads here:

    • Semantic models

    • Reports

    • Paginated reports

    • Apps

    Assign these workspaces to Capacity A.

  • Hi Hussein_charif

     

    There are two options to resolve this: 
    1: Optimize the reports/semantic models so they are more efficient and use less capacity, or

    2: get a bigger capacity. 

     

    It is very common for the Fabric ETL pipelines to be on a different capacity than the end user reports so that end users can't cause the ETL to fail. 

     

    What I would do: 

    1: create a new workspace for the Power BI reports

    2: move all reports into that new workspace 

    3: Either optimize the reports, or purchase a second capacity, and assign the report workspace to the new capacity. 

     

     

7 Replies

  • Hi Hussein_charif 

     

    You're experiencing the throttling issue because of your near real-time refresh coupled with high query load and large data volume. There are a few things you could do to improve this -

     

    1. Optimise Paginated Reports

    • Reduce the complexity of queries by encouraging users to use aggregated tables rather than querying Fact tables directly. Applying filters and avoiding “select all” options also helps improve performance.
    • Create summary tables within your model to provide pre-aggregated data for common reporting needs.

    2. Capacity Management

    • You could consider upgrading to a higher Fabric F SKU with greater compute resources. Autoscale features can also help manage peak workloads efficiently.

    3. Optimise Incremental Refresh

    • Ensure data partitions are efficient by using smaller partitions for recent data and archiving older data in separate tables, which reduces processing overhead.
    • Try to avoid scheduling data refreshes during periods of heavy querying.

    4. Use DirectQuery or Hybrid Tables

    • For scenarios requiring near real-time data, consider using DirectQuery or Hybrid tables to access the most recent information and reduce refresh demands.

    5. Monitoring and Tuning

    • Leverage the Capacity Metrics App to identify reports generating the highest load and to pinpoint periods of throttling.
    • Implement query caching and aggregation tables wherever possible to further optimise performance.

    Quick Wins

    • Schedule the generation of paginated reports during off-peak hours.
    • Advise users not to run multiple large paginated reports at the same time.
    • Apply row-level filters to paginated reports to reduce data volumes and improve efficiency.

    Hope this helps - please appreciate by leaving a Kudos or accepting as a Solution

  • 1) Separate “daytime BI” from “back-end pipelines” (biggest win)

    If your pipelines are near real-time, they’re constantly consuming capacity in the background. When users then run paginated exports, you get contention.

    Best practice is to put these on different capacities if possible:

    • Capacity A: Data Engineering / pipelines / refresh

    • Capacity B: BI consumption / paginated exports

    Even a smaller F-SKU for pipelines can stabilize the user-facing F16 dramatically.

     

    2) Control paginated export concurrency / behavior

    • Encourage users to export from a filtered view (not “all data”)

    • Avoid huge tablix detail exports; consider summary-first layouts

    • For repeated needs, use subscriptions or Power Automate so exports run on schedule rather than ad-hoc bursts

    • If the same paginated report is repeatedly run with the same parameters, consider caching patterns (where applicable)

    Also check whether they’re exporting PDF/Excel with a lot of pages—that is significantly heavier than small extracts.

     

    3) Stagger and/or reduce refresh pressure

    Near real-time incremental loads + daily 1.5M rows can be fine, but three pipelines continuously running can keep the capacity “warm” all day.

    • Reduce frequency if truly “near real-time” isn’t required (e.g., every 15 min vs every 2–5 min)

    • Batch changes (micro-batch) rather than constant processing

    • Ensure staging → fact merge is optimized (partitioning, incremental updates, avoid full table scans)

    If you have a daily peak period for exports, avoid running heavy pipeline steps during that window.

     

    4) Optimize the semantic model + paginated reports (reduce CPU/memory per export)

    4.1 Use the right connection type for paginated reports

    Paginated → Semantic model

    • Pros: reusable business logic, security (RLS), consistent metrics

    • Cons: exports can be heavy because each run triggers DAX query execution + rendering, often more expensive on capacity when the model is large/complex.

    Paginated → SQL endpoint / Warehouse / Lakehouse SQL

    • Pros: can be more predictable for large “table exports” (simple SELECTs), pushes work to the SQL engine

    • Cons: you must manage security there (or mirror RLS logic), and you may lose measure logic unless you rebuild it in SQL/views.

    Rule of thumb:

    • If paginated is mainly detail table extracts (many rows/columns) → SQL endpoint often scales better.

    • If paginated is pixel-perfect with semantic measures → semantic model is fine, but must be optimized.

    4.2 Reduce “export cost” by shaping paginated output

    • Avoid huge detail tablix with thousands of rows and complex expressions.

    • Require parameters (Date range, Business unit) and set sensible defaults.

    • Turn off expensive features unless needed (interactive sorting, complex grouping, nested tablix).

    4.3 Optimize the semantic model for paginated/export workload

    • Reduce cardinality: replace text keys with integers, remove high-cardinality text columns from fact.

    • Prefer star schema (dimension filters → fact), avoid many-to-many where possible.

    • Reduce measure cost:

      • Avoid iterators (SUMX) over large tables when a native aggregation works

      • Be careful with DISTINCTCOUNT on huge columns; consider alternatives (or aggregations)

    • Use aggregations / composite model if you have very large facts and repeated patterns.

    • Ensure incremental refresh partitions are correct and not forcing large scans.

    • If many exports hit the same visuals/queries, consider cache warming patterns (scheduled query hits), so exports don’t always pay full cold-start cost.

    Disclosure:
    This response was written based on my own professional experience and understanding of Microsoft Fabric. AI-based tools were used solely as a supporting aid for wording and structure refinement. All technical interpretations, validations, and conclusions are my own responsibility.

      • cengizhanarslan's avatar
        cengizhanarslan
        Super User

        In Fabric/Power BI, a workspace is assigned to exactly one capacity at a time. So “two capacities” really means: two sets of workspaces, each mapped to a different capacity.

         

        1) Buy / provision a second capacity

        You’ll need:

        • Capacity A (existing): e.g., F16

        • Capacity B (new): e.g., F4/F8 (often enough for pipelines)

         

        2) Split by workspace (the only supported way)

        Create two workspace groups:

        A) “Data Engineering” workspace(s) → Capacity B

        Put pipelines / ingestion / staging here:

        • Data pipelines that load from Dataverse

        • Lakehouses / Warehouses used for staging and transformations

        • Scheduled jobs

        Assign these workspaces to Capacity B.

        B) “BI / Consumption” workspace(s) → Capacity A (F16)

        Put user workloads here:

        • Semantic models

        • Reports

        • Paginated reports

        • Apps

        Assign these workspaces to Capacity A.

  • Hi Hussein_charif

     

    There are two options to resolve this: 
    1: Optimize the reports/semantic models so they are more efficient and use less capacity, or

    2: get a bigger capacity. 

     

    It is very common for the Fabric ETL pipelines to be on a different capacity than the end user reports so that end users can't cause the ETL to fail. 

     

    What I would do: 

    1: create a new workspace for the Power BI reports

    2: move all reports into that new workspace 

    3: Either optimize the reports, or purchase a second capacity, and assign the report workspace to the new capacity. 

     

     

  • v-prasare's avatar
    v-prasare
    Community Support

    Hi Hussein_charif,

    We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.


    tayloramy, deborshi_nag & cengizhanarslan ,Thanks for your prompt response

     

     

     

    Thank you for your patience and look forward to hearing from you.
    Best Regards,
    Prashanth Are
    MS Fabric community support

  • v-prasare's avatar
    v-prasare
    Community Support

    Hi Hussein_charif,

    We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.

     

     

     

    Thank you for your patience and look forward to hearing from you.
    Best Regards,
    Prashanth Are
    MS Fabric community support