Control refresh order and priority across Power BI datasets

[ign_]

Detailed Description

Today, Power BI Service executes dataset refresh operations independently and often in parallel within the same workspace. There is no native mechanism to define execution order, priority, or dependencies between datasets. This creates operational challenges in environments with multiple datasets competing for shared resources such as on-premises data gateways, capacity (Premium or Fabric), and source systems.

In scenarios with large and small datasets scheduled for refresh at similar times, the lack of orchestration can lead to:

• Resource contention on gateways (CPU, memory, connection limits)
• Increased refresh failures due to overload or timeouts
• Unpredictable refresh completion times
• Difficulty meeting SLA requirements for critical datasets
• Inefficient use of capacity when heavy datasets start after lighter ones

Real-world scenario

An environment with 30 to 50 datasets:

• 5 large datasets (each taking 30 to 60 minutes)
• 25+ smaller datasets (5 to 10 minutes each)

When all refresh jobs trigger at the same time:

• Smaller datasets start first and consume gateway slots
• Larger datasets are queued or throttled
• Overall refresh window increases
• Failures occur due to long wait times or source system limits

Current workarounds

To control execution order, teams must implement external orchestration:

• Fabric Pipelines or Data Factory pipelines
• Power Automate flows
• Custom scripts using Power BI REST API
• Dataflows as intermediate dependency layers

These approaches add complexity, require additional infrastructure, and increase maintenance overhead.

Proposed Enhancement

Introduce native refresh orchestration capabilities in Power BI Service:

1. Dataset Priority

Allow assigning priority levels:

• High
• Medium
• Low

Scheduler executes higher priority datasets first.

---

2. Refresh Order Control

Allow defining explicit execution order within a workspace or schedule group:

Example:

1. Dataset A (large)
2. Dataset B
3. Dataset C

---

3. Dependency Management

Allow datasets to depend on others:

• Dataset B starts only after Dataset A completes successfully
• Support simple dependency chains and DAG structures

Expected Benefits

• Predictable refresh execution
• Better SLA compliance for critical datasets
• Reduced refresh failures
• Improved gateway and capacity utilization
• Elimination of external orchestration dependencies
• Simpler operational management

---

Business Impact

Organizations with enterprise-scale deployments face real constraints in managing refresh workloads. Native orchestration would reduce operational complexity and increase reliability, especially in regulated environments where data freshness and availability are critical.

---

Summary

Power BI needs built-in control over dataset refresh order, priority, and dependencies to support enterprise-scale workloads efficiently and reduce reliance on external orchestration tools.