Forum Discussion

wayersGM's avatar
wayersGM
Frequent Visitor
1 year ago
Solved

Gen2 vs. Gen1 Dataflow and api refresh limits

Hey Team! I currently have a gen1 dataflow that's being refreshed with a Power Automate flow on a loop (so every 20s give or take). This system has been working well and has no max daily refresh lim...
capacities
Real-Time hub
  • rosha_rosha's avatar
    rosha_rosha
    1 year ago

    Solutions for Unlimited Refreshes with Gen2

    1. Use a Data Pipeline:
    Create a Fabric Data Pipeline with a Dataflow activity.
    Loop every 20 seconds using a ForEach or Wait activity.
    Trigger via Power Automate. This may bypass the 150-refresh limit.
    Note: Test for throttling or delays.

    2. Write to Fabric Lakehouse with Direct Lake:
    Configure Gen2 Dataflow to output to a Lakehouse Delta table.
    Use Direct Lake mode in Power BI for near-real-time querying.
    Reduce Dataflow refresh to every 1-2 minutes (30-60 refreshes/day, under the limit).

    Note: Balances Gen2 features with refresh limits.

    3. External Script with Fabric REST API:
    Use a Python script (via Azure Functions) to call the /refreshes endpoint every 20 seconds.
    Implement retry logic for throttling.
    Note: Requires more setup.

    4. Adjust Refresh Frequency:
    Increase interval to 1-2 minutes (under 150 refreshes/day).
    Use Gen2’s refresh monitoring to ensure reliability.
    Note: Simplest but may impact data freshness.

nilendraFabric's avatar
nilendraFabric
Super User
1 year ago

Hello wayersGM 

 

Gen1 allows unlimited refreshes, making it suitable for high-frequency scenarios.

 


• Gen2 imposes a strict 150 refreshes/day limit, regardless of trigger method.

 


• No official workaround exists to bypass this Gen2 limit for a single dataflow.

 


• To approach near-real-time updates with Gen2, consider architectural changes: use Lakehouse outputs, pipelines, or incremental refresh to optimize data freshness within the quota.