Forum Discussion
Gen2 vs. Gen1 Dataflow and api refresh limits
Hey Team!
I currently have a gen1 dataflow that's being refreshed with a Power Automate flow on a loop (so every 20s give or take). This system has been working well and has no max daily refresh limit.
We would like to take advantage of some of the options gen2 dataflows have but when using the same method as mentioned above we are limited to 150 refreshes a day.
Does anyone have some options that would help us stay with a gen2 dataflow but still refresh the dataflow on an unlimited basis? This is not streaming data, rather it's a file that's being read in over and over, then updates to the latest data along with merging a few other data source, then feeding a direct query report that is on a page refresh for every 20s. The report is served out as an app from the Power BI service.
Any feedback is appreciated!!
Solutions for Unlimited Refreshes with Gen2
1. Use a Data Pipeline:
Create a Fabric Data Pipeline with a Dataflow activity.
Loop every 20 seconds using a ForEach or Wait activity.
Trigger via Power Automate. This may bypass the 150-refresh limit.
Note: Test for throttling or delays.
2. Write to Fabric Lakehouse with Direct Lake:
Configure Gen2 Dataflow to output to a Lakehouse Delta table.
Use Direct Lake mode in Power BI for near-real-time querying.
Reduce Dataflow refresh to every 1-2 minutes (30-60 refreshes/day, under the limit).
Note: Balances Gen2 features with refresh limits.
3. External Script with Fabric REST API:
Use a Python script (via Azure Functions) to call the /refreshes endpoint every 20 seconds.
Implement retry logic for throttling.
Note: Requires more setup.
4. Adjust Refresh Frequency:
Increase interval to 1-2 minutes (under 150 refreshes/day).
Use Gen2’s refresh monitoring to ensure reliability.
Note: Simplest but may impact data freshness.
3 Replies
Solutions for Unlimited Refreshes with Gen2
1. Use a Data Pipeline:
Create a Fabric Data Pipeline with a Dataflow activity.
Loop every 20 seconds using a ForEach or Wait activity.
Trigger via Power Automate. This may bypass the 150-refresh limit.
Note: Test for throttling or delays.
2. Write to Fabric Lakehouse with Direct Lake:
Configure Gen2 Dataflow to output to a Lakehouse Delta table.
Use Direct Lake mode in Power BI for near-real-time querying.
Reduce Dataflow refresh to every 1-2 minutes (30-60 refreshes/day, under the limit).
Note: Balances Gen2 features with refresh limits.
3. External Script with Fabric REST API:
Use a Python script (via Azure Functions) to call the /refreshes endpoint every 20 seconds.
Implement retry logic for throttling.
Note: Requires more setup.
4. Adjust Refresh Frequency:
Increase interval to 1-2 minutes (under 150 refreshes/day).
Use Gen2’s refresh monitoring to ensure reliability.
Note: Simplest but may impact data freshness.rosha_rosha Thanks for the feedback! I'll test these options.
Hello wayersGM
Gen1 allows unlimited refreshes, making it suitable for high-frequency scenarios.
• Gen2 imposes a strict 150 refreshes/day limit, regardless of trigger method.
• No official workaround exists to bypass this Gen2 limit for a single dataflow.
• To approach near-real-time updates with Gen2, consider architectural changes: use Lakehouse outputs, pipelines, or incremental refresh to optimize data freshness within the quota.
