Power BI dataflows timing out when attempting to download csv files from Heroku

[tmcdonald6472]

We have 35 dataflow jobs spread out across 13 workspaces that have been experiencing job failures since 1/10/2025. The datasource for each job is a Heroku dataclip URL that's downloads a csv file  (e.g. https://data.heroku.com/dataclips/hgxupghsgstjwmyqlagkdubmesbu.csv).   It appears the jobs are timing out after running for about 5 1/2 minutes.  When I go to edit the job in Power Query, it returns a status message that it's waiting on data.heroku.com.  It also returns a message that it's taking longer than usual to return the query data.  After about six minutes or Power BI returns a message that it's cancelling execution of the query.  I reviewed all of the dataflow jobs and observed that all of them are failing most of the time and a couple have not completed successfully since 1/9.  I did take a URL from a couple of the datasources that were failing and configured them as a data source in Power BI desktop.  In every case the Power BI desktop was able to retreive the data from the data source.  I talked to Heroku support and they did not see any issues on their end that would cause this issue.  

[v-veshwara-msft]

Hi @tmcdonald6472 ,

We wanted to follow up and check if the solutions provided addressed your needs. If so, kindly mark it as a solution to assist others. If you require further assistance, please don't hesitate to reach out.

Thank you!

[v-veshwara-msft]

Hi @tmcdonald6472 ,

We just wanted to check in again regarding your issue. If you’ve found a solution, marking the reply as the solution and leaving a kudos would be greatly appreciated, it helps the community and others with similar questions.

If you’re still facing challenges or have further questions, please don't hesitate to reach out.

[v-veshwara-msft]

Hi @tmcdonald6472 ,

We haven’t heard from you since our last response and just wanted to check whether the solutions provided addressed your needs. If they did, we’d appreciate it if you could mark the response as the solution to help others benefit from it.

If the issue persists or you need further assistance, please don’t hesitate to reach out. We’re here to help!

Thank you.

[v-veshwara-msft · Accepted solution]

Hi @tmcdonald6472,
Thank you for reaching out to us.
Based on the details you've shared, it looks like the issue is related to the time it takes for Power BI Dataflows to pull data from your Heroku dataclip URLs. Here’s why this might be happening and some steps you can take to fix it.

Possible Causes and Troubleshooting Steps:

1. Timeout Limits in Power BI Dataflows:
Power BI Dataflows may have a time limit for how long they can run, and if the data takes too long to load, the job will fail. This could be happening if the Heroku dataclip is slow to respond or if it's pulling a large amount of data.

Steps to try:

 -Test the Heroku URLs directly in a browser to see how long they take to load.

 -If they take too long, consider simplifying your query or reducing the amount of data being pulled.

2. Optimize Power Query:
If your queries are complex or you're downloading too much data at once, it can slow things down and cause timeouts.
Steps to try:

-Try simplifying your query by removing unnecessary columns or filtering out unnecessary rows.

-Edit a failing Dataflow query and preview the data with fewer transformations to see if it loads faster.

3. Power BI Capacity Issues:
If your Power BI workspace is under a lot of use, it could be running out of resources, which might cause delays or failures in your Dataflows.

Steps to try:

-Check your workspace's capacity usage to make sure it's not overloaded and optimize the workload in your workspace.

Alternative Option: Use Data Pipelines

If optimizing the queries and adjusting schedules doesn’t fix the problem, consider using Data Pipelines in Microsoft Fabric:

1. Create a Data Pipeline to retrieve the CSV data from Heroku using the HTTP connector and save it in Lakehouse.
2. Use the Delay and Until Activities: If the data doesn't load or encounters issues, you can implement Delay or Until activities in your pipeline:
        Delay: Add a delay between retries if the Heroku URL is responding slowly. For example, you can delay the process for a few minutes before trying again.
        Until: Use the Until activity to repeatedly try fetching the data until it succeeds or reaches a retry limit.
3. Reconfigure your Dataflows to pull data from the Lakehouse. This reduces the risk of timeouts and provides a more stable, efficient data retrieval process.
   
   

Hope this helps. Please reach out if you need further assistance.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and a kudos would be appreciated.

[SaiTejaTalasila]

Hi @tmcdonald6472 ,

Based on provided details, it looks like the issue is with the power gateway. Please use tools like fiddler to monitor the traffic on your gateway and you find more details.

I hope it will be helpful.

Thanks,

Sai Teja 

[GilbertQ]

Hi @tmcdonald6472 

Could it be possible to put some details into the connection to the URL where it can extend the timeout?

[tmcdonald6472]

Unfortunately, there's not. For the csv/text data source, it takes the file path or URL as the location of the file and give the option to either link to the file or upload a file to OneDrive.  What's interesting is that if I go into Power BI desktop and set the same URL for the data source there, it has no issues retrieving the data.  I've tried this with other Heroku URL's and I see the same thing.  Power BI desktop has no problems retrieving the data, but the Power BI service dataflow job times out when trying to retrieve data from the same URL.