Avoiding the authoring timeout of ten minutes in Gen2 dataflows.

[dbeavon3]

There is a "web.contents" PQ operation that I need to execute.  It regularly lasts more than ten minutes to respond.

I thought that moving to GEN2 dataflows, together with recent enhancements (to power query online) would resolve this arbitrary limit.  I was hopeful about the "publish" operation that performs the evaluation of the PQ as an offline operation. 

The publish operation should not be impacted the arbitrary ten minute limit that is defined under "authoring limits" documentation, right?

https://learn.microsoft.com/en-us/power-query/power-query-online-limits#authoring-limits

I still seem to be encountering a ten minute cancellation/retry loop when performing the publish operation for a GEN2 dataflow.

What can we do to publish a long-running GEN2 dataflow?  I know that the publish operation is the step that causes the problem.  As long as I can get past the step, I know have seen GEN2 dataflows run for longer than ten minutes.  However in order to get past the "publish" operation, I need to get lucky at least one time, and for that to happen the "web.contents" needs to complete in an unreasonably short amount of time.  Given the growth of the data, we don't believe we will be this lucky anymore!

Any tips would be appreciated.  I have not found a good solution for this yet.  Moreover, the ten minute limit during a "publish" operation doesn't appear to be well documented.  (Is that a bug? Or is it assumed to be an "authoring" operation?)

The web service that is referenced in "web.contents" is a custom site that was created by another team at my company.  Ideally the mashup engine would have a way to send it a url parameter or http header that indicated when an "authoring" operation was underway, (so the ten minutes of work could be short-circuited).  But I don't think there is any way to parameterize, or distinguish between "authoring" requests to the site, and "actual" refreshes.

Any tips would be appreciated.

[dbeavon3 · Accepted solution]

I was able to open a support case, and request for the timeout to be documented.

Apparently the "publish" operation has a similar timeout to the "authoring" experience.

For every entity, it must be refreshed in under ten minutes or else the dataflow cannot be published.  Below are the details.  This constraint is very dramatic, and serves to greatly restrict the types of solutions we can build with GEN2 dataflows.  In GEN1 dataflows we were able to compile entities that were far bigger than GEN2.  Those entities could be transmitted to the PBI service even if they took up to five hours to be compiled.

https://learn.microsoft.com/en-us/fabric/data-factory/data-factory-limitations#data-factory-dataflow...
Data Factory Dataflow Gen2 limitations

After you save/publish your dataflow gen2 we require the validation/publish process to finish within 10 minutes per query. If you exceed this 10 minute limit try to simplify your queries or split your queries in dataflow gen2.

[v-karpurapud]

Hi @dbeavon3 

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-karpurapud]

Hi @dbeavon3 

Thanks for actively participating in MS Fabric Community Support.

I understand your frustration and concerns. As of now, there is no documented setting to increase this limit.

I suggest raising a support ticket. If you have already done so, I kindly request you to wait for the response.

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

In addition to that, I recommend submitting your detailed feedback and ideas through Microsoft's official feedback channels, such as the Microsoft Fabric Ideas.  . Feedback submitted through these channels is frequently reviewed by the product teams and can contribute to meaningful improvements.

If my response has resolved your query, please mark it as the Accepted Solution to help others. Additionally, I would appreciate a 'Kudos' if you found my response helpful.

Thank you!

 

[tharunkumarRTK]

@dbeavon3 

I understand that max evaluation time out limit for authoring in Power Query online or dataflow gen 1 or gen 2 is 10 mins, Are you sure that publish button will not appear till the evaualtion is complete? 

I am not sure if Microsoft would increase this limit, may be you can put an idea in ideas forum. 

Meantime the workaround is to breakdown the query into simple blocks. Checkout the below link for more information

https://community.fabric.microsoft.com/t5/Service/Evaluation-was-cancell/m-p/4388026

Need a Power BI Consultation? Hire me on Upwork

Connect on LinkedIn

Did I answer your question? Mark my post as a solution!
If I helped you, click on the Thumbs Up to give Kudos. Proud to be a Super User!

[rajendraongole1]

Hi @dbeavon3  - You're right that the documentation focuses on "authoring limits," but it doesn't clearly address whether the publish operation is subject to the same 10-minute timeout.

Possible alternatives:

1.Preload Data Elsewhere Before Power Query Requests It
If the Web.Contents call is retrieving data from a custom API, could you introduce a caching mechanism on the API side? That way, Power Query wouldn't need to wait as long.

2.Use a Dataflow Staging Approach
If your data is large and slow to retrieve, consider setting up a first-stage dataflow that pulls in a limited portion of the data, then a second-stage dataflow that processes it further.

Microsoft Fabric Support and Status | Microsoft Fabric

[dbeavon3]

Hi @rajendraongole1 

I might submit a bug.  Perhaps this is a regression.  If we can't publish, then it seems to mean that no dataflow can ever run for longer than ten mins, since it can't be published in the first place.

The error messages during "publishing" are so frustrating.  They simply say "something went wrong" and leave it at that.  It wouldn't surprise me if the majority of the PG team doesn't even know how their stuff is supposed ot work.

I don't think most people would include the "publish" operation in the same category as the authoring activities.  If there are timeouts for the "publish" operation then they should be documented separately.

[dbeavon3]

Hi @rajendraongole1  

I submitted a case and there is a serious known issue (bug) that has been affecting customers since November.  The November gateway is broken for dataflows.

Here is a link:

https://learn.microsoft.com/en-us/fabric/known-issues/known-issue-844-intermittent-refresh-failure-g...

Its described as an "intermittent issue", yet it causes repeated failures again and again and again, every time I try to publish a GEN2 dataflow.

The cancellations and retries seem to occur about 4 or 5 times and are ten minutes apart. 

The software developers on the gateway team appear to be applying the ten-minute "authoring" restriction to the actual published version of the dataflow.  (The Mindtree team I talked to has already taken a lot of support calls about this bug.)

[dbeavon3]

There are clues from a year ago that seem to indicate things worked differently.  For example, the following post on reddit seems to say that while the "authoring" was restricted to a ten minute timeout, the "publishing" WAS NOT restricted:

... so this has the appearance of a regression bug that is affecting us in 02/02/2025