Dataflow GEN 2 publish operation is buggy and introduces ten-minute timeouts

[dbeavon3]

I'm struggling with GEN2 dataflows.  They are extremely buggy.  I've upgraded to the February gateway, as suggested by Mindtree support, but it isn't helping.  I know there is a "known issue" about the unreliability of gateways, and that was created back in September (#844 link below).  But Mindtree claimed the issue would be mitigated by upgrading to the latest.  Unfortunately that doesn't appear to be the case.

Known issue - Intermittent refresh failure through on-premises data gateway - Microsoft Fabric | Mic...

There are unexplained timeouts happening after ten minutes during a "publish" operation.  These timeouts are not introduced by my own PQ code, they must be originating from Microsoft.  The logs don't say why my mashup is being cancelled after the ten minute timeout has expired.

Here is an example of a dataflow publish operation that failed after ten mins.  The fully refresh should have taken about an hour, and so would the publish/validation:

Here is a section of the logs that seems to indicate that a remote agent is pre-emptively cancelling my mashup:

Please Notice the parts saying:

• RemoteCancellationServiceFactory/Proxy/CancelAll
• SoftCancellingDocumentEvaluator/Evaluation/Cancel
• ContainerProcess/Kill
• DocumentEvaluator/GetResult at 1:21:17
• MashupCommand/Cancel at 1:31:17
• "CommandTimeout":"600" (ie. ten mins?)

I have seen announcements about well-known bugs in the gateway, eg as linked here:
https://community.fabric.microsoft.com/t5/Service/Known-Issue-PSA-Your-refreshes-may-die-2025-02-05/...

... however it is not easy to determine whether my bug is another recurrence of those well-known bugs, or if it is an altogether different bug.  So far I have NOT seen any "known issue" that describes the most obvious symptom of my bug, which is that it causes failures after exactly ten minutes (10 minutes) during a publish operation.

I'm aware that there is an "authoring" limit that causes cancellation after ten minutes.  But I'm told this should NOT impact the normal publish or the normal refresh.  More about that here:

https://community.fabric.microsoft.com/t5/Service/Avoiding-the-authoring-timeout-of-ten-minutes-in-G...

Any information would be appreciated.  I've also spent a couple weeks waiting for help from Mindtree but there is very little transparency or communication that originates from their corresponding Microsoft PG.  It is likely that I will need to move my ticket over to "unified support", before the PG will be willing to help with this bug.  Even if I do this, I'm not vey optimistic.  It seems like the bugs in this gateway have overwhelmed the PG, and things are starting to look a bit hopeless.  Things are almost at the tipping point, where the number of failures are as high as the number of successful refreshes:

I wish this platform wasn't so unreliable, or we might be able to actually use it for some mission-critical workloads.

[dbeavon3 · Accepted solution]

I wanted to close the loop here.  The primary problem I was facing was not the "authoring timeout" nor the "intermittent refresh failure thru the on-prem gateway".

The primary problem was an unexpected and undocumented timeout in the GEN2 dataflows that prevent them from being published if an entity takes more then 10 mins to refresh.    This took us by surprise, after an entity slowly grew to exceed ten minutes, and then the dataflow was not able to be published anymore.

I was able to open a support case, and request for the timeout to be documented.

Apparently the "publish" operation has a similar timeout to the "authoring" experience.

For every entity, it must be refreshed in under ten minutes or else the dataflow cannot be published.  Below are the details.  This constraint is very dramatic, and serves to greatly restrict the types of solutions we can build with GEN2 dataflows.  In GEN1 dataflows we were able to compile entities that were far bigger than GEN2.  Those entities could be transmitted to the PBI service even if they took up to five hours to be compiled.

https://learn.microsoft.com/en-us/fabric/data-factory/data-factory-limitations#data-factory-dataflow...
Data Factory Dataflow Gen2 limitations

After you save/publish your dataflow gen2 we require the validation/publish process to finish within 10 minutes per query. If you exceed this 10 minute limit try to simplify your queries or split your queries in dataflow gen2.

[V-yubandi-msft]

Hi @dbeavon3 ,

As this is an ongoing recurring issue. Currently, the support tickets and ICM tracking details are not available for us to provide exact timelines. We understand how important this is and want to assure you that the relevant team is actively working on it. The ICM is still in a mitigation phase. We truly appreciate your understanding and patience during this time. 

If you have Premier Access, you can try accessing it through that.

A Premier Access ID is a unique identifier assigned to each Microsoft Premier Support customer. It is used to track support cases and manage customer accounts. A Contract ID is a unique identifier assigned to each Microsoft support contract. It is used to track support cases and manage customer accounts.

Thank you for your understanding.

Regards,

Yugandhar.