Sporadic Gateway Error (Mashup error) Dataflow

[Gabry]

Hi everyone,

I'm encountering a sporadic issue with one of my Power BI Dataflows, and I'm hoping someone can shed some light on it. Below is the error message I receive:

Error: Encountered user gateway exception: '<ccon>We're sorry  an error occurred during evaluation.</ccon> [DM_ErrorDetailNameCode_UnderlyingErrorCode]=-2147467259 [DM_ErrorDetailNameCode_UnderlyingErrorMessage]=<ccon>We're sorry  an error occurred during evaluation.</ccon> [DM_ErrorDetailNameCode_UnderlyingHResult]=-2147467259 [InnerType]=ErrorException <ccon>Could not find a part of the path 'C:\Windows\ServiceProfiles\PBIEgwService\AppData\Local\Temp\Microsoft\MashupProvider\Cache\0e540563-7d05-4a71-a27c-b0922e4afc82\0f744602-9404-48c1-b285-c0d93779ee1b.dat'.</ccon> [GatewayPipelineErrorCode]=DM_GWPipeline_Gateway_MashupDataAccessError [ErrorShortName]=GatewayClientErrorResponseException[GatewayId=244216]/MashupDataAccessInternalException[ErrorCode=-2147467259 HResult=-2147467259]/Wrapped(InternalMashupException)[ErrorCode=-2147467259 HResult=-2147467259]/Wrapped(ErrorException)[HResult=-2146233088]/Wrapped(ErrorException)[HResult=-2146233088]/Wrapped(ErrorException)[HResult=-2146233088]/Wrapped(ErrorException)[HResult=-2146233088]/Wrapped(ErrorException)[HResult=-2146233088] [ExceptionErrorShortName]=GatewayClientErrorResponseException[GatewayId=244216]'. RootActivityId = 50d231b8-6f09-4fc0-8a7c-110ec6bc1fcd.Param1 = <ccon>We're sorry  an error occurred during evaluation.</ccon> [DM_ErrorDetailNameCode_UnderlyingErrorCode]=-2147467259 [DM_ErrorDetailNameCode_UnderlyingErrorMessage]=<ccon>We're sorry  an error occurred during evaluation.</ccon> [DM_ErrorDetailNameCode_UnderlyingHResult]=-2147467259 [InnerType]=ErrorException <ccon>Could not find a part of the path 'C:\Windows\ServiceProfiles\PBIEgwService\AppData\Local\Temp\Microsoft\MashupProvider\Cache\0e540563-7d05-4a71-a27c-b0922e4afc82\0f744602-9404-48c1-b285-c0d93779ee1b.dat'.</ccon> [GatewayPipelineErrorCode]=DM_GWPipeline_Gateway_MashupDataAccessError [ErrorShortName]=GatewayClientErrorResponseException[GatewayId=244216]/MashupDataAccessInternalException[ErrorCode=-2147467259 HResult=-2147467259]/Wrapped(InternalMashupException)[ErrorCode=-2147467259 HResult=-2147467259]/Wrapped(ErrorException)[HResult=-2146233088]/Wrapped(ErrorException)[HResult=-2146233088]/Wrapped(ErrorException)[HResult=-2146233088]/Wrapped(ErrorException)[HResult=-2146233088]/Wrapped(ErrorException)[HResult=-2146233088] [ExceptionErrorShortName]=GatewayClientErrorResponseException[GatewayId=244216] Request ID: 9686bb16-eede-4ebc-94ec-97c40b286a62.

 Has anyone encountered a similar error before?

Could this be related to timeout or cache handling within the gateway?

Are there additional troubleshooting steps I should try to prevent this from happening again?

Any insights or advice would be greatly appreciated!

[lbendlin]

Could not find a part of the path 'C:\Windows\ServiceProfiles\PBIEgwService\AppData\Local\Temp\Microsoft\MashupProvider\Cache\0e540563-7d05-4a71-a27c-b0922e4afc82\0f744602-9404-48c1-b285-c0d93779ee1b.dat'.

Running out of disk space?

[Gabry]

Hi @lbendlin ,

thanks for the reply. It was my first thought but there are more than 40gb space. Do you think it's possible that it cache more than that?

Do you know how can I monitor this? There's like some windows tool that records free space?

[lbendlin]

Could network corruption play a role? Where is the data source in relation to the gateway, what is the latency?

[Gabry]

It could be yeah but I don't know how tò check.

The source Is a SQL server, the gateway Is on the same machine of the SQL server. How can I check latency?

If it's a network issue why the data flow Just doesn't try again when It fails, automatically

[lbendlin]

The source Is a SQL server, the gateway Is on the same machine of the SQL server.

Wait what?!?! Why?!?

There's no latency but there will be a bitter fight for resources.  Don't do that.

[Gabry]

Why? What's wrong? I usually build this setup with no issue. I do the etl during night hours.

What should I do instead?

[lbendlin]

Gateway VMs need to be dedicated to just the gateway service. It is designed to consume all the CPU and all the memory on that VM. As well as all the network for DQ. 

You are supposed to have multiple VMs per gateway anyway.

[Gabry]

I'm looking further into your suggestions and I keep investigating the issue, in the meantime, I noticed that I can set up a Power Automate flow to rerun the dataflow in case of failure, so I set this up, and it's working great.

[lbendlin]

I noticed that I can set up a Power Automate flow to rerun the dataflow in case of failure,

That's not what you want to do. First understand the reason for the failure then decide if you need to fix that or if a retry is appropriate.

[Gabry]

I see, I'm trying but I'm not able to understand the root cause.

I checked the gateway's logs but I can't understand a thing

Customer's system engineer told me we can't have a separate dedicated VM 

I checked the windows log in the event viewer and I didn't find any warning about disk space. I found an error on the mashup engine, this is the message:

Application: Microsoft.Mashup.Container.NetFX45.exe
Framework Version: v4.0.30319
Description: The application requested process termination through System.Environment.FailFast(string message).
Message: Could not find a part of the path 'C:\Windows\ServiceProfiles\PBIEgwService\AppData\Local\Temp\Microsoft\MashupProvider\Cache\28edb145-f663-4146-82de-bd70a008e155\90ed647b-8a5f-4658-82d4-489a614c7aed.dat'.
Stack:
at System.Environment.FailFast(System.String, System.Exception)
at Microsoft.Mashup.Container.EvaluationContainerMain+ExceptionHandler.TryHandleException(System.Exception)
at Microsoft.Mashup.Evaluator.SafeThread2.HandleException(System.Exception)
at Microsoft.Mashup.Evaluator.SafeThread2+<>c__DisplayClass9_0.<CreateAction>b__0(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
ata System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart(System.Object)

​Same, can't understand what this error is.

I didn't find any log about network issue, I don't know how to track it.
So I don't know what else to do right now

[lbendlin]

Customer's system engineer told me we can't have a separate dedicated VM 

Did they give a reason?

[Gabry]

They said they have already implemented a large machine 100% dedicated to us (which is true), and there’s no need to modify it. The atmosphere may not be ideal, as it could seem like we are trying to shift the issue onto their side instead of resolving it.

In the previous weeks I've already asked to increase the disk size, also to grant us working days to optimize the infrastructure. Which I have done, but it is still not enough to avoid having dataflows fail occasionally. Anyway, this is more on the relational side of things. I wish I could understand what this bug is on the technical side.
AS you suggested a new machine could help probably, but I usually build this type of set up and I've never had any issue, never encountered this type of error, even with bigger flows.

[lbendlin]

You just got lucky until now or didn't have the data volumes for the issue to manifest itself.   We've been through that phase.