Oauth2 gateway bugs affecting datasets not dataflows (WHY IN 2024)

One thing that is particularly frustrating about the dataset oauth2 bug is the inconsistency with dataflows.  I've been trying to attach a debugger to find out why the datasets don't properly refresh their oauth2 tokens.  

One curious thing I've found with a debugger is that the mashup engine knows that it needs a new oauth2 token after an hour has passed, yet it can't manage to get one.  It spends two minutes trying, and keeps giving up.  Below is a callstack showing the point of code where the mashup engine spends lots of time, for every round-trip to an AAD kind of connector (... this starts happening once an hour has passed):

 	System.Core.dll!System.IO.Pipes.PipeStream.ReadFileNative(Microsoft.Win32.SafeHandles.SafePipeHandle handle, byte[] buffer, int offset, int count, System.Threading.NativeOverlapped* overlapped, out int hr)	Unknown
 	System.Core.dll!System.IO.Pipes.PipeStream.ReadCore(byte[] buffer, int offset, int count)	Unknown
>	mscorlib.dll!System.IO.BinaryReader.FillBuffer(int numBytes) Line 748	C#
 	mscorlib.dll!System.IO.BinaryReader.ReadInt32() Line 281	C#
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.MessageSerializer.Deserialize(System.IO.BinaryReader reader)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.StreamMessenger.Read()	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.ChannelMessenger.Read(Microsoft.Mashup.Evaluator.ChannelMessenger.MessageChannel channel)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.ChannelMessenger.MessageChannel.Read()	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.Interface.IMessageChannelExtensions.WaitFor<Microsoft.Mashup.EngineHost.Services.RemoteCredentialServiceFactory.CredentialResponseMessage>(Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.EngineHost.Services.RemoteCredentialServiceFactory.Proxy.RefreshCredential(Microsoft.Mashup.Engine.Interface.IResource resource, bool forceRefresh)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Runtime.Extensibility.InvocationEngineHost.DataSourceInvocationHost.RefreshCredential(Microsoft.Mashup.Engine.Interface.IResource resource, bool forceRefresh)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Http.OAuthCredentialExtensions.RefreshTokenAsNeeded(Microsoft.Mashup.Engine.Interface.OAuthCredential credential, Microsoft.Mashup.Engine.Interface.IEngineHost engineHost, Microsoft.Mashup.Engine.Interface.IResource resource, bool forceRefresh)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Http.HttpResourceCredentialDispatcher.WebRequestCredentialDispatcher.ApplyOAuthCredential(Microsoft.Mashup.Engine.Interface.OAuthCredential credential, bool useBase64Encoding)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Http.HttpResourceCredentialDispatcher.WebRequestCredentialDispatcher.Apply(Microsoft.Mashup.Engine.Interface.OAuthCredential credential)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Http.HttpServices.ApplyCredentials(Microsoft.Mashup.Engine1.Library.Http.MashupHttpWebRequest httpRequest, string resourceKind, System.Uri requestUri, Microsoft.Mashup.Engine.Interface.ResourceCredentialCollection credentials, Microsoft.Mashup.Engine.Interface.IEngineHost hostEnvironment, string oAuthResource)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Http.HttpRequest.ApplyCredentialsToRequest(System.Net.WebRequest webRequest, Microsoft.Mashup.Engine.Interface.ResourceCredentialCollection credentials)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Http.Request.CreateWebRequest(System.Uri uri, Microsoft.Mashup.Engine.Interface.ResourceCredentialCollection credentials)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Http.Request.CreateResponse(Microsoft.Mashup.Engine.Interface.ResourceCredentialCollection credentials)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Http.Request.GetResponseCore(Microsoft.Mashup.Engine.Interface.ResourceCredentialCollection credentials)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Http.RetryPolicy.Execute<System.__Canon>(Microsoft.Mashup.Engine.Interface.IEngineHost host, Microsoft.Mashup.Engine.Interface.IResource resource, System.Func<System.__Canon> func, Microsoft.Mashup.Engine1.Library.Common.Tracer tracer)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Http.Request.GetResponse(Microsoft.Mashup.Engine.Interface.ResourceCredentialCollection credentials, Microsoft.Mashup.Engine1.Library.Http.Request.SecurityExceptionCreator securityExceptionCreator, bool tokenRefreshed)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Web.WebContentsBinaryValue.GetResponse(Microsoft.Mashup.Engine1.Library.Http.Request request)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Web.WebContentsBinaryValue.OpenStream()	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Library.Web.WebContentsBinaryValue.AsBytes.get()	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Runtime.Library.Binary.BufferFunctionValue.TypedInvoke(Microsoft.Mashup.Engine1.Runtime.Value value)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Runtime.NativeFunctionValue1<Microsoft.Mashup.Engine1.Runtime.Value, Microsoft.Mashup.Engine1.Runtime.Value>.Invoke(Microsoft.Mashup.Engine1.Runtime.Value arg0)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction1.Execute(ref Microsoft.Mashup.Engine1.Language.MembersFrame1 frame)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.MembersFunctionValue1.Invoke(Microsoft.Mashup.Engine1.Runtime.Value arg0)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.this[int].get(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.this[int].get(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction2.Execute(Microsoft.Mashup.Engine1.Runtime.Value frame)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.this[int].get(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.Instruction.Execute(Microsoft.Mashup.Engine1.Runtime.Value frame, Microsoft.Mashup.Engine1.Language.Instruction[] instructions)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction.Execute(Microsoft.Mashup.Engine1.Runtime.Value frame)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.this[int].get(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.InstructionInvocationInstruction2.Execute(Microsoft.Mashup.Engine1.Runtime.Value frame)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.Force(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RecordInstruction.RuntimeRecordValue.this[int].get(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.RuntimeFunctionValueN.Invoke(Microsoft.Mashup.Engine1.Runtime.Value[] args)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Runtime.TransformValueReference.Value.get()	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Runtime.RecordValue.ReferenceArrayRecordValue.this[int].get(int index)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.Query.ExpandListColumnQuery.GetRowsets()	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.Query.ExpandListColumnQuery.GetRows()	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine1.Language.Query.SelectRowsQuery.SelectRowsEnumerable.SelectRowsEnumerator.MoveNext()	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.OleDb.Serialization.DataReaderPageReader.ObjectPage.Read(System.Data.IDataReader reader, int rowCount)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.OleDb.Serialization.DataReaderPageReader.Read(Microsoft.OleDb.Serialization.DataReaderPageReader.ObjectPage page)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Engine.Interface.Tracing.TracingPageReader.Read(Microsoft.Data.Serialization.IPage page)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.RemotePageReader.RunStub.AnonymousMethod__0()	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.EvaluationHost.ReportExceptions(string entryName, Microsoft.Mashup.Engine.Interface.IEngineHost engineHost, Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, System.Action action)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.RemotePageReader.RunStub(Microsoft.Mashup.Engine.Interface.IEngineHost engineHost, Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, System.Func<Microsoft.Data.Serialization.IPageReader> getPageReader)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.Service.OnBeginGetResult.AnonymousMethod__0()	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.EvaluationHost.ReportExceptions(Microsoft.Mashup.Engine.Interface.Tracing.IHostTrace trace, Microsoft.Mashup.Engine.Interface.IEngineHost engineHost, Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, System.Action action)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.Service.OnBeginGetResult<Microsoft.Mashup.Engine.Interface.IDataReaderSource>(Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.BeginGetResultMessage message, System.Action<Microsoft.Mashup.Evaluator.Interface.EvaluationResult2<Microsoft.Mashup.Engine.Interface.IDataReaderSource>> action)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.Service.OnBeginGetDataReaderSource(Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, Microsoft.Mashup.Evaluator.RemoteDocumentEvaluator.BeginGetDataReaderSourceMessage message)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, Microsoft.Mashup.Evaluator.Interface.Message message)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, Microsoft.Mashup.Evaluator.Interface.Message message)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, Microsoft.Mashup.Evaluator.Interface.Message message)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.ChannelMessenger.OnMessageWithUnknownChannel(Microsoft.Mashup.Evaluator.Interface.IMessageChannel baseChannel, Microsoft.Mashup.Evaluator.ChannelMessenger.MessageWithUnknownChannel messageWithUnknownChannel)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.MessageHandlers.TryDispatch(Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, Microsoft.Mashup.Evaluator.Interface.Message message)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.ChannelMessenger.ChannelMessageHandlers.TryDispatch(Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, Microsoft.Mashup.Evaluator.Interface.Message message)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.MessageHandlers.Dispatch(Microsoft.Mashup.Evaluator.Interface.IMessageChannel channel, Microsoft.Mashup.Evaluator.Interface.Message message)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.EvaluationHost.Run()	Unknown
 	Microsoft.Mashup.Container.NetFX45.exe!Microsoft.Mashup.Container.EvaluationContainerMain.Run(object args)	Unknown
 	Microsoft.MashupEngine.dll!Microsoft.Mashup.Evaluator.SafeThread2.CreateAction.AnonymousMethod__0(object o)	Unknown
 	Microsoft.Mashup.Container.NetFX45.exe!Microsoft.Mashup.Container.EvaluationContainerMain.SafeRun(string[] args)	Unknown
 	Microsoft.Mashup.Container.NetFX45.exe!Microsoft.Mashup.Container.EvaluationContainerMain.Main(string[] args)	Unknown

Notice that the stack spends a lot of time waiting for a response within the method:

OAuthCredentialExtensions.RefreshTokenAsNeeded

Would sure be nice if the dataset would refresh the token as needed!

I know that Microsoft probably had the best of intentions when it fixed the oauth behavior of dataflows.  But having done so in dataflows, they need to also do the same for datasets.  Else this just has the appearance of a silly bug.

It is a massive problem when PQ is not compatible or portable between datasets and dataflows.  Users waste massive amounts of time flailing about with issues like this one!