topic Re: Dataflows Gen2: Issue loading a bigger table from on prem SQL Server to Lakehouse in Dataflow

Dataflows Gen2: Issue loading a bigger table from on prem SQL Server to Lakehouse

Anonymous — Tue, 11 Jul 2023 15:01:49 GMT

Hi,

I have come across similar issues during the same process (often connected to handshake and gateway), but our error seems to be different. Therefore, I am creating a new ticket for it.

We are currently loading data from an on-prem SQL Server using Gateway and Dataflows Gen2. Most of the objects are loading successfully without any problems.

However, we are encountering an issue when trying to ingest a table with 14 million records of transactions. The process fails after running for about an hour, specifically during the source dataset part.

The error message is a bit weird as it references Parquet, which makes me think it might be related to a staging area during the load process. It appears that the write to the Lakehouse never started, suggesting a problem with the staging mechanism.

Thank you for your attention.

Error:

There was a problem refreshing the dataflow. Please review the error message(s) below, fix the problem, and try again. Error Code: Mashup Exception Error, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Parquet: class parquet::ParquetStatusException (message: 'Unknown error: Microsoft.Mashup.Engine.Interface.ResourceAccessAuthorizationException: Exception of type 'Microsoft.Mashup.Engine.Interface.ResourceAccessAuthorizationException' was thrown. at Microsoft.Mashup.Engine1.Library.Http.Request.GetResponse(ResourceCredentialCollection credentials, RetryPolicy retryPolicy, SecurityExceptionCreator securityExceptionCreator, Boolean tokenRefreshed) at Microsoft.Mashup.Engine1.Library.Http.Request.GetResponse(ResourceCredentialCollection credentials, RetryPolicy retryPolicy, SecurityExceptionCreator securityExceptionCreator, Boolean tokenRefreshed) at Microsoft.Mashup.Engine1.Library.AzureBase.AzureBaseHelper.GetResponse(Request request, SecurityExceptionCreator securityExceptionCreator, Int32[] ignoredStatusCodes) at Microsoft.Mashup.Engine1.Library.Http.PagedHttpBinaryValue.<>c__DisplayClass30_0.<GetStreamWithRetry>b__0(Int64 streamOffset) at Microsoft.Internal.RetryStream.<>c__DisplayClass10_0.<Read>b__0() at Microsoft.Mashup.Engine1.Library.Http.RetryPolicy.Execute[TResult](IEngineHost host, Func`1 func) at Microsoft.Mashup.Common.StreamExtensions.ErrorTranslatingStream.Read(Byte[] buffer, Int32 offset, Int32 count) at Microsoft.Mashup.Common.StreamExtensions.CopyTo(Stream stream, Stream destination, Byte[] buffer) at Microsoft.Mashup.Engine1.Library.Http.PagedHttpBinaryValue.GetPage(IPagedStorage storage, String etag, Int32 pageIndex) at Microsoft.Internal.PagedSeekableStream.get_PageStream() at Microsoft.Internal.PagedSeekableStream.Read(Byte[] buffer, Int32 offset, Int32 count) at ParquetSharp.IO.ManagedRandomAccessFile.Read(Int64 nbytes, IntPtr bytesRead, IntPtr dest, String& exception)')

Re: Dataflows Gen2: Issue loading a bigger table from on prem SQL Server to Lakehouse

miguel — Fri, 14 Jul 2023 15:39:33 GMT

Could you please reach out customer support so an engineer can take a closer look at it?

https://support.fabric.microsoft.com/

Re: Dataflows Gen2: Issue loading a bigger table from on prem SQL Server to Lakehouse

tegrus-coutinho — Fri, 14 Jul 2023 19:44:19 GMT

Could you update us on this issue? Has it been resolved? I'm going through the same problem.
https://community.fabric.microsoft.com/t5/Service/Data-Fabric-Dataflow-Gen2-Error-Unable-to-Retrieve-Data-from/td-p/3329897#M97367

Re: Dataflows Gen2: Issue loading a bigger table from on prem SQL Server to Lakehouse

Anonymous — Sun, 16 Jul 2023 20:56:46 GMT

Hi tegrus-coutinho, it looks different. Anyway...

We tried it to run again.

It worked for one of bigger tables.

We run into issue with token expiration with another one 🙂 (probably dataflows engine did not handle it well)

So we added filter condition to query and divided an initial load to more runs. That worked for us.

So we have not contacted a support miguel thanks.

Re: Dataflows Gen2: Issue loading a bigger table from on prem SQL Server to Lakehouse

Anonymous — Wed, 19 Jul 2023 08:13:44 GMT

It appears that we are encountering regular issues with this load, and unfortunately, we are still facing some challenges. It seems we will discover all Error msgs. in Dataflows Gen2 early. :))

During the write phase, we recently came across the following error message: "There was a problem refreshing the dataflow. Please try again later." 😞