Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now

Reply
Anonymous
Not applicable

Dataflows Gen2: Issue loading a bigger table from on prem SQL Server to Lakehouse

Hi,

 

I have come across similar issues during the same process (often connected to handshake and gateway), but our error seems to be different. Therefore, I am creating a new ticket for it.

 

We are currently loading data from an on-prem SQL Server using Gateway and Dataflows Gen2. Most of the objects are loading successfully without any problems.

 

However, we are encountering an issue when trying to ingest a table with 14 million records of transactions. The process fails after running for about an hour, specifically during the source dataset part.

 

The error message is a bit weird as it references Parquet, which makes me think it might be related to a staging area during the load process. It appears that the write to the Lakehouse never started, suggesting a problem with the staging mechanism.

 

Thank you for your attention.

 

Error:

 

 

There was a problem refreshing the dataflow. Please review the error message(s) below, fix the problem, and try again.

Error Code: Mashup Exception Error, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Parquet: class parquet::ParquetStatusException (message: 'Unknown error: Microsoft.Mashup.Engine.Interface.ResourceAccessAuthorizationException: Exception of type 'Microsoft.Mashup.Engine.Interface.ResourceAccessAuthorizationException' was thrown.
   at Microsoft.Mashup.Engine1.Library.Http.Request.GetResponse(ResourceCredentialCollection credentials, RetryPolicy retryPolicy, SecurityExceptionCreator securityExceptionCreator, Boolean tokenRefreshed)
   at Microsoft.Mashup.Engine1.Library.Http.Request.GetResponse(ResourceCredentialCollection credentials, RetryPolicy retryPolicy, SecurityExceptionCreator securityExceptionCreator, Boolean tokenRefreshed)
   at Microsoft.Mashup.Engine1.Library.AzureBase.AzureBaseHelper.GetResponse(Request request, SecurityExceptionCreator securityExceptionCreator, Int32[] ignoredStatusCodes)
   at Microsoft.Mashup.Engine1.Library.Http.PagedHttpBinaryValue.<>c__DisplayClass30_0.<GetStreamWithRetry>b__0(Int64 streamOffset)
   at Microsoft.Internal.RetryStream.<>c__DisplayClass10_0.<Read>b__0()
   at Microsoft.Mashup.Engine1.Library.Http.RetryPolicy.Execute[TResult](IEngineHost host, Func`1 func)
   at Microsoft.Mashup.Common.StreamExtensions.ErrorTranslatingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Microsoft.Mashup.Common.StreamExtensions.CopyTo(Stream stream, Stream destination, Byte[] buffer)
   at Microsoft.Mashup.Engine1.Library.Http.PagedHttpBinaryValue.GetPage(IPagedStorage storage, String etag, Int32 pageIndex)
   at Microsoft.Internal.PagedSeekableStream.get_PageStream()
   at Microsoft.Internal.PagedSeekableStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at ParquetSharp.IO.ManagedRandomAccessFile.Read(Int64 nbytes, IntPtr bytesRead, IntPtr dest, String& exception)')

 

 

 

4 REPLIES 4
tegrus-coutinho
Frequent Visitor

Could you update us on this issue? Has it been resolved? I'm going through the same problem.
https://community.fabric.microsoft.com/t5/Service/Data-Fabric-Dataflow-Gen2-Error-Unable-to-Retrieve...

Anonymous
Not applicable

Hi @tegrus-coutinho, it looks different.  Anyway...  

 

We tried it to run again.

It worked for one of bigger tables.

We run into issue with token expiration with another one 🙂 (probably dataflows engine did not handle it well) 

 

So we added filter condition to query and divided an initial load to more runs.  That worked for us.  

 

So we have not contacted a support @miguel thanks.  

 

 

 

Anonymous
Not applicable

It appears that we are encountering regular issues with this load, and unfortunately, we are still facing some challenges. It seems we will discover all Error msgs. in Dataflows Gen2 early. :))

During the write phase, we recently came across the following error message: "There was a problem refreshing the dataflow. Please try again later." 😞

 

miguel
Community Admin
Community Admin

Could you please reach out customer support so an engineer can take a closer look at it?

 

https://support.fabric.microsoft.com/

Helpful resources

Announcements
November Carousel

Fabric Community Update - November 2024

Find out what's new and trending in the Fabric Community.

Live Sessions with Fabric DB

Be one of the first to start using Fabric Databases

Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.

November Update

Fabric Monthly Update - November 2024

Check out the November 2024 Fabric update to learn about new features.

Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Early Bird pricing ends December 9th.