topic Re: Gen2 Dataflow failures when loading medium sized datasets in Dataflow

Gen2 Dataflow failures when loading medium sized datasets

motoray — Wed, 19 Jul 2023 17:13:39 GMT

I've been trying to get a medium-sized (100,000 records for now) dataset to load from a SQL database (on prem and Azure) to a Fabric Warehouse from a Gen2 dataflow. I can get it to load 1000 or 10000 records, but once I step up to higher numbers, I get the error below. And yes--port 1433 is open. Also--I can load all the data no problem with a native Power BI dataset or a Gen1 dataflow.

It's almost like the connection to the source database times out after a certain period of time.

Mashup Exception Data Source Error Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Microsoft SQL: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) Details: DataSourceKind = Lakehouse;DataSourcePath = Lakehouse;Message = A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.);ErrorCode = -2146232060;Number = 10054;Class = 20

Re: Gen2 Dataflow failures when loading medium sized datasets

Meister_Kaio — Fri, 21 Jul 2023 09:23:45 GMT

Hi!

It appears that I'm encountering a similar problem, which relates to the number of rows in a file. I can successfully load files with over 100K rows from Azure ALDS Gen2 Storage to a Lakehouse. However, when attempting to transfer files between two Lakehouses, only the files with more than 100K rows encounter failure, while files with fewer rows transfer without any issues.

Re: Gen2 Dataflow failures when loading medium sized datasets

motoray — Mon, 24 Jul 2023 15:40:27 GMT

Now none of my Gen2 dataflows will complete so I'm not sure what's going on anymore. Time to call in Microsoft.

Re: Gen2 Dataflow failures when loading medium sized datasets

motoray — Mon, 24 Jul 2023 21:43:15 GMT

And as of right now, I still can't get any Gen2 dataflows to complete--not since last Thursday around 1 pm PST. Something is hard broken now. Have built new workspaces, lakehouse, dataflows...nothing works. The refreshes simply spin on forever now.

Re: Gen2 Dataflow failures when loading medium sized datasets

ShunOshidari — Tue, 25 Jul 2023 03:40:29 GMT

I'm glad I came across your post. We're experiencing a similar issue in our environment. We've recently begun to evaluate Fabric, and we were unsure if this issue had been occurring from the start.

Here's our situation:

Dataflow: Gen2

Source: SharePoint folder (Excel file, 3MB, 6000 rows)

Destination: Warehouse

Capacity region: Japan East

With this Excel file, we managed to successfully import up to approximately 4600 rows. I believe the actual limit might also depend on the number of columns.

As Gen2 uses Lakehouse for Staging, the actual data flow should be as follows:

Excel (SharePoint) -> Staging Lakehouse -> Warehouse

The error message we received is identical to the one pasted in the original post. It appears that the data successfully reaches the Staging Lakehouse, but the Staging Lakehouse is timing out when moving it to the Warehouse.

Interestingly, when I retried the Gen2 job a few hours after encountering the error, it completed successfully. All the data seemed to be present in the Warehouse. However, when I attempted the job again the next day, I encountered the same error.

Our environment is in the Japan East region. Judging by the time of your post, I assume you're likely in a different region. Therefore, I believe this issue isn't confined to a specific region.

I'm currently seeking the right contact within our organization to reach out to Microsoft.

If others viewing this thread are encountering the same situation, it would be greatly appreciated if you could share the circumstances under which it occurred here.

Re: Gen2 Dataflow failures when loading medium sized datasets

miguel — Tue, 25 Jul 2023 05:01:10 GMT

Hey!

Could you please share a screenshot of the error message that you're seeing inside of the refresh history dialog?

Re: Gen2 Dataflow failures when loading medium sized datasets

ShunOshidari — Tue, 25 Jul 2023 05:20:34 GMT

Thank you for taking a look.

We will share the screenshots of our error screens.

Since the error message is identical to what motoray posted, we believe our situation is similar to motoray's.

Re: Gen2 Dataflow failures when loading medium sized datasets

SidJay — Tue, 25 Jul 2023 05:55:21 GMT

Hi motoray,

If you could please also paste the Request ID for some of your failed dataflow refreshes that would be appreciated.

There are some known ongoing issues with SQL endpoints and we would like to see whether you are impacted by those or encountering separate issues.

Thanks

Re: Gen2 Dataflow failures when loading medium sized datasets

SidJay — Tue, 25 Jul 2023 05:58:04 GMT

Hi ShunOshiadari,

Could you please share the error message you're receving in Refresh History? If you're able to share the Request ID, that would also be helpful to better understand the failure.

Thanks

Re: Gen2 Dataflow failures when loading medium sized datasets

ShunOshidari — Tue, 25 Jul 2023 07:29:59 GMT

I am conducting a post test because my responses keep disappearing within seconds no matter how many times I post them. If this post does not disappear, I will then proceed with my original post.

Re: Gen2 Dataflow failures when loading medium sized datasets

ShunOshidari — Tue, 25 Jul 2023 07:32:10 GMT

My original post (1/2)
---

Thanks SidJay!

The error message is identical to what motoray posted originally, which is:

Re: Gen2 Dataflow failures when loading medium sized datasets

ShunOshidari — Tue, 25 Jul 2023 07:32:57 GMT

My original post (2/2)
---

Some of the Request IDs are:

1f446887-bd7a-48e3-8965-c79217c74965

6cd137a6-19cb-47e6-abe2-21053a7a3038 (different dataflow)

7115718e-662b-472a-873b-9ee100e57f50 (another different dataflow)

Hope these help to resolve the issue!

Re: Gen2 Dataflow failures when loading medium sized datasets

ShunOshidari — Tue, 25 Jul 2023 07:38:42 GMT

Ah I see. If I include the actual error message in my post, my post gets deleted. (I posted it as "My original post (1/2)")

Please refer to the original post of motoray on the top of this page, which contains exactly the same error message I got.

Also you can view some of my screenshots at: https://community.fabric.microsoft.com/t5/Dataflows/Gen2-Dataflow-failures-when-loading-medium-sized-datasets/m-p/3347893#M513

Re: Gen2 Dataflow failures when loading medium sized datasets

motoray — Tue, 25 Jul 2023 14:34:15 GMT

Here are some request IDs:

30ea6d08-cd77-439d-b094-1c3a9c3222ca

a103dcc1-d9cd-4f20-9848-539af2c6be68

cc31824d-9766-4dac-8c66-affda2fe69ce

e8ac09a0-f667-4f88-a3bf-c94a531b2827

9db665b1-3ea9-4ee0-a9c6-ec2133fe5227

d4f414ea-8f21-4bde-8130-9155dcc3f1f2

6751049f-79ff-44ae-9bca-3cab134cbcf6

c7eca806-2a1f-4443-b040-7811607935d9

Re: Gen2 Dataflow failures when loading medium sized datasets

motoray — Tue, 25 Jul 2023 14:35:55 GMT

Here are a couple of screenshots--same message, different dataflows and requests.

Re: Gen2 Dataflow failures when loading medium sized datasets

miguel — Tue, 25 Jul 2023 15:58:44 GMT

thank you for sharing all of this info! We're investigating internally, but please do reach out to our support team so that they can create an incident case and get the support directly from our team.

Use this link to reach the support team: https://support.fabric.microsoft.com/support

I'll update this thread once we have more information to share.

Re: Gen2 Dataflow failures when loading medium sized datasets

motoray — Tue, 25 Jul 2023 16:03:13 GMT

Already there--case # 2307240010003563

Re: Gen2 Dataflow failures when loading medium sized datasets

SidJay — Tue, 25 Jul 2023 17:26:08 GMT

Hi motoray & ShunOshidari,

Thank you for all the information you've provided. The request ids have helped us better identify the symptoms and we are now investigating the root cause. This is not a widespread issue, but some users loading to Lakehouses and Warehouses appear to be getting errors from the SQL endpoint. We are investigating.

Thanks

Re: Gen2 Dataflow failures when loading medium sized datasets

SidJay — Tue, 25 Jul 2023 18:03:07 GMT

Hi motoray & ShunOshidari,

A question to help us with the investigation: Have you modified/deleted the automatically generated staging artifacts that were generated in the workspace? These are the automatically generated Lakehouse, Warehouse, etc. with "Staging" in the name.

Thanks

Re: Gen2 Dataflow failures when loading medium sized datasets

motoray — Tue, 25 Jul 2023 18:29:50 GMT

I have not deleted the staging artifacts. Learned early on that everything breaks when you delete those, so I deleted those workspaces and started fresh and left the staging lakehouses/warehouses alone. Also--I don't put data in the staging areas or use dataflows to pull data from them either. At most, I open and run a query to check and see if the data is loaded to staging.