Forum Discussion

motoray's avatar
motoray
Advocate V
3 years ago

Gen2 Dataflow failures when loading medium sized datasets

I've been trying to get a medium-sized (100,000 records for now) dataset to load from a SQL database (on prem and Azure) to a Fabric Warehouse from a Gen2 dataflow. I can get it to load 1000 or 10000...
Dataflows
Need Help
ShunOshidari's avatar
ShunOshidari
Helper I
3 years ago
I'm glad I came across your post. We're experiencing a similar issue in our environment. We've recently begun to evaluate Fabric, and we were unsure if this issue had been occurring from the start.

Here's our situation:

Dataflow: Gen2
Source: SharePoint folder (Excel file, 3MB, 6000 rows)
Destination: Warehouse
Capacity region: Japan East

With this Excel file, we managed to successfully import up to approximately 4600 rows. I believe the actual limit might also depend on the number of columns.

As Gen2 uses Lakehouse for Staging, the actual data flow should be as follows:
Excel (SharePoint) -> Staging Lakehouse -> Warehouse

The error message we received is identical to the one pasted in the original post. It appears that the data successfully reaches the Staging Lakehouse, but the Staging Lakehouse is timing out when moving it to the Warehouse.

Interestingly, when I retried the Gen2 job a few hours after encountering the error, it completed successfully. All the data seemed to be present in the Warehouse. However, when I attempted the job again the next day, I encountered the same error.

Our environment is in the Japan East region. Judging by the time of your post, I assume you're likely in a different region. Therefore, I believe this issue isn't confined to a specific region.

I'm currently seeking the right contact within our organization to reach out to Microsoft.

If others viewing this thread are encountering the same situation, it would be greatly appreciated if you could share the circumstances under which it occurred here.
SidJay's avatar
SidJay
Microsoft Employee
3 years ago

Hi ShunOshiadari,

 

Could you please share the error message you're receving in Refresh History? If you're able to share the Request ID, that would also be helpful to better understand the failure.

 

Thanks

  • ShunOshidari's avatar
    ShunOshidari
    Helper I
    3 years ago

    I am conducting a post test because my responses keep disappearing within seconds no matter how many times I post them. If this post does not disappear, I will then proceed with my original post.

    • miguel's avatar
      miguel
      Community Admin
      3 years ago

      thank you for sharing all of this info! We're investigating internally, but please do reach out to our support team so that they can create an incident case and get the support directly from our team.

       

      Use this link to reach the support team: https://support.fabric.microsoft.com/support

       

      I'll update this thread once we have more information to share.

      • motoray's avatar
        motoray
        Advocate V
        3 years ago

        Already there--case # 2307240010003563

  • ShunOshidari's avatar
    ShunOshidari
    Helper I
    3 years ago

    My original post (1/2)
    ---

    Thanks SidJay!

    The error message is identical to what motoray posted originally, which is:

    Mashup Exception Data Source Error Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Microsoft SQL: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) Details: DataSourceKind = Lakehouse;DataSourcePath = Lakehouse;Message = A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.);ErrorCode = -2146232060;Number = 10054;Class = 20
  • ShunOshidari's avatar
    ShunOshidari
    Helper I
    3 years ago

    My original post (2/2)
    ---

    Some of the Request IDs are:
    1f446887-bd7a-48e3-8965-c79217c74965
    6cd137a6-19cb-47e6-abe2-21053a7a3038 (different dataflow)
    7115718e-662b-472a-873b-9ee100e57f50 (another different dataflow)

    Hope these help to resolve the issue!