Forum Discussion

motoray's avatar
motoray
Advocate V
3 years ago

Gen2 Dataflow failures when loading medium sized datasets

I've been trying to get a medium-sized (100,000 records for now) dataset to load from a SQL database (on prem and Azure) to a Fabric Warehouse from a Gen2 dataflow. I can get it to load 1000 or 10000...
Dataflows
Need Help
SidJay's avatar
SidJay
Microsoft Employee
3 years ago

Hi motoray & ShunOshidari,

 

A question to help us with the investigation: Have you modified/deleted the automatically generated staging artifacts that were generated in the workspace? These are the automatically generated Lakehouse, Warehouse, etc. with "Staging" in the name.

 

Thanks

  • motoray's avatar
    motoray
    Advocate V
    3 years ago

    I have not deleted the staging artifacts. Learned early on that everything breaks when you delete those, so I deleted those workspaces and started fresh and left the staging lakehouses/warehouses alone. Also--I don't put data in the staging areas or use dataflows to pull data from them either. At most, I open and run a query to check and see if the data is loaded to staging.

  • ShunOshidari's avatar
    ShunOshidari
    Helper I
    3 years ago
    Yes, I have previously deleted staging artifacts. After that, when I created a new dataflow item, the artifacts were recreated, so I thought it was fine. However, following your advice, I created a new workspace to test it out and noticed that only DataflowsStagingLakehouse1 was being recreated in the original workspace, while DataflowsStagingWarehouse_xxx remained deleted.

    Moving forward, I plan to use the newly created workspace.

    However, the issue continues to occur even in the new workspace where no artifacts have been deleted. Like the previous workspace, it sometimes completes successfully and sometimes fails.

    These are the Request IDs for an error that occurred in the new workspace. The error message is the same.
    ac6411df-22d3-42eb-8f95-5154bab3a21b
    5b273482-69b2-4372-9901-bbaa45d71255

    What's interesting is that even in cases where the process completes successfully, it takes about 12 to 14 minutes. This seems excessively long for an import process of a 6000-row, 3MB Excel file. When it fails due to an error, it usually fails within about 5 to 7 minutes.

    Following miguel's advice, I will also report the same content to https://support.fabric.microsoft.com/support.

    Thanks for your support.