Forum Discussion
Gen2 Dataflow failures when loading medium sized datasets
I've been trying to get a medium-sized (100,000 records for now) dataset to load from a SQL database (on prem and Azure) to a Fabric Warehouse from a Gen2 dataflow. I can get it to load 1000 or 10000...
Hi motoray & ShunOshidari,
A question to help us with the investigation: Have you modified/deleted the automatically generated staging artifacts that were generated in the workspace? These are the automatically generated Lakehouse, Warehouse, etc. with "Staging" in the name.
Thanks
Yes, I have previously deleted staging artifacts. After that, when I created a new dataflow item, the artifacts were recreated, so I thought it was fine. However, following your advice, I created a new workspace to test it out and noticed that only DataflowsStagingLakehouse1 was being recreated in the original workspace, while DataflowsStagingWarehouse_xxx remained deleted.
Moving forward, I plan to use the newly created workspace.
However, the issue continues to occur even in the new workspace where no artifacts have been deleted. Like the previous workspace, it sometimes completes successfully and sometimes fails.
These are the Request IDs for an error that occurred in the new workspace. The error message is the same.
ac6411df-22d3-42eb-8f95-5154bab3a21b
5b273482-69b2-4372-9901-bbaa45d71255
What's interesting is that even in cases where the process completes successfully, it takes about 12 to 14 minutes. This seems excessively long for an import process of a 6000-row, 3MB Excel file. When it fails due to an error, it usually fails within about 5 to 7 minutes.
Following miguel's advice, I will also report the same content to https://support.fabric.microsoft.com/support.
Thanks for your support.