Forum Discussion

motoray's avatar
motoray
Icon for Advocate V rankAdvocate V
3 years ago

Gen2 Dataflow failures when loading medium sized datasets

I've been trying to get a medium-sized (100,000 records for now) dataset to load from a SQL database (on prem and Azure) to a Fabric Warehouse from a Gen2 dataflow. I can get it to load 1000 or 10000...
Dataflows
Need Help
SidJay's avatar
SidJay
Icon for Microsoft Employee rankMicrosoft Employee
3 years ago

Hi motoray & ShunOshidari,

 

A question to help us with the investigation: Have you modified/deleted the automatically generated staging artifacts that were generated in the workspace? These are the automatically generated Lakehouse, Warehouse, etc. with "Staging" in the name.

 

Thanks

motoray's avatar
motoray
Icon for Advocate V rankAdvocate V
3 years ago

I have not deleted the staging artifacts. Learned early on that everything breaks when you delete those, so I deleted those workspaces and started fresh and left the staging lakehouses/warehouses alone. Also--I don't put data in the staging areas or use dataflows to pull data from them either. At most, I open and run a query to check and see if the data is loaded to staging.