Forum Discussion
Gen2 Dataflow failures when loading medium sized datasets
I've been trying to get a medium-sized (100,000 records for now) dataset to load from a SQL database (on prem and Azure) to a Fabric Warehouse from a Gen2 dataflow. I can get it to load 1000 or 10000...
I'm glad I came across your post. We're experiencing a similar issue in our environment. We've recently begun to evaluate Fabric, and we were unsure if this issue had been occurring from the start.
Here's our situation:
Dataflow: Gen2
Source: SharePoint folder (Excel file, 3MB, 6000 rows)
Destination: Warehouse
Capacity region: Japan East
With this Excel file, we managed to successfully import up to approximately 4600 rows. I believe the actual limit might also depend on the number of columns.
As Gen2 uses Lakehouse for Staging, the actual data flow should be as follows:
Excel (SharePoint) -> Staging Lakehouse -> Warehouse
The error message we received is identical to the one pasted in the original post. It appears that the data successfully reaches the Staging Lakehouse, but the Staging Lakehouse is timing out when moving it to the Warehouse.
Interestingly, when I retried the Gen2 job a few hours after encountering the error, it completed successfully. All the data seemed to be present in the Warehouse. However, when I attempted the job again the next day, I encountered the same error.
Our environment is in the Japan East region. Judging by the time of your post, I assume you're likely in a different region. Therefore, I believe this issue isn't confined to a specific region.
I'm currently seeking the right contact within our organization to reach out to Microsoft.
If others viewing this thread are encountering the same situation, it would be greatly appreciated if you could share the circumstances under which it occurred here.
Hi ShunOshiadari,
Could you please share the error message you're receving in Refresh History? If you're able to share the Request ID, that would also be helpful to better understand the failure.
Thanks