Forum Discussion
Gen2 Dataflow failures when loading medium sized datasets
Hi!
It appears that I'm encountering a similar problem, which relates to the number of rows in a file. I can successfully load files with over 100K rows from Azure ALDS Gen2 Storage to a Lakehouse. However, when attempting to transfer files between two Lakehouses, only the files with more than 100K rows encounter failure, while files with fewer rows transfer without any issues.
Now none of my Gen2 dataflows will complete so I'm not sure what's going on anymore. Time to call in Microsoft.
And as of right now, I still can't get any Gen2 dataflows to complete--not since last Thursday around 1 pm PST. Something is hard broken now. Have built new workspaces, lakehouse, dataflows...nothing works. The refreshes simply spin on forever now.
Hey!
Could you please share a screenshot of the error message that you're seeing inside of the refresh history dialog?
