Forum Discussion
Gen2 Dataflow failures when loading medium sized datasets
So have been doing some more testing (things magically started working again last week) and am starting to think there might be a max # of times per [time period] that I can load data to the warehouse. I tried with a dataset with 85000 rows and got a failure--host closed the connection--but then tried an hour later and it worked. When I hit the 85000 mark, I'd been gradually ramping up from 1000, so 4-5 runs in the space of under an hour. Still testing to see if this is the real issue, but would be great if someone from Microsoft could let me know whether there is indeed some kind of throttling going on that prevents repreated loads into the same table...
Hi motoray,
This is unexpected behavior with the SQL endpoint and we are still actively debugging the issue. The issue (connection forcibly closed by remote host) is sporadic and there isn't a specific limit or throttle that you're running into.
Thanks
Thanks for working with me on this. It was really good to hear from you and have the issue confirmed yesterday.
Things seem to have stabilized for me now and I'm able to successfully load full datasets from onprem SQL directly to a Warehouse or Lakehouse. The one thing I'm still seeing--and just noticed this yesterday--is that if I have a brand new warehouse/lakehouse and try do a load to a new table with a Gen2 dataflow, it will fail if I attempt to load the full dataset the first time. If I load 100 rows, it succeeds and then I'm able to pull the filter and load everything.
