Forum Discussion
Dataflow runs since hours, cannot be stopped?
I looked into a few runs for your Gen 1 and Gen 2 refresh - the main difference is the time span in OData/GetResponse. There are thousands of these OData requests, they average to 0.5s each on the Gen 1 nodes:
However, the on the Gen 2 nodes, they average around 3.79s each
We see the network throughput on Gen 2 node at around 100kbps, and the entire refresh is network IO bound.
Both the Gen 1 and Gen 2 nodes are from our Germany data center, and so there shouldn't be any latency differences.
Is the SAP server perhaps throttling the Gen 2 refresh for whatever reason?
Thank you for checking. The Gen1 and Gen2 dataflows have the exact same queries. Only the Gen2 Dataflow has destinations in a Lakehouse. They both use the exact same data sources in the same Gateway.
I just tried re-running one of the Gen1 flows, no issues. However, at this time I cannot even load the smallest table anymore to a Lakehouse using a Gen2 flow with the same queries, see here:
I re-created the Gen2 flows many times already, tried a different Lakehouse, installed the newest version of the Data Gateway .. nothing is helping. I don't know what we can do at this point...