Scenario:
Ingesting 42 tables through Data Gateway in a Foreach pipeline activity. Sequential is off and Batch Count is 10. Some tables are much larger than others and take hours to pull the data. Gateway is balanced across two servers. Both online and updated to version 3000.254.6.
Expectation: Would expect to see at minimum 10 items processing at a time until we are close to the end. Only see two running currently. Even though a couple of tables are taking an inordinate amount of time, the others items should start processing. I see two items In Progress within the pipeline and two respective FabricWorker.exe running on the data gateway servers.
Why doesn't the data pipeline queue up all requests and process as many as possible concurrently?
