https://community.fabric.microsoft.com/t5/Pipelines/Foreach/m-p/4418138#M7115 <P>Scenario:</P><P>Ingesting 42 tables through Data Gateway in a Foreach pipeline activity.&nbsp; Sequential is off and Batch Count is 10.&nbsp; Some tables are much larger than others and take hours to pull the data.&nbsp; Gateway is balanced across two servers.&nbsp; Both online and updated to version 3000.254.6.&nbsp;&nbsp;</P><P>&nbsp;</P><P>Expectation: Would expect to see at minimum 10 items processing at a time until we are close to the end.&nbsp; Only see two running currently.&nbsp; Even though a couple of tables are taking an inordinate amount of time, the others items should start processing.&nbsp; I see two items In Progress within the pipeline and two respective FabricWorker.exe running on the data gateway servers.&nbsp;&nbsp;</P><P>&nbsp;</P><P>Why doesn't the data pipeline queue up all requests and process as many as possible concurrently?</P><P>&nbsp;</P><P>&nbsp;</P> Wed, 19 Feb 2025 17:05:37 GMT mhseko 2025-02-19T17:05:37Z https://community.fabric.microsoft.com/t5/Pipelines/Foreach/m-p/4418138#M7115 <P>Scenario:</P><P>Ingesting 42 tables through Data Gateway in a Foreach pipeline activity.&nbsp; Sequential is off and Batch Count is 10.&nbsp; Some tables are much larger than others and take hours to pull the data.&nbsp; Gateway is balanced across two servers.&nbsp; Both online and updated to version 3000.254.6.&nbsp;&nbsp;</P><P>&nbsp;</P><P>Expectation: Would expect to see at minimum 10 items processing at a time until we are close to the end.&nbsp; Only see two running currently.&nbsp; Even though a couple of tables are taking an inordinate amount of time, the others items should start processing.&nbsp; I see two items In Progress within the pipeline and two respective FabricWorker.exe running on the data gateway servers.&nbsp;&nbsp;</P><P>&nbsp;</P><P>Why doesn't the data pipeline queue up all requests and process as many as possible concurrently?</P><P>&nbsp;</P><P>&nbsp;</P> Wed, 19 Feb 2025 17:05:37 GMT https://community.fabric.microsoft.com/t5/Pipelines/Foreach/m-p/4418138#M7115 mhseko 2025-02-19T17:05:37Z https://community.fabric.microsoft.com/t5/Pipelines/Foreach/m-p/4418527#M7117 <P>How many mashup containers did you configure on the gateway VMs?&nbsp; How much memory?</P> Thu, 20 Feb 2025 01:02:09 GMT https://community.fabric.microsoft.com/t5/Pipelines/Foreach/m-p/4418527#M7117 lbendlin 2025-02-20T01:02:09Z