Skip to main content
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Did you hear? There's a new SQL AI Developer certification (DP-800). Start preparing now and be one of the first to get certified. Register now

Reply
mhseko
Frequent Visitor

Foreach

‎02-19-2025 09:05 AM

Scenario:

Ingesting 42 tables through Data Gateway in a Foreach pipeline activity.  Sequential is off and Batch Count is 10.  Some tables are much larger than others and take hours to pull the data.  Gateway is balanced across two servers.  Both online and updated to version 3000.254.6.  

 

Expectation: Would expect to see at minimum 10 items processing at a time until we are close to the end.  Only see two running currently.  Even though a couple of tables are taking an inordinate amount of time, the others items should start processing.  I see two items In Progress within the pipeline and two respective FabricWorker.exe running on the data gateway servers.  

 

Why doesn't the data pipeline queue up all requests and process as many as possible concurrently?

 

 

Labels:
Message 1 of 2
730 Views
0
1 REPLY 1
lbendlin
Super User
Super User

‎02-19-2025 05:02 PM

How many mashup containers did you configure on the gateway VMs?  How much memory?

Message 2 of 2
704 Views
0

Helpful resources

Announcements
April Fabric Update Carousel

Fabric Monthly Update - April 2026

Check out the April 2026 Fabric update to learn about new features.

Fabric SQL PBI Data Days

Data Days 2026 coming soon!

Sign up to receive a private message when registration opens and key events begin.

New to Fabric survey Carousel

New to Fabric Survey

If you have recently started exploring Fabric, we'd love to hear how it's going. Your feedback can help with product improvements.

View All