Pipeline that works fine on azure data factory dont work on factory

[NunoTrigo1986]

Hi,

I have a pipeline reads an excel file with a table with around 20 rows. each row has parameters to use on a post request. 

excel looks like:

pipeline should send this requests ang save json file that get in response. 

This works fine when using Azure Data Factory, completing in about a minute or two. However, when running the same flow on Microsoft Fabric, it keeps running indefinitely. 

some of the tasks show following error:

-> ErrorCode=FlattenFailedInHierarchicalFlattenStage,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to flatten data, error message: Exception of type 'System.OutOfMemoryException' was thrown.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.OutOfMemoryException,Message=Exception of type 'System.OutOfMemoryException' was thrown.,Source=mscorlib,'

it seems the system runs out of memory, what should i do to prevent that ? when running on azure data fabrick i do use default runtime. 

[v-nuoc-msft · Accepted solution]

Hi @NunoTrigo1986 

ForEach activity is used to iterate over a collection and executes specified activities in a loop.

You can try breaking the data into smaller batches to reduce memory usage.

ForEach activity - Microsoft Fabric | Microsoft Learn

Check if you can allocate more memory for the pipe in Microsoft Fabric.

Scale your Fabric capacity - Microsoft Fabric | Microsoft Learn

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.