Pipeline that works fine on azure data factory dont work on factory

Anonymous — Wed, 03 Jul 2024 10:33:10 GMT

Hi,

I have a pipeline reads an excel file with a table with around 20 rows. each row has parameters to use on a post request.

excel looks like:

pipeline should send this requests ang save json file that get in response.

This works fine when using Azure Data Factory, completing in about a minute or two. However, when running the same flow on Microsoft Fabric, it keeps running indefinitely.

some of the tasks show following error:

-> ErrorCode=FlattenFailedInHierarchicalFlattenStage,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to flatten data, error message: Exception of type 'System.OutOfMemoryException' was thrown.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.OutOfMemoryException,Message=Exception of type 'System.OutOfMemoryException' was thrown.,Source=mscorlib,'

it seems the system runs out of memory, what should i do to prevent that ? when running on azure data fabrick i do use default runtime.

Re: Pipeline that works fine on azure data factory dont work on factory

Anonymous — Thu, 04 Jul 2024 02:59:31 GMT

Hi Anonymous

ForEach activity is used to iterate over a collection and executes specified activities in a loop.

You can try breaking the data into smaller batches to reduce memory usage.

ForEach activity - Microsoft Fabric | Microsoft Learn

Check if you can allocate more memory for the pipe in Microsoft Fabric.

Scale your Fabric capacity - Microsoft Fabric | Microsoft Learn

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

topic Re: Pipeline that works fine on azure data factory dont work on factory in Pipelines

Pipeline that works fine on azure data factory dont work on factory

Re: Pipeline that works fine on azure data factory dont work on factory