SystemErrorOutOfMemory - Pipeline

[Dhanapal]

Hello, Recent two days, we are facing SystemErrorOutOfMemory, there is nothing changed in the pipeline and we have 3 session as parallel run as originally designed.. But still we are getting error message from the copy data activities.. 

We have F64 capacity for the Fabric.. Please help us on this issue

ErrorCode=SystemErrorOutOfMemory,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A task failed with out of memory.,Source=Microsoft.DataTransfer.TransferTask,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An unknown error occurred.,Source=Microsoft.DataTransfer.Common,''Type=System.InsufficientMemoryException,Message=Insufficient available memory to meet the expected demands of an operation at this time. Please try again later.,Source=mscorlib,'

[rajendraongole1]

Hi @Dhanapal  There are few troubleshooting steps and suggestions to help resolve the issue:

Check Data Volume and Parallelism: If the data volume has grown over time or if each session processes large datasets, it might be putting more load on the memory than anticipated. Try reducing the parallel sessions temporarily to see if it alleviates the issue. Adjusting the degree of parallelism can sometimes balance memory usage more efficiently.

Optimize Data Transformation Logic: Review the transformation logic in the pipeline, especially in the Copy Data activities. Complex transformations or operations with high memory requirements (such as joins or aggregations) can be optimized. You may also split the workload into smaller activities if possible.

Adjust the Capacity Size or Move to a Dedicated Capacity: If the current capacity cannot handle the workload, consider upgrading to a larger SKU or a dedicated capacity tier to increase the memory and CPU resources available.

Enable Dataflow Staging: For large data transformations, enabling dataflow staging in Fabric can offload some intermediate steps to a staging environment, reducing memory demand on your primary capacity.

Review Fabric Settings and Usage: Fabric provides diagnostic information on memory and CPU utilization. Use this data to understand how much memory is being used per activity and to identify potential bottlenecks.