topic Re: Out of memory Error While Copying Data into Lakehouse Table Partitioned in Pipelines

Out of memory Error While Copying Data into Lakehouse Table Partitioned

v-cyu — Tue, 28 May 2024 22:37:12 GMT

We are currently trying to load data into Lakehouse Table using copy activity and are trying to partition them by a column named "DateId". However, we have encounted the below error and need help understanding it.

What is the below error telling exactly?
Any solution that would allow us to copy data partitioned?

ErrorCode=ParquetErroneousStatus,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 76461504 failed'),Source=Microsoft.DataTransfer.Common,'

Re: Out of memory Error While Copying Data into Lakehouse Table Partitioned

Anonymous — Wed, 29 May 2024 15:11:58 GMT

Hi v-cyu
Thanks for using Fabric Community.

The error message 'Out of memory: realloc of size 76461504 failed' indicates that the system ran out of memory while trying to allocate additional memory for the operation. This error is often encountered when dealing with large datasets or complex operations that require a significant amount of memory.

In your case, it seems like the error occurred while trying to partition the data by the “DateId” column and load it into a Lakehouse Table using the copy activity. The operation might be memory-intensive, especially if the dataset is large.

A similar thread for reference: Solved: Dataflow Gen2 Error: Out of memory: realloc of siz... - Microsoft Fabric Community

Here are a few potential solutions:

Increase the available memory: If you’re running this operation on a machine with limited memory, you might consider increasing the available memory or using a machine with more memory. In the context of Microsoft Fabric, increasing the memory often refers to scaling up your Fabric capacity. This involves allocating more resources to your Fabric service, which can help handle larger datasets and more complex operations
Evaluate and optimize your Microsoft Fabric capacity - Microsoft Fabric | Microsoft Learn
Scale your Fabric capacity - Microsoft Fabric | Microsoft Learn

Optimize your data processing: Try to optimize your data processing to use less memory. This could involve filtering unnecessary data early in the process, using more efficient data structures, or breaking the operation into smaller, more manageable tasks.

Hope this helps. Please let me know if there are any further questions.

Re: Out of memory Error While Copying Data into Lakehouse Table Partitioned

Anonymous — Thu, 30 May 2024 16:28:58 GMT

Hi v-cyu
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

Re: Out of memory Error While Copying Data into Lakehouse Table Partitioned

Anonymous — Mon, 03 Jun 2024 15:59:02 GMT

Hi @v-cyu
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks