Forum Discussion

bluelemon64's avatar
bluelemon64
Regular Visitor
2 years ago
Solved

Dataflow Gen 2 Error: ParquetStatusException (message: 'Out of memory: realloc of size 33554432 fai

Hi,   I am trying to use a data flow Gen 2 to merge 3x tables together. All of which come from the same lakehouse. I see each of the 3 tables updates successfully but the Append_writetodatadestinat...
Dataflows
  • bluelemon64's avatar
    bluelemon64
    2 years ago

    Hi pqian_MSFT, appreciate your support. In the end I wrote a PyScript and did the operation in a notebook using my Fabric capacity. Although it took a bit of manual intepretation of the schemas, it's working well. it does however use up most of my capacity due to the table sizes but I think I need to optimize my tables before the operation to optimize.

    Thanks!

pqian_MSFT's avatar
pqian_MSFT
Microsoft Employee
2 years ago

bluelemon64 what is the input source for your 3 tables? Did you mark these 3 tables as "enable staging" and then build a merge on top of the entities? Can you share your dataflow.json (from export) as well as the dataflow ID?

 

I think way to go about doing this is ensure the three entities are loaded straight to LH, and then merge them from the LH itself as a separate entity\dataflow.

bluelemon64's avatar
bluelemon64
Regular Visitor
2 years ago

Hi pqian_MSFT, appreciate your support. In the end I wrote a PyScript and did the operation in a notebook using my Fabric capacity. Although it took a bit of manual intepretation of the schemas, it's working well. it does however use up most of my capacity due to the table sizes but I think I need to optimize my tables before the operation to optimize.

Thanks!