Forum Discussion

bluelemon64's avatar
bluelemon64
Regular Visitor
2 years ago
Solved

Dataflow Gen 2 Error: ParquetStatusException (message: 'Out of memory: realloc of size 33554432 fai

Hi,   I am trying to use a data flow Gen 2 to merge 3x tables together. All of which come from the same lakehouse. I see each of the 3 tables updates successfully but the Append_writetodatadestinat...
Dataflows
  • bluelemon64's avatar
    bluelemon64
    2 years ago

    Hi pqian_MSFT, appreciate your support. In the end I wrote a PyScript and did the operation in a notebook using my Fabric capacity. Although it took a bit of manual intepretation of the schemas, it's working well. it does however use up most of my capacity due to the table sizes but I think I need to optimize my tables before the operation to optimize.

    Thanks!