Forum Discussion

bluelemon64's avatar
bluelemon64
Regular Visitor
2 years ago
Solved

Dataflow Gen 2 Error: ParquetStatusException (message: 'Out of memory: realloc of size 33554432 fai

Hi,   I am trying to use a data flow Gen 2 to merge 3x tables together. All of which come from the same lakehouse. I see each of the 3 tables updates successfully but the Append_writetodatadestinat...
Dataflows
  • bluelemon64's avatar
    bluelemon64
    2 years ago

    Hi pqian_MSFT, appreciate your support. In the end I wrote a PyScript and did the operation in a notebook using my Fabric capacity. Although it took a bit of manual intepretation of the schemas, it's working well. it does however use up most of my capacity due to the table sizes but I think I need to optimize my tables before the operation to optimize.

    Thanks!

bluelemon64's avatar
bluelemon64
Regular Visitor
2 years ago

Hi Anonymous,

 

I read that thread however the solution there is not to use data flow gen 1 and gen 2. I'm not using any gen 1 here as far as I understand. 

 

It looks as if pqian_MSFT was investigating a solution but no resolution that I can see from the thread.

 

Thanks,

Charlie