Forum Discussion
Dataflow Gen 2 Error: ParquetStatusException (message: 'Out of memory: realloc of size 33554432 fai
Hi,
I am trying to use a data flow Gen 2 to merge 3x tables together. All of which come from the same lakehouse. I see each of the 3 tables updates successfully but the Append_writetodatadestination action fails:
TableNameAppend_WriteToDataDestination: Mashup Exception Data Format Error Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataFormat.Error: Error in replacing table's content with new data in a version: #{0}., Underlying error: Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 33554432 failed') Details: Reason = DataFormat.Error;Message = Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 33554432 failed');Message.Format = Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 33554432 failed');Microsoft.Data.Mashup.Error.Context = User
Hi pqian_MSFT, appreciate your support. In the end I wrote a PyScript and did the operation in a notebook using my Fabric capacity. Although it took a bit of manual intepretation of the schemas, it's working well. it does however use up most of my capacity due to the table sizes but I think I need to optimize my tables before the operation to optimize.
Thanks!
5 Replies
Hi bluelemon64
There seems to be someone here who is having the same problem as you, here are some solutions:
Solved: Dataflow Gen2 Error: Out of memory: realloc of siz... - Microsoft Fabric Community
Regards,
Nono Chen
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi Anonymous,
I read that thread however the solution there is not to use data flow gen 1 and gen 2. I'm not using any gen 1 here as far as I understand.
It looks as if pqian_MSFT was investigating a solution but no resolution that I can see from the thread.
Thanks,
Charlie
bluelemon64 what is the input source for your 3 tables? Did you mark these 3 tables as "enable staging" and then build a merge on top of the entities? Can you share your dataflow.json (from export) as well as the dataflow ID?
I think way to go about doing this is ensure the three entities are loaded straight to LH, and then merge them from the LH itself as a separate entity\dataflow.
Hi pqian_MSFT, appreciate your support. In the end I wrote a PyScript and did the operation in a notebook using my Fabric capacity. Although it took a bit of manual intepretation of the schemas, it's working well. it does however use up most of my capacity due to the table sizes but I think I need to optimize my tables before the operation to optimize.
Thanks!
Hi bluelemon64
pqian_MSFT Thank you very much for your reply.
Can you tell me if your problem is solved? If yes, please accept it as solution.
Regards,
Nono Chen
