Forum Discussion
Resolver IIDataflow Gen2 Error: Out of memory: realloc of size [...] failed
To anyone wondering what the solution is: (1) accepting that there are limitations when mixing Gen1 and Gen2 Dataflows. And (2) not mixing Gen1 and Gen2 dataflows as a consequence.
Microsoft EmployeeSince Gen 1 dataflow writes out CSV and Lakehouse expects Parquet, the dataflow engine is buffering all that data and converting it to Parquet, which is quite memory intensive.
Can you share the request ID so I can see how much memory it's taking?
JayJay11Hi pqian_MSFT thank you, the full error message is this:
104100 Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataFormat.Error: Error in replacing table's content with new data in a version: #{0}., Underlying error: Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 156590976 failed') Details: Reason = DataFormat.Error;Message = Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 156590976 failed');Message.Format = Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 156590976 failed');Microsoft.Data.Mashup.Error.Context = User
Request ID = b55b0d6d-89b8-47d3-b698-636ba5791825
Just as a note: I understand that this is not an ideal ETL setup, but I was surprised that I get an error here as well ..
Hi JayJay11
We haven’t heard from you on the last response and was just checking back to see if your query has been resolved. Otherwise, will respond back with the more details and we will try to help.
ThanksHi @JayJay11
We haven’t heard from you on the last response and was just checking back to see if your query has been resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks
- JayJay11
To anyone wondering what the solution is: (1) accepting that there are limitations when mixing Gen1 and Gen2 Dataflows. And (2) not mixing Gen1 and Gen2 dataflows as a consequence.