Dataflow Gen2 Error: Out of memory: realloc of size [...] failed

[JayJay11]

Hello all,

I have the following case: A Gen2 Dataflow picks up data from a Gen1 Dataflow and shall load it to a Lakehouse. Then I get the following error after the dataflow has run for a while:

104100 Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataFormat.Error: Error in replacing table's content with new data in a version: #{0}., Underlying error: Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 156590976 failed') Details: Reason = DataFormat.Error;Message = Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 156590976 failed');Message.Format = Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 156590976 failed');Microsoft.Data.Mashup.Error.Context = User

I don't get what the error is trying to tell me.. I can load this table without problems to a semantic model within a couple of minutes.

[pqian_MSFT]

Thanks, looks like it wants to write out 100k rows and needs 4GB of commit to do so. The cloud engine today allows for 6GB of commit, the parquet write and the CSV buffering together ate all of it.

We'll need to investigate what would it take to make this work on the cloud. It's a valid scenario and should just work out of the box, unfortunately our current limitations makes this a little challenging, but I'm pretty confident that this can be solved.

Meanwhile, trying this sceanrio on a Gateway where there is no commit limit should get you unblocked. But I guess you are having difficulties with your GW server as well?

[pqian_MSFT]

Since Gen 1 dataflow writes out CSV and Lakehouse expects Parquet, the dataflow engine is buffering all that data and converting it to Parquet, which is quite memory intensive.

Can you share the request ID so I can see how much memory it's taking?

[JayJay11]

Hi @pqian_MSFT thank you, the full error message is this:

104100 Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataFormat.Error: Error in replacing table's content with new data in a version: #{0}., Underlying error: Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 156590976 failed') Details: Reason = DataFormat.Error;Message = Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 156590976 failed');Message.Format = Parquet: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 156590976 failed');Microsoft.Data.Mashup.Error.Context = User

Request ID = b55b0d6d-89b8-47d3-b698-636ba5791825

Just as a note: I understand that this is not an ideal ETL setup, but I was surprised that I get an error here as well .. 

[JayJay11 · Accepted solution]

To anyone wondering what the solution is: (1) accepting that there are limitations when mixing Gen1 and Gen2 Dataflows. And (2) not mixing Gen1 and Gen2 dataflows as a consequence.

[v-nikhilan-msft]

Hi @JayJay11 
We haven’t heard from you on the last response and was just checking back to see if your query has been resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

[v-nikhilan-msft]

Hi @JayJay11 
We haven’t heard from you on the last response and was just checking back to see if your query has been resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks