Forum Discussion

JayJay11's avatar
JayJay11
Icon for Resolver II rankResolver II
2 years ago
Solved

Dataflow Gen2 Error: Out of memory: realloc of size [...] failed

Hello all,   I have the following case: A Gen2 Dataflow picks up data from a Gen1 Dataflow and shall load it to a Lakehouse. Then I get the following error after the dataflow has run for a while: ...
Dataflows
  • JayJay11's avatar
    JayJay11
    2 years ago

    To anyone wondering what the solution is: (1) accepting that there are limitations when mixing Gen1 and Gen2 Dataflows. And (2) not mixing Gen1 and Gen2 dataflows as a consequence.

pqian_MSFT's avatar
pqian_MSFT
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Thanks, looks like it wants to write out 100k rows and needs 4GB of commit to do so. The cloud engine today allows for 6GB of commit, the parquet write and the CSV buffering together ate all of it.

 

We'll need to investigate what would it take to make this work on the cloud. It's a valid scenario and should just work out of the box, unfortunately our current limitations makes this a little challenging, but I'm pretty confident that this can be solved.

 

Meanwhile, trying this sceanrio on a Gateway where there is no commit limit should get you unblocked. But I guess you are having difficulties with your GW server as well?