Forum Discussion
Out of Memory Error in Dataflow Gen2 When Reading Parquet File - F64 Capacity
Hi rubayatyasmin , Thank you for reaching out to the Microsoft Community Forum.
tayloramy correctly suggested to open a Fabric notebook and try reading the same Parquet file directly using: df = spark.read.parquet("/lakehouse/Files/yourpath/yourfile.parquet") df.limit(1).show()
If that fails, it does confirm the Parquet file itself is likely corrupted, which can trigger the “Out of memory: realloc failed” error even for small files. This test helps isolate whether the problem lies in the file or in the Dataflow Gen2 service.
Might I add that if your dataflow reads a Parquet file stored in OneLake (as most Fabric Dataflows do), the on-premises gateway won’t be involved at all. The read operation happens entirely within Fabric’s managed compute, so gateway memory isn’t a factor. Given that the problem started after an internal outage, it’s possible that the Parquet file or its metadata footer became partially corrupted. If the notebook test fails, regenerate or re-export that file; if it succeeds but the Dataflow still errors, open a Microsoft Support ticket sharing them all the details.
For additional confirmation, you can delete and re-upload the file to OneLake, then re-run the refresh. These steps usually clear any residual corruption from interrupted service operations.
Below is the link to help create Microsoft Support ticket:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Please check below doc for more information:
Differences between Dataflow Gen1 and Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
Ingest Data with Dataflows in Microsoft Fabric - Training | Microsoft Learn
Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn
Load data into your lakehouse with a notebook - Microsoft Fabric | Microsoft Learn
Thank you tayloramy for your valuable response.
Hi, I created support ticket already, but it's not that helpful so far.
Hi rubayatyasmin , I’m sorry to hear that. Although I’m not a member of the PG team, I am confident that they are working diligently to resolve it as soon as possible. Your patience and understanding are greatly appreciated. Please share any updates or insights you receive here, as this information could be helpful to others experiencing similar challenges.
Thank you.