Dataflow no longer working due to memory issue

Hi Anonymous ,

For the memory issue, Lakehouse requires Parquet if the configured destination is lakehouse, so the dataflow engine buffers all this data and converts it to Parquet, which is quite memory intensive.

Monitor the CPU, memory, and network usage of the dataflow job to identify any potential bottlenecks. This can help you understand if the dataflow is running out of memory due to resource constraints.

In addition, proper use of staging can optimize the performance of processing, refer to the following documentation.

Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn

An overview of refresh history and monitoring for dataflows. - Microsoft Fabric | Microsoft Learn

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.