Out of Memory Error in Dataflow Gen2 When Reading Parquet File - F64 Capacity

[rubayatyasmin]

I am encountering an Out of Memory error while refreshing a Dataflow Gen2 that simply reads a Parquet file. The error message is as follows:

MashupException.Error: DataFormat.Error: Parquet.Document: class parquet::ParquetStatusException (message: 'Out of memory: realloc of size 4194304 failed')

This issue started occurring recently after an internal outage two weeks ago. Prior to that, the dataflow worked fine for 7-8 months without any issues.

Details:

• File Size: The Parquet file size is around 1MB per week, so it's relatively small.

• Fabric Capacity: We are using F64 capacity, and I’ve verified that it’s not hitting the limits during the refresh.

• Dataflow: The original dataflow is not CI/CD-based; it's just a Dataflow Gen2 reading a Parquet file without any transformations.

• Staging: I have staging disabled, and still encountering the same error.

• Issue Timeline: The issue began after an internal outage two weeks ago, during which the refreshes failed. Prior to that, it had been working fine for several months.

What I Have Tried:

1. Created a New Dataflow: I created a new Dataflow Gen2 to test if the issue persists, but I’m still encountering the same memory error.

2. Verified Capacity: I checked the capacity metrics, and F64 should be sufficient, but I still see the error.

3. Disabled Staging: Staging was already disabled, so no changes were made there.

4. Checked File Size: The Parquet file size is small (1MB per week), so this shouldn't be a large data issue.