Forum Discussion
Out of Memory Error in Dataflow Gen2 When Reading Parquet File - F64 Capacity
Hi rubayatyasmin,
What's your data source? Are you using an on prem gateway at all?
I've seen gateway memory errors bubble up in unexpected ways before, if you're using a gateway can you check on the memory usage on the gateway server?
Are you able to read from the specific parquet file that's causing the issue? In a new notebook try
df = spark.read.parquet("/lakehouse/Files/yourpath/yourfile.parquet")
df.limit(1).show()
If this crashes, then the parquet file is likely bad and needs to be regenerated.
If you found this helpful, cosnider giving Kudos. If I solved your problem or answered your question, mark this post as a solution.
Hi,
I’m not using an on-premises gateway. My files are stored in a Lakehouse. I considered using incremental load, but since my query isn’t folding, that option is ruled out for now.
All files are fine — when I load smaller chunks, the dataflow refreshes successfully. Currently, I have 39 files, and if I select any 36 of them, the dataflow works without issue. However, selecting more than that causes an out-of-memory error.
However, I am trying to use notebook and pipeline to load the tables. Failing so far. Need some more RnD