Forum Discussion

Naveen-004's avatar
Naveen-004
Frequent Visitor
2 years ago
Solved

Notebook fails to write data to a Delta table

Code: df.write.format("delta").mode("overwrite").option("mergeSchema", "true").save(delta_table_path) Error: Py4JJavaError: An error occurred while calling o29386.save. : org.apache.spark.SparkE...
data engineering
lakehouse
notebook
  • richbenmintz's avatar
    richbenmintz
    2 years ago

    Hi Naveen-004,

    Are you able to share all the Code in your Notebook, Spark is a lazily evaluated there could be issues in earlier commands that are only getting executed when you try to write the dataframe.

Marienklin's avatar
Marienklin
New Member
2 months ago

I ran into a very similar issue a while back where the write operation kept failing with an ExecutorLostFailure error even though the cluster resources looked sufficient. In my case, the root cause ended up being a problem on one of the worker nodes rather than the Delta write itself.

One thing that helped was checking the logs of delta for the task and reviewing the Spark UI to see whether the same executor or node was failing repeatedly. It may also be worth testing with a smaller dataset to determine whether the issue is data-related or infrastructure-related.