Forum Discussion

Naveen-004's avatar
Naveen-004
Frequent Visitor
2 years ago
Solved

Notebook fails to write data to a Delta table

Code: df.write.format("delta").mode("overwrite").option("mergeSchema", "true").save(delta_table_path) Error: Py4JJavaError: An error occurred while calling o29386.save. : org.apache.spark.SparkE...
data engineering
lakehouse
notebook
  • richbenmintz's avatar
    richbenmintz
    2 years ago

    Hi Naveen-004,

    Are you able to share all the Code in your Notebook, Spark is a lazily evaluated there could be issues in earlier commands that are only getting executed when you try to write the dataframe.

v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi Naveen-004 

 

Thanks for using Microsoft Fabric Community.

As I understand that you are trying to write data from a DataFrame (df) into a Delta table specified by delta_table_path using PySpark. However, the write operation encountered an error.

 

The error message indicates that the Spark job failed due to a stage failure, specifically task 106 in stage 2171.0. The task failed 4 times, and the most recent failure was due to an executor loss. The error message also mentions that the container from a bad node (vm-0c596022) exited with a status code of 50. This suggests that there was an issue with the Spark executor running on that node.

 

Here are some possible reasons that might be causing the error:

Node failure: The Spark executor node (vm-0c596022) might have failed or become unavailable, causing the task to fail. Look at the Spark UI to see if there are any errors or warnings related to the failed task or executor.

Resource constraints: The node might not have had sufficient resources (e.g., memory, CPU) to execute the task, leading to a failure. Try increasing the resources (e.g., memory, CPU) available to the Spark executor nodes.

Network issues: There might have been network connectivity issues between the Spark driver and the executor node, causing the task to fail. Please try after sometime.

Spark configuration: The Spark configuration might be incorrect or suboptimal, leading to executor losses or task failures. Review the Spark configuration to ensure it is correct and optimal for your workload.

Retry the Write Operation: Sometimes, transient issues can cause task failures. Consider retrying the write operation to see if it succeeds on the second attempt.

 

I hope this information helps.

 

Thank you.

Naveen-004's avatar
Naveen-004
Frequent Visitor
2 years ago

I checked the Spark history and found that one of the stages was skipped, leading to a failure. Can you assist with troubleshooting the error?

 

I don't believe resource constraints are the issue. The computer we are using has 8 Spark driver cores, 56GB Spark driver memory, 8 Spark executor cores, and 56GB Spark executor memory.

Dynamically allocate executors: Enabled
Spark executor instances: 9

Spark Properties:
I have set the first two properties based on below community forum
https://community.fabric.microsoft.com/t5/Data-Engineering/Writing-dataframe-to-delta-table-fails-with-error-Spark/m-p/3833372/highlight/true

  • spark.sql.parquet.datetimeRebaseModeInRead: CORRECTED
  • spark.sql.parquet.datetimeRebaseModeInWrite: CORRECTED
  • spark.sql.parquet.vorder.enabled: true
  • spark.ms.autotune.enabled: false
  • spark.microsoft.delta.optimizeWrite.enabled: true
  • spark.microsoft.delta.merge.lowShuffle.enabled: true

I retried the write operation, but it failed with the same error.

 

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi Naveen-004 

     

    Apologies for the inconvenience.

    Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

    Please go ahead and raise a support ticket to reach our support team:

    https://support.fabric.microsoft.com/support

    After creating a Support ticket please provide the ticket number as it would help us to track for more information.

     

    Thank you.