Forum Discussion

Dietz's avatar
Dietz
New Member
1 year ago
Solved

Fabric Notebook Error, please help!

Hi There

 

Would really appreciate any assistance.

Will try keep this short.... I have a notebook in Fabric, at the end of the notebook a try to write the dataframe to a delta a table.

With this code.

 

matched_customers_df.write.format("delta").option("overwriteSchema", "true").mode("overwrite").saveAsTable("Lakehouse.gold_table_final")
 
This dataframe has 5.1 million rows.
 
But I get the following error.
 
org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task 123:0 was 156335848 bytes, which exceeds max allowed: spark.rpc.message.maxSize (134217728 bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values.
 
I have searched for answer with ChatGPT and Copilot, and they both reccomend chaning the spark seting with either, 
spark.conf.set("spark.rpc.message.maxSize", "200MB") spark.conf.set("spark.driver.maxResultSize", "2g")
 
But you cannot execute this statement in Fabric notebooks. As you are not allowed to change the spark settings directly in Fabric.
I have also tried to parition the data and the delta table but they all get the same error.
 
I also cannot change these setting in the Fabric workspace spark configuration.
 
As a last resort I increased the Fabric capacity from F8 to F16 and then F32, still the same error.
Please help myself and chatGPT are out of ideas.
Data Integration
  • FabianSchut's avatar
    FabianSchut
    1 year ago

    Hi, can you create a custom environment in Fabric and change the spark settings there? You can change the settings in the spark properties field as shown in the screenshot. You can select that custom environment in your notebook.

     

5 Replies

  • FabianSchut's avatar
    FabianSchut
    Solution Sage
    1 year ago

    Hi, can you create a custom environment in Fabric and change the spark settings there? You can change the settings in the spark properties field as shown in the screenshot. You can select that custom environment in your notebook.

     

    • Dietz's avatar
      Dietz
      New Member
      1 year ago

      Thansk FabianSchut, really appreciaet your help, but I cannot seem to get to the screen in yoru message, I have created a custom environment and gone to eedit but I dont see teh screen that you posted?

  • Dietz's avatar
    Dietz
    New Member
    1 year ago

    or anyone who wants to know the complete solution , I created a custom environemnt and set the node sice to large, and teh issue was resolved.