Fabric Notebook Error, please help!

[Dietz]

Hi There

Would really appreciate any assistance.

Will try keep this short.... I have a notebook in Fabric, at the end of the notebook a try to write the dataframe to a delta a table.

With this code.

matched_customers_df.write.format("delta").option("overwriteSchema", "true").mode("overwrite").saveAsTable("Lakehouse.gold_table_final")

This dataframe has 5.1 million rows.

But I get the following error.

org.apache.spark.SparkException: Job aborted due to stage failure: Serialized task 123:0 was 156335848 bytes, which exceeds max allowed: spark.rpc.message.maxSize (134217728 bytes). Consider increasing spark.rpc.message.maxSize or using broadcast variables for large values.

I have searched for answer with ChatGPT and Copilot, and they both reccomend chaning the spark seting with either, 

spark.conf.set("spark.rpc.message.maxSize", "200MB") spark.conf.set("spark.driver.maxResultSize", "2g")

But you cannot execute this statement in Fabric notebooks. As you are not allowed to change the spark settings directly in Fabric.

I have also tried to parition the data and the delta table but they all get the same error.

I also cannot change these setting in the Fabric workspace spark configuration.

As a last resort I increased the Fabric capacity from F8 to F16 and then F32, still the same error.

Please help myself and chatGPT are out of ideas.

[FabianSchut · Accepted solution]

Hi, can you create a custom environment in Fabric and change the spark settings there? You can change the settings in the spark properties field as shown in the screenshot. You can select that custom environment in your notebook.

[torsten · Accepted solution]

The screenshot is part of the "Environment"-item which you can create by clicking on "+ New item" in your workspace and search for "environment". Afterwards you can attach the environment in the "Workspace settings" and "Data Engineering/Science" / "Spark settings" / "Environment". More information can be found here: Create, configure, and use an environment in Fabric - Microsoft Fabric | Microsoft Learn

[Dietz]

Awesome, Thank You!

[Dietz]

or anyone who wants to know the complete solution , I created a custom environemnt and set the node sice to large, and teh issue was resolved.

[FabianSchut · Accepted solution]

Hi, can you create a custom environment in Fabric and change the spark settings there? You can change the settings in the spark properties field as shown in the screenshot. You can select that custom environment in your notebook.

[Dietz]

Thansk FabianSchut, really appreciaet your help, but I cannot seem to get to the screen in yoru message, I have created a custom environment and gone to eedit but I dont see teh screen that you posted?