Forum Discussion

Scott_Powell's avatar
Scott_Powell
Icon for Helper V rankHelper V
1 year ago

How to choose optimal spark environment settings?

Hi all, I'm working on a notebook with Spark SQL to move and transform data between our bronze and silver lakehouses. In general, most of our ETL is joining multiple tables and writing an output tabl...
data engineering
Environment
Notebook
Srisakthi's avatar
Srisakthi
Icon for Super User rankSuper User
1 year ago

Hi Scott_Powell ,

 

Please try with these properties,

spark.conf.set("spark.sql.parquet.vorder.enabled","true")
spark.conf.set("spark.databricks.delta.optimizeWrite.enabled","true")
spark.conf.set("spark.databricks.delta.optimizeWrite.binSize","1073741824")

 

For spark optimization and configuration related things, please check your notebook execution logs. There you can find information on how many stages used, executors used, idle executors etc. based on all these factors we can tune node configuration