Forum Discussion
Scott_Powell
Advocate IV
Advocate IVHow to choose optimal spark environment settings?
Hi all, I'm working on a notebook with Spark SQL to move and transform data between our bronze and silver lakehouses. In general, most of our ETL is joining multiple tables and writing an output tabl...
AndyDDC
Most Valuable Professional
Most Valuable ProfessionalHi Scott_Powell there's a blog here which may help visualise the Spark execution plan:
Visualizing Spark Execution Plans – Gerhard Brueckl on BI & Data
Scott_PowellAdvocate IV
Advocate IVhi AndyDDC , I just wanted to reply back - the article you linked seems to have an easier / more graphical way of seeing the explain plans for a query, but that still doesn't give much info on how to adjust spark settings to make it faster. I'm finding it really hard to get good info on that. I'm going to explore the autotuning frithjof_v mentioned, hoping that helps.
Thanks!
Scott