Forum Discussion

Icon for Helper V rank

Helper V

1 year ago

How to choose optimal spark environment settings?

Hi all, I'm working on a notebook with Spark SQL to move and transform data between our bronze and silver lakehouses. In general, most of our ETL is joining multiple tables and writing an output tabl...

data engineering

Icon for Most Valuable Professional rank

Most Valuable Professional

1 year ago

Hi Scott_Powell there's a blog here which may help visualise the Spark execution plan:

Visualizing Spark Execution Plans – Gerhard Brueckl on BI & Data

Scott_Powell
Helper V
1 year ago
Thanks AndyDDC , really appreciate it! I'll check it out and see if it helps!

Microsoft - would be awesome if I could run a query in a notebook, and then have CoPilot suggest optimal settings to make it run quickly....

Thanks,
Scott
- frithjof_v
  Community Champion
  1 year ago
  There is something called Autotune in preview.
  
  I must be honest, I haven't tested it. I haven't even read about it. I just heard the name. Perhaps it is relevant for what you're asking for: https://learn.microsoft.com/en-us/fabric/data-engineering/autotune?tabs=sparksql
  - Scott_Powell
    Helper V
    1 year ago
    hi frithjof_v , I had heard about the autotune back at the Fabric conference in Vegas, but didn't realize it was actually available yet, and that it was disabled by default. I'll definitely turn this on and see what happens, and post back.
    
    Thanks for the heads up!
    Scott
Scott_Powell
Helper V
1 year ago
hi AndyDDC , I just wanted to reply back - the article you linked seems to have an easier / more graphical way of seeing the explain plans for a query, but that still doesn't give much info on how to adjust spark settings to make it faster. I'm finding it really hard to get good info on that. I'm going to explore the autotuning frithjof_v mentioned, hoping that helps.

Thanks!
Scott