Forum Discussion
How to choose optimal spark environment settings?
Hi all, I'm working on a notebook with Spark SQL to move and transform data between our bronze and silver lakehouses. In general, most of our ETL is joining multiple tables and writing an output tabl...
There is something called Autotune in preview.
I must be honest, I haven't tested it. I haven't even read about it. I just heard the name. Perhaps it is relevant for what you're asking for: https://learn.microsoft.com/en-us/fabric/data-engineering/autotune?tabs=sparksql
Here's a Fabric Espresso video on the Autotune feature:
https://youtu.be/iv1kAz_ShEo?si=arP4FA7wYhJisnvc
Unfortunately, I haven't had the time to watch it myself yet.