Spark Environment Properties Set Up Suggestion

Hi Fellow members,

As we progressed ahead with setting up our spark properties, we came to know that spark node counts and auto dynamic scaling is set up while we configure our pool.

While setting up bronze environment, we had set up another set of executors configurations which should alogn with nodes and dynamic scaling setting from starter/custom pool.

These settings are necessary and should be in sync with the number of nodes as per choosen FSKU.

Thanks.

Hi ati_puri ,

Spark properties applied on, Custom Pool Settings (Data Engineering Workspace --> Pools). It Controls physical resources, number of nodes, dynamic scaling of nodes. These settings define the cluster size and elasticity at the infrastructure level.

Spark Environment Properties, Controls logical Spark configurations for jobs and notebooks. It includes properties like spark.dynamicAllocation.enabled, spark.sql.*, etc. These apply inside the Spark application running on the pool.

Environment properties cannot override node count or pool-level scaling because those are infrastructure-level. Executor-level dynamic allocation (spark.dynamicAllocation.*) works within the limits of the pool.

For example:

Pool--> min nodes = 2, max nodes = 10
Environment--> spark.dynamicAllocation.maxExecutors = 100
Actual max executors will be limited by the pool capacity nodes × cores.

Note: Pool dynamic scaling = adjusts nodes based on workload. Spark dynamic allocation = adjusts executors within the nodes allocated. If both are enabled, Spark will request more executors as needed. Pool will add nodes if executor demand exceeds current capacity. They work together, not override each other.

Please try below things.

1. Enable dynamic allocation in Spark environment for flexibility.

2. Set reasonable min/max executors aligned with pool size and SKU.

For example:

Pool max nodes = 10, each node = 16 cores --> ~160 cores total.
Set spark.dynamicAllocation.maxExecutors ~ 160 / executor cores.

3. Avoid setting unrealistic values like 1000 executors on a small pool.

Note: In your example, It is fine as long as your pool can support it. If pool max nodes = 5, you will never reach 100 executors.

Keep pool settings for node scaling. Use environment settings for executor scaling. Align both with FSKU capacity (nodes × cores × memory). Please document these settings per layer Bronze/Silver/Gold for consistency.

I hope this information helps. Please do let us know if you have any further queries.

Regards,

Dinesh