Unable to execute notebook with custom spark pool in Microsoft Fabric (Synapse Data Engineering)

[KarthikeyanR]

Unable to execute notebook with custom spark pool in Microsoft Fabric (Synapse Data Engineering).

Please find below the details releated to this issue. 

Fabric Capacity :

Steps followed for configuration setup :

 Step I : Changed the SKU to F64

Step II : Created custom pool with large size

Step III : Set custom pool as default

Error snapshot

Also facing below error when tried to recreate the custom pool

• Error: [InvalidSparkSettings] Unable to submit this request because spark settings specified are invalid. Error: Code = SparkSettingsInvalidNodeCount, Message = 'Invalid node count. Based on your current capacity SKU, node count lower limit: 1, upper limit: 1.'. Please correct the spark settings that align with your capacity Sku and SkuType and try again.

[v-cboorla-msft]

Hi @KarthikeyanR 

Thanks for using Microsoft Fabric Community.

Apologies for the error that you are facing here.

Error 1: Invalid NodeSize

• Cause: The SKU F64 doesn't support "Large" nodes.
• Solution:
  • Choose a compatible NodeSize from the available options in your SKU (likely "Small" or "Medium").
  • Access custom pool settings and adjust the NodeSize accordingly.
  • If Large nodes are crucial, consider upgrading to a compatible SKU.

Error 2: Invalid NodeCount

• Cause: F64 SKU allows only 1 node, but your pool configuration likely specifies more.
• Solution:
  • Set the node count to 1 in the custom pool settings.

Try to clear browser cache and cookies to eliminate potential issues.

I hope this helps. Please do let us know if you have any further questions.

1.  

[KarthikeyanR]

@v-cboorla-msft Thanks for your response.

I'm referring below two documentations to understand relation between Custom Spark Pool and SKU capacity:

https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing

Referred chart :

https://learn.microsoft.com/en-us/fabric/data-engineering/spark-compute

Referred chart :

As per the documentation Large sized pool takes 16 vCore and F64 capacity allows us to scale up to 128 vCore that means we can create custom Large pool with this configuration. Is the understanding correct?

Please let me know if more detailed documentation is available.

Also, I tried to create a custom large pool again with SKU capacity F64, but this time, I have disabled the below two options and run the code

1. Enable autoscale
2. Enable allocate

Code executed successfully. If we keep these two options enabled then getting the same error.

Please share your inputs for this issue. 

[v-cboorla-msft]

Hi @KarthikeyanR 

Apologies for the delay in response.

This might require a deeper investigation from our engineering team about your workspace and the logic behind it to properly understand what might be happening. 

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

Thanks.

[v-cboorla-msft]

Hi @KarthikeyanR 

We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes please provide the details here. Otherwise, will respond back with the more details and we will try to help.

Thanks