In Fabric Failed to create session for executing notebook

Hi Anonymous,

Thank you for reaching out to Microsoft Fabric Community.

The error message indicates the pipeline failed due to an issue with creating a Spark session. please follow the below solutions:

• In the new configuration, the minimum executor instances were increased, this higher demand may exceed the cluster's available resources during peak load times, leading to session initialization failures. Adjust the number of executors to a more manageable range (similar to the old configuration). This reduces the likelihood of cluster resource exhaustion.
• The higher number of executors combined with 16 cores and 112 GB per executor can exhaust cluster capacity, especially if other workloads are running concurrently. And also driver memory was reduced drastically from 224 GB to 56 GB, while executor memory remains high (112 GB). This imbalance can lead to spark application inefficiencies. So, increase driver memory to match the executor memory more closely.
• Running the notebook every 15 minutes (96 times per day) puts significant stress on the cluster. Increase the interval (20–30 minutes) between the pipeline runs to reduce cluster load.
• Ensure all libraries or dependencies required by the notebook are pre-installed and verified to avoid personalization errors.
• Add a retry mechanism to the pipeline to handle transient failures automatically.

Implement these changes to stabilize the pipeline and reduce the likelihood of failures

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

Thanks and regards,

Anjan Kumar Chippa