Skip to main content
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Next up in the FabCon + SQLCon recap series: The roadmap for Microsoft SQL and Maximizing Developer experiences in Fabric. All sessions are available on-demand after the live show. Register now

Reply
sparkie
Microsoft Employee
Microsoft Employee

Scaling out spark applicaton

‎07-08-2024 04:15 AM

Hello,

I am executing a long running Spark appliation from a notebook. In SparkUI I see that not all possible resources are used, so I would like to repartition the data to the maximum available cores. However, I don't know how to programmatically obtain the maximum available number of executors.
Please advise.

Labels:
Message 1 of 2
532 Views
0
1 ACCEPTED SOLUTION
Anonymous
Not applicable

‎07-08-2024 06:35 PM

HI @sparkie,

For use rest api to getting spark settings, you can refer to the following document:

Workspace Settings - Get Spark Settings - REST API (Spark) | Microsoft Learn

If the above not suitable for your requirement, you can also check the spark computer and pool setting at the 'Workspace settings':

Create custom Apache Spark pools in Fabric - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

View solution in original post

Message 2 of 2
494 Views
0
1 REPLY 1
Anonymous
Not applicable

‎07-08-2024 06:35 PM

HI @sparkie,

For use rest api to getting spark settings, you can refer to the following document:

Workspace Settings - Get Spark Settings - REST API (Spark) | Microsoft Learn

If the above not suitable for your requirement, you can also check the spark computer and pool setting at the 'Workspace settings':

Create custom Apache Spark pools in Fabric - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng

Message 2 of 2
495 Views
0

Helpful resources

Announcements
FabCon and SQLCon Highlights Carousel

FabCon &SQLCon Highlights

Experience the highlights from FabCon & SQLCon, available live and on-demand starting April 14th.

New to Fabric survey Carousel

New to Fabric Survey

If you have recently started exploring Fabric, we'd love to hear how it's going. Your feedback can help with product improvements.

Join our Fabric User Panel

Join our Fabric User Panel

Share feedback directly with Fabric product managers, participate in targeted research studies and influence the Fabric roadmap.

March Fabric Update Carousel

Fabric Monthly Update - March 2026

Check out the March 2026 Fabric update to learn about new features.

View All