MS Fabric resource allocation

Hi jwryu ,

Thanks for using Fabric Community.

Scenario 1: Resource Allocation and Concurrent Execution

• If Fabric's default resource allocation behavior is followed, resources (CPUs) might be reallocated dynamically from the running notebook to accommodate incoming jobs. This would lead to slower execution of the already running notebook due to reduced resources.
• However, you can configure custom resource limits for individual users, notebooks, or jobs to prevent unexpected resource reduction from existing tasks.
  
  

Scenario 2: No Re-allocation and Queuing/Failure

• If autoscaling is disabled or if there are no available resources (even in a defined Spark pool), later triggered jobs will likely be queued or fail.
• You can configure resource queues to prioritize or manage the execution order of queued jobs.

Considerations for Your Spark Pool Definition:

• If your Spark pool has more available resources than the running notebook requires, additional jobs might be able to execute concurrently without impacting the original job, depending on Fabric's configuration.
• However, be mindful of potential contention for shared resources like memory and network bandwidth, which could still slow down tasks.
  
  

Key Factors Influencing Behavior:

• Fabric configuration: Specific settings for resource allocation, autoscaling, queues, and priority levels.
• Notebook resource usage: The amount of resources (CPUs, memory) actively used by the running notebook.
• Available resources: Total resources in your Fabric environment and within the specified Spark pool.
• Job characteristics: Resource requirements (CPUs, memory) of other triggered jobs.

For more information, please refer to this documentation:
Spark workspace administration settings in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Hope this is helpful. Please let me know incase of further queries