After removing managed private endpoint, spark is not using the starter pool

[PeterDaniels]

I was testing Managed Private endpoints (MPE) in a workspace. I know that when we create managed private endpoints, that also creates a custom Spark pool in a managed vnet and that pool will take 3-5 minutes to start. I then deleted my MPE, hoping it would revert back to the fast starting starter pool, but that is not the case.  Somehow, it's still using the slow starting custom pool.  How can I force it to go back to using the starter pool?

[PeterDaniels]

Thank you, @nilendraFabric . Glad to see there is a workaround, but really bummed that it means I have to move all of my items to a new workspace. Hoping to get a simpler way to revert back to starter pools in the future.

[nilendraFabric · Accepted solution]

Hi @PeterDaniels 

When you create managed private endpoints (MPEs) in a Microsoft Fabric workspace, it indeed creates a custom Spark pool in a managed virtual network. However, deleting the MPE does not automatically revert the workspace to using the starter pool. This behavior is by design and there are a few important points to consider:

Managed Virtual Network Persistence: Once a managed virtual network is provisioned for a workspace (which happens when you create an MPE), it remains in place even after deleting the MPE. This is why you’re still experiencing the slower start times associated with custom pools.

Starter Pool Availability: When a managed virtual network is associated with a workspace, the starter pools are disabled. This is because starter pools are pre-warmed clusters hosted in a shared virtual network, which is incompatible with the private networking setup.
3. Custom Pool Usage: After the managed virtual network is set up, Spark jobs run on custom pools created on-demand within the dedicated managed virtual network of the workspace. These pools typically take 3-5 minutes to start sessions, as you’ve observed.

Solution:

Create a New Workspace: The most straightforward solution is to create a new Fabric workspace without MPEs. This new workspace will have access to the starter pool by default.
2. Contact Support: If you need to keep your current workspace but want to revert to using the starter pool, you may need to contact Microsoft Support. They might be able to assist in removing the managed virtual network association from your workspace, though this is not a standard user-accessible operation.

Please give kudos and mark the solution as accepted if this is helpful 

thanks

Nilendra