Issues with Custom Live Pool in Fabric Environment

[rshah2]

Hello everyone,

I am testing the Custom Live Pool feature in a Fabric Environment and have run into a couple of issues that I’m hoping others may have experienced or can help explain.

Issue 1: Unable to access Lakehouse tables when using Live Pool

I enabled Custom Live Pool in my Environment, configured a schedule, and initially set Max Clusters = 1.

When I connected a notebook to this Environment with Live Pool enabled, the session started very quickly (around 5 seconds), which indicates that the pool was working from a startup-time perspective.

However, once the session was established, I was unable to query tables from any of the Lakehouses attached to the notebook, including the default Lakehouse. Any attempt to query Lakehouse tables resulted in errors.

What makes this more confusing is that when I connected the same notebook to the same Environment without using Live Pool, the session startup took longer (around 2.5 minutes), but I was able to query tables from all attached Lakehouses successfully without any issues.

Issue 2: Live Pool appears to stop after notebook session ends

I also noticed another behavior related to pool availability.

With Max Clusters = 1, when I connect a notebook to the Environment using Live Pool, the status in the Monitor tab changes from “In Progress” to “Succeeded.”

After I stop the session in that notebook, I would expect the Live Pool to remain available for the rest of the configured schedule window so that another notebook could connect to it. However, even after waiting for over an hour, it appears that the Live Pool is no longer running for the remainder of the day, and it only becomes active again during the next scheduled run.

This seems unexpected, since the schedule window is still active and I would expect the pool to remain available or reusable during that time.

[v-sgandrathi]

Hi @rshah2,

Following up to check whether you got a chance to review the suggestions given. If the issue still persists please let us know. Glad to help. 

Thank you.

[v-sgandrathi]

Hi @rshah2,

Following up to check whether you got a chance to review the suggestions given. If the issue still persists please let us know. Glad to help. 

Thank you.

[v-sgandrathi · Accepted solution]

Hi @rshah2,

Thank you for providing more details—this helps clarify the behavior.

Regarding Issue 1, since attaching the Lakehouse after the session starts is still not working, it suggests a deeper limitation related to how Live Pool reuses pre-warmed clusters. Sometimes, the Spark session may not properly update or bind the Lakehouse metadata after attachment. You might try restarting the session after attaching the Lakehouse and checking table visibility with commands like spark.catalog.listTables(). Also, make sure the environment settings (runtime, dependencies, default Lakehouse mapping) match those of the non-Live Pool setup. If the issue remains, it is likely a product limitation or backend issue with Live Pool and Lakehouse integration, as the same setup works outside Live Pool.

For Issue 2, the behavior you described matches the current design of Custom Live Pools. The “Max Clusters” setting limits the number of clusters that can be provisioned per schedule, and these are not reused within the same cycle. Even if notebook sessions are stopped, the cluster slots do not reset until the next schedule. That’s why increasing Max Clusters to 4 allows only 4 notebook connections per cycle, not continuous reuse. Currently, Live Pool acts as a capped allocation per schedule, not a reusable or auto-scaling pool.

As a workaround, consider raising Max Clusters or using shorter, more frequent schedules to allow new clusters to be provisioned more often. For workloads needing frequent attach/detach or higher reusability, using non-Live Pool sessions may be preferable until these behaviors are improved.

Based on your observations, Issue 2 aligns with expected behavior, while Issue 1 may indicate a gap or bug that could need further review by the product team.

Thank you.

[rshah2]

For Issue 1, even when I attach the Lakehouse after starting the session, I am still encountering the same issue. It is not able to query the Lakehouse.

For Issue 2, even when I changed the Max Clusters to 4, only a total of 4 notebooks can connect to the Live Pool during the schedule of the live pool, which means that number of available clusters does not replenish at all. I would expect that with Max Clusters equal to 4, 4 notebooks can connect at the same time, but as those notebooks disconnect, it would allow others notebooks to connect, but that is not the case as of right now.

[v-sgandrathi]

Hi @rshah2,

Thank you for reaching out and sharing your observations.

Issue 1: When using Custom Live Pool, compute resources are pre-warmed and reused, which can cause differences in session initialization compared to on-demand clusters. This may mean the Lakehouse context isn’t always fully established automatically, resulting in errors when querying tables. We suggest attaching the required Lakehouse to the notebook after the session starts, checking that the environment configuration matches the non-Live Pool setup, and reinitializing the session to ensure proper binding of Lakehouse metadata. Since the setup works without Live Pool, this is likely due to session context initialization differences in Live Pool scenarios.

Issue 2: The behavior you noticed is consistent with the current design of Custom Live Pools. The schedule defines when the pool can be used, but it doesn’t guarantee clusters will stay active. With Max Clusters set to 1, after the session ends, the cluster may be deallocated, and a new one is usually provisioned when a new session starts or during the next scheduled trigger.

In summary, Issue 1 is likely caused by session context differences in Live Pool environments, and Issue 2 is expected based on the current lifecycle of Live Pools.

Thank you,

CST Member.

[RajMamidalaQ]

Hi @v-sgandrathi ,

I am referring to Issue 2. As per the configuration options for the live pool, if "Deactivate pool when idle for" option is set to 20 minutes, then we expect that the Custom Live Pool (CLP) remains active for at least 20 minutes. In our scenario, the CLP starts and the notebook uses the session from CLP however CLP stops after the notebook is run, so the next notebook run is not using the CLP. The expectation is that once CLP starts and the notebooks are run one after the other with 2-3 minutes time gap, then all these notebooks should use the sessions from CLP. It appears that the design is not correct.

Thanks

Raj

[v-sgandrathi]

Hi @RajMamidalaQ,

Thank you for the clarification.

I understand your expectation. Based on the "Deactivate pool when idle for" setting, it is reasonable to expect that once a Custom Live Pool is provisioned, it remains available during the configured idle period and can be reused by subsequent notebook sessions started within that window.

The behavior you're observing, where the pool appears to stop immediately after the notebook execution completes and subsequent notebook runs do not utilize the existing pool, does not appear to align with that expectation.

To better understand whether this is expected behavior or a potential issue, could you please share:

A screenshot of the Custom Live Pool configuration (including the idle timeout setting).

The Fabric capacity SKU being used.

Monitoring screenshots showing the pool status before and after the notebook execution.

With these details, we can further validate the behavior and determine whether additional investigation is required.

Thank you.

[RajMamidalaQ]

Hi @v-sgandrathi,

We are on F8 SKU.

Custom Live Pool (CLP) configuration:

Before notebook run:

After the 3 notebooks executed one after the other :

Screenshots of the Environment run details:

The first notebook (nb-successFactors-employee-validate) has used session from the CLP. The second notebook (nb-dnst-core-sendeclipsenotification-validation-utility) reused the session of the first notebook as it's Livy Id is same as that of the first notebook. The two notebooks were triggered by the same pipeline. The third notebook (HC_nb-successFactors-employee-transform) that got triggered by another pipeline, is having same session tag as first and second notebook, did not reuse the session from CLP. Why has CLP stopped after 9 minutes when we configured "Deactivate pool when idle for" as "20 minutes"? Has it not been stopped, the third notebook would have reused the session from CLP.

Thanks

Raj

[v-sgandrathi]

Hi @RajMamidalaQ,

Thank you for sharing the additional details and screenshots.

Based on the information provided, I can see that the first notebook used the Custom Live Pool session and the second notebook successfully reused the same Livy session. However, the third notebook did not reuse the CLP session even though the pool was configured with a 20-minute idle timeout and appears to have been triggered within that window.

From the details shared, the observed behavior does not clearly align with the expected interpretation of the idle timeout setting. Since this requires validation of the Custom Live Pool lifecycle and session reuse behavior, I recommend opening a Microsoft Fabric support ticket for further investigation.

Create a Fabric and Power BI Support Ticket - Power BI | Microsoft Learn

When creating the support request, please include:

• The Environment ID and Workspace ID.
• Capacity SKU (F8).
• The Custom Live Pool configuration screenshots.
• Notebook run timestamps.
• Livy IDs/session details demonstrating the reuse pattern.
• The monitoring screenshots shared above.

This will allow the support team to review backend telemetry and determine whether the behavior is expected or if there is a product issue affecting Custom Live Pool session reuse.

Thank you.