Forum Discussion
Issues with Custom Live Pool in Fabric Environment
Hi rshah2,
Thank you for providing more details—this helps clarify the behavior.
Regarding Issue 1, since attaching the Lakehouse after the session starts is still not working, it suggests a deeper limitation related to how Live Pool reuses pre-warmed clusters. Sometimes, the Spark session may not properly update or bind the Lakehouse metadata after attachment. You might try restarting the session after attaching the Lakehouse and checking table visibility with commands like spark.catalog.listTables(). Also, make sure the environment settings (runtime, dependencies, default Lakehouse mapping) match those of the non-Live Pool setup. If the issue remains, it is likely a product limitation or backend issue with Live Pool and Lakehouse integration, as the same setup works outside Live Pool.
For Issue 2, the behavior you described matches the current design of Custom Live Pools. The “Max Clusters” setting limits the number of clusters that can be provisioned per schedule, and these are not reused within the same cycle. Even if notebook sessions are stopped, the cluster slots do not reset until the next schedule. That’s why increasing Max Clusters to 4 allows only 4 notebook connections per cycle, not continuous reuse. Currently, Live Pool acts as a capped allocation per schedule, not a reusable or auto-scaling pool.
As a workaround, consider raising Max Clusters or using shorter, more frequent schedules to allow new clusters to be provisioned more often. For workloads needing frequent attach/detach or higher reusability, using non-Live Pool sessions may be preferable until these behaviors are improved.
Based on your observations, Issue 2 aligns with expected behavior, while Issue 1 may indicate a gap or bug that could need further review by the product team.
Thank you.
Hi v-sgandrathi ,
I am referring to Issue 2. As per the configuration options for the live pool, if "Deactivate pool when idle for" option is set to 20 minutes, then we expect that the Custom Live Pool (CLP) remains active for at least 20 minutes. In our scenario, the CLP starts and the notebook uses the session from CLP however CLP stops after the notebook is run, so the next notebook run is not using the CLP. The expectation is that once CLP starts and the notebooks are run one after the other with 2-3 minutes time gap, then all these notebooks should use the sessions from CLP. It appears that the design is not correct.
Thanks
Raj
Hi RajMamidalaQ,
Thank you for the clarification.
I understand your expectation. Based on the "Deactivate pool when idle for" setting, it is reasonable to expect that once a Custom Live Pool is provisioned, it remains available during the configured idle period and can be reused by subsequent notebook sessions started within that window.
The behavior you're observing, where the pool appears to stop immediately after the notebook execution completes and subsequent notebook runs do not utilize the existing pool, does not appear to align with that expectation.
To better understand whether this is expected behavior or a potential issue, could you please share:
A screenshot of the Custom Live Pool configuration (including the idle timeout setting).
The Fabric capacity SKU being used.
Monitoring screenshots showing the pool status before and after the notebook execution.
With these details, we can further validate the behavior and determine whether additional investigation is required.
Thank you.
Hi v-sgandrathi,
We are on F8 SKU.
Custom Live Pool (CLP) configuration:
Before notebook run:
After the 3 notebooks executed one after the other :
Screenshots of the Environment run details:
The first notebook (nb-successFactors-employee-validate) has used session from the CLP. The second notebook (nb-dnst-core-sendeclipsenotification-validation-utility) reused the session of the first notebook as it's Livy Id is same as that of the first notebook. The two notebooks were triggered by the same pipeline. The third notebook (HC_nb-successFactors-employee-transform) that got triggered by another pipeline, is having same session tag as first and second notebook, did not reuse the session from CLP. Why has CLP stopped after 9 minutes when we configured "Deactivate pool when idle for" as "20 minutes"? Has it not been stopped, the third notebook would have reused the session from CLP.
Thanks
Raj
Hi RajMamidalaQ,
Thank you for sharing the additional details and screenshots.
Based on the information provided, I can see that the first notebook used the Custom Live Pool session and the second notebook successfully reused the same Livy session. However, the third notebook did not reuse the CLP session even though the pool was configured with a 20-minute idle timeout and appears to have been triggered within that window.
From the details shared, the observed behavior does not clearly align with the expected interpretation of the idle timeout setting. Since this requires validation of the Custom Live Pool lifecycle and session reuse behavior, I recommend opening a Microsoft Fabric support ticket for further investigation.
Create a Fabric and Power BI Support Ticket - Power BI | Microsoft Learn
When creating the support request, please include:
- The Environment ID and Workspace ID.
- Capacity SKU (F8).
- The Custom Live Pool configuration screenshots.
- Notebook run timestamps.
- Livy IDs/session details demonstrating the reuse pattern.
- The monitoring screenshots shared above.
This will allow the support team to review backend telemetry and determine whether the behavior is expected or if there is a product issue affecting Custom Live Pool session reuse.
Thank you.