Forum Discussion
Issues with Custom Live Pool in Fabric Environment
Hi rshah2,
Thank you for providing more details—this helps clarify the behavior.
Regarding Issue 1, since attaching the Lakehouse after the session starts is still not working, it suggests a deeper limitation related to how Live Pool reuses pre-warmed clusters. Sometimes, the Spark session may not properly update or bind the Lakehouse metadata after attachment. You might try restarting the session after attaching the Lakehouse and checking table visibility with commands like spark.catalog.listTables(). Also, make sure the environment settings (runtime, dependencies, default Lakehouse mapping) match those of the non-Live Pool setup. If the issue remains, it is likely a product limitation or backend issue with Live Pool and Lakehouse integration, as the same setup works outside Live Pool.
For Issue 2, the behavior you described matches the current design of Custom Live Pools. The “Max Clusters” setting limits the number of clusters that can be provisioned per schedule, and these are not reused within the same cycle. Even if notebook sessions are stopped, the cluster slots do not reset until the next schedule. That’s why increasing Max Clusters to 4 allows only 4 notebook connections per cycle, not continuous reuse. Currently, Live Pool acts as a capped allocation per schedule, not a reusable or auto-scaling pool.
As a workaround, consider raising Max Clusters or using shorter, more frequent schedules to allow new clusters to be provisioned more often. For workloads needing frequent attach/detach or higher reusability, using non-Live Pool sessions may be preferable until these behaviors are improved.
Based on your observations, Issue 2 aligns with expected behavior, while Issue 1 may indicate a gap or bug that could need further review by the product team.
Thank you.
Hi v-sgandrathi,
We are on F8 SKU.
Custom Live Pool (CLP) configuration:
Before notebook run:
After the 3 notebooks executed one after the other :
Screenshots of the Environment run details:
The first notebook (nb-successFactors-employee-validate) has used session from the CLP. The second notebook (nb-dnst-core-sendeclipsenotification-validation-utility) reused the session of the first notebook as it's Livy Id is same as that of the first notebook. The two notebooks were triggered by the same pipeline. The third notebook (HC_nb-successFactors-employee-transform) that got triggered by another pipeline, is having same session tag as first and second notebook, did not reuse the session from CLP. Why has CLP stopped after 9 minutes when we configured "Deactivate pool when idle for" as "20 minutes"? Has it not been stopped, the third notebook would have reused the session from CLP.
Thanks
Raj
Hi RajMamidalaQ,
Thank you for sharing the additional details and screenshots.
Based on the information provided, I can see that the first notebook used the Custom Live Pool session and the second notebook successfully reused the same Livy session. However, the third notebook did not reuse the CLP session even though the pool was configured with a 20-minute idle timeout and appears to have been triggered within that window.
From the details shared, the observed behavior does not clearly align with the expected interpretation of the idle timeout setting. Since this requires validation of the Custom Live Pool lifecycle and session reuse behavior, I recommend opening a Microsoft Fabric support ticket for further investigation.
Create a Fabric and Power BI Support Ticket - Power BI | Microsoft Learn
When creating the support request, please include:
- The Environment ID and Workspace ID.
- Capacity SKU (F8).
- The Custom Live Pool configuration screenshots.
- Notebook run timestamps.
- Livy IDs/session details demonstrating the reuse pattern.
- The monitoring screenshots shared above.
This will allow the support team to review backend telemetry and determine whether the behavior is expected or if there is a product issue affecting Custom Live Pool session reuse.
Thank you.