Issues with Custom Live Pool in Fabric Environment

Hi rshah2,

Thank you for providing more details—this helps clarify the behavior.

Regarding Issue 1, since attaching the Lakehouse after the session starts is still not working, it suggests a deeper limitation related to how Live Pool reuses pre-warmed clusters. Sometimes, the Spark session may not properly update or bind the Lakehouse metadata after attachment. You might try restarting the session after attaching the Lakehouse and checking table visibility with commands like spark.catalog.listTables(). Also, make sure the environment settings (runtime, dependencies, default Lakehouse mapping) match those of the non-Live Pool setup. If the issue remains, it is likely a product limitation or backend issue with Live Pool and Lakehouse integration, as the same setup works outside Live Pool.

For Issue 2, the behavior you described matches the current design of Custom Live Pools. The “Max Clusters” setting limits the number of clusters that can be provisioned per schedule, and these are not reused within the same cycle. Even if notebook sessions are stopped, the cluster slots do not reset until the next schedule. That’s why increasing Max Clusters to 4 allows only 4 notebook connections per cycle, not continuous reuse. Currently, Live Pool acts as a capped allocation per schedule, not a reusable or auto-scaling pool.

As a workaround, consider raising Max Clusters or using shorter, more frequent schedules to allow new clusters to be provisioned more often. For workloads needing frequent attach/detach or higher reusability, using non-Live Pool sessions may be preferable until these behaviors are improved.

Based on your observations, Issue 2 aligns with expected behavior, while Issue 1 may indicate a gap or bug that could need further review by the product team.

Thank you.