Forum Discussion
Maintaining notebook Spark session within pipeline
Hi gmangiante ,
You are right. Each notebook step would start a new Spark session.
We do have a plan to enable session sharing across pipeline steps with high concurrency for pipelines which would allow you to reuse sessions and avoid additional delays.
The ETA for the deployment is planned for this semester and is currently in design phase . Stay tuned for more updates.
Appreciate your patience.
Hope this helps. Please let us know if you have any further questions. Glad to help.
If anyone finds this thread, it is scheduled for Q2 2024 (https://learn.microsoft.com/en-us/fabric/release-plan/data-engineering#concurrency)
High concurrency in pipelines
Estimated release timeline: Q2 2024
In addition to high concurrency in notebooks, we will also enable high concurrency in pipelines. This capability will allow you to run multiple notebooks in a pipeline with a single session.
Hi gmangiante ,
You are right. Each notebook step would start a new Spark session.
We do have a plan to enable session sharing across pipeline steps with high concurrency for pipelines which would allow you to reuse sessions and avoid additional delays.
The ETA for the deployment is planned for this semester and is currently in design phase . Stay tuned for more updates.
Appreciate your patience.
Hope this helps. Please let us know if you have any further questions. Glad to help.
Advocate IHi,
This feature would be very helpful. Is there an update on the ETA? Or is it still expected in December?