Forum Discussion
Maintaining notebook Spark session within pipeline
Hi gmangiante ,
You are right. Each notebook step would start a new Spark session.
We do have a plan to enable session sharing across pipeline steps with high concurrency for pipelines which would allow you to reuse sessions and avoid additional delays.
The ETA for the deployment is planned for this semester and is currently in design phase . Stay tuned for more updates.
Appreciate your patience.
Hope this helps. Please let us know if you have any further questions. Glad to help.
If anyone finds this thread, it is scheduled for Q2 2024 (https://learn.microsoft.com/en-us/fabric/release-plan/data-engineering#concurrency)
High concurrency in pipelines
Estimated release timeline: Q2 2024
In addition to high concurrency in notebooks, we will also enable high concurrency in pipelines. This capability will allow you to run multiple notebooks in a pipeline with a single session.
Hi gmangiante ,
You are right. Each notebook step would start a new Spark session.
We do have a plan to enable session sharing across pipeline steps with high concurrency for pipelines which would allow you to reuse sessions and avoid additional delays.
The ETA for the deployment is planned for this semester and is currently in design phase . Stay tuned for more updates.
Appreciate your patience.
Hope this helps. Please let us know if you have any further questions. Glad to help.
SorenSparsoAdvocate I
If anyone finds this thread, it is scheduled for Q2 2024 (https://learn.microsoft.com/en-us/fabric/release-plan/data-engineering#concurrency)
High concurrency in pipelines
Estimated release timeline: Q2 2024
In addition to high concurrency in notebooks, we will also enable high concurrency in pipelines. This capability will allow you to run multiple notebooks in a pipeline with a single session.
jjaeger94Helper II
Thank you for posting. I have the exact same issue and concern. As we are now finishing up Q2 and I am still having the same issue, does anyone know if this was actually released?
Thanks - this totally makes sense, and I was guessing that it was on the roadmap, looking at the current high-concurrency capability for interactive notebooks - that would naturally extend to pipelines, and I'm sure I'm not the only person who's come up with this issue. I look forward to future developments, and I appreciate the quick response!
MartinMasonResolver I
We're also experiencing quite a bit of performance issues with pipelines and hoping that high concurrency with help in our case as well.
Not sure if this is helpful for you, but for now, we've decided to go with pure Spark job definitions rather than leveraging Pipelines. It's not as modular and transparent as a Pipeline would be, but it gets the job done efficiently and operates in batch mode instead of interactive (https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing), which lets us schedule refreshes more easily, since they're queued. Would still like to get back to a notebook-powered Pipeline at some point, but having managed Spark available alongside our lakehouse is extremely useful.
- SorenSparso
Hi,
This feature would be very helpful. Is there an update on the ETA? Or is it still expected in December?
ajaroraMicrosoft Employee
To help prioritize this workitem, please submit a new idea on Fabric Pipelines, if it doesnt already exist.