Forum Discussion
Maintaining notebook Spark session within pipeline
Hi gmangiante ,
You are right. Each notebook step would start a new Spark session.
We do have a plan to enable session sharing across pipeline steps with high concurrency for pipelines which would allow you to reuse sessions and avoid additional delays.
The ETA for the deployment is planned for this semester and is currently in design phase . Stay tuned for more updates.
Appreciate your patience.
Hope this helps. Please let us know if you have any further questions. Glad to help.
If anyone finds this thread, it is scheduled for Q2 2024 (https://learn.microsoft.com/en-us/fabric/release-plan/data-engineering#concurrency)
High concurrency in pipelines
Estimated release timeline: Q2 2024
In addition to high concurrency in notebooks, we will also enable high concurrency in pipelines. This capability will allow you to run multiple notebooks in a pipeline with a single session.
Resolver IWe're also experiencing quite a bit of performance issues with pipelines and hoping that high concurrency with help in our case as well.
Not sure if this is helpful for you, but for now, we've decided to go with pure Spark job definitions rather than leveraging Pipelines. It's not as modular and transparent as a Pipeline would be, but it gets the job done efficiently and operates in batch mode instead of interactive (https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing), which lets us schedule refreshes more easily, since they're queued. Would still like to get back to a notebook-powered Pipeline at some point, but having managed Spark available alongside our lakehouse is extremely useful.