Forum Discussion
Solved
Maintaining notebook Spark session within pipeline
Hello - I'm working on building a Fabric Pipeline that uses multiple PySpark notebooks within its flow. I'm noticing that, although the notebooks run pretty quickly on their own, they take at least a...
SorenSparso
Advocate I
Advocate IIf anyone finds this thread, it is scheduled for Q2 2024 (https://learn.microsoft.com/en-us/fabric/release-plan/data-engineering#concurrency)
High concurrency in pipelines
Estimated release timeline: Q2 2024
In addition to high concurrency in notebooks, we will also enable high concurrency in pipelines. This capability will allow you to run multiple notebooks in a pipeline with a single session.
jjaeger94
Helper II
Helper IIThank you for posting. I have the exact same issue and concern. As we are now finishing up Q2 and I am still having the same issue, does anyone know if this was actually released?