Forum Discussion
Solved
Maintaining notebook Spark session within pipeline
Hello - I'm working on building a Fabric Pipeline that uses multiple PySpark notebooks within its flow. I'm noticing that, although the notebooks run pretty quickly on their own, they take at least a...
Thanks - this totally makes sense, and I was guessing that it was on the roadmap, looking at the current high-concurrency capability for interactive notebooks - that would naturally extend to pipelines, and I'm sure I'm not the only person who's come up with this issue. I look forward to future developments, and I appreciate the quick response!
MartinMason
Resolver I
Resolver IWe're also experiencing quite a bit of performance issues with pipelines and hoping that high concurrency with help in our case as well.