Forum Discussion

gmangiante's avatar
gmangiante
Frequent Visitor
2 years ago
Solved

Maintaining notebook Spark session within pipeline

Hello - I'm working on building a Fabric Pipeline that uses multiple PySpark notebooks within its flow. I'm noticing that, although the notebooks run pretty quickly on their own, they take at least a minute longer when run within the pipeline. My assumption is that this is due to having to start up a new Spark session for each notebook invocation. Could someone confirm this for me? If that is the case, is there any way to maintain the session for the entire pipeline to avoid these extended start-up times? (Note: I don't believe the %run magic will work here, because I need to parameterize each notebook dynamically along the way, but please correct me if I'm wrong.) Thanks!

Need Help
pipelines
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi gmangiante ,

    You are right. Each notebook step would start a new Spark session.

    We do have a plan to enable session sharing across pipeline steps with high concurrency for pipelines which would allow you to reuse sessions and avoid additional delays.

    The ETA for the deployment is planned for this semester and is currently in design phase . Stay tuned for more updates.

    Appreciate your patience.

     

    Hope this helps. Please let us know if you have any further questions. Glad to help.

9 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi gmangiante ,

    You are right. Each notebook step would start a new Spark session.

    We do have a plan to enable session sharing across pipeline steps with high concurrency for pipelines which would allow you to reuse sessions and avoid additional delays.

    The ETA for the deployment is planned for this semester and is currently in design phase . Stay tuned for more updates.

    Appreciate your patience.

     

    Hope this helps. Please let us know if you have any further questions. Glad to help.

      • jjaeger94's avatar
        jjaeger94
        Icon for Helper II rankHelper II
        2 years ago

        Thank you for posting. I have the exact same issue and concern. As we are now finishing up Q2 and I am still having the same issue, does anyone know if this was actually released?

    • gmangiante's avatar
      gmangiante
      Frequent Visitor
      2 years ago

      Thanks - this totally makes sense, and I was guessing that it was on the roadmap, looking at the current high-concurrency capability for interactive notebooks - that would naturally extend to pipelines, and I'm sure I'm not the only person who's come up with this issue. I look forward to future developments, and I appreciate the quick response!

      • MartinMason's avatar
        MartinMason
        Icon for Resolver I rankResolver I
        2 years ago

        We're also experiencing quite a bit of performance issues with pipelines and hoping that high concurrency with help in our case as well.

    • SorenSparso's avatar
      SorenSparso
      Icon for Advocate I rankAdvocate I
      2 years ago

      Hi,

      This feature would be very helpful. Is there an update on the ETA? Or is it still expected in December?

       

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi gmangiante ,

    Thanks for using Fabric Community and reporting this.

    Apologies for the issue you have been facing. I would like to check are you still facing this issue? 

    It's difficult to tell what could be the reason for this performance.

     

    I have reached to the internal team for help on this. I will update you once I hear back from them.
    Appreciate your patience.