Forum Discussion

RajMamidalaQ's avatar
RajMamidalaQ
Regular Visitor
3 months ago
Solved

Notebooks Not Reusing Spark Sessions Despite Same Session Tag

Scenario: We have a Master pipeline calling Validate pipeline followed by the Transform pipeline. The Validation pipeline calls the Validation notebook and then Notification notebook. The Transform ...
capacities
workspaces
  • oussamahaimoud's avatar
    oussamahaimoud
    3 months ago

    Hi RajMamidalaQ,

     

    Keep your 3 pipelines as-is for standalone/reusable use. In the Master pipeline specifically, instead of using "Invoke Pipeline" activity for Validation, inline the three notebook activities directly in the Master pipeline (not as child pipelines), all sharing the same session tag.

    Your child pipelines (Validate, Transform) remain intact for independent use, the Master pipeline becomes a flat orchestrator with direct notebook calls and a shared session tag.

     

    Hope this helps! Don't forget to accept as solution ✅ and thumbs up 👍 in order to keep helping others 🙂 

     

    Best regards,

    Oussama (Data Consultant - Expert Fabric & Power BI)

v-tejrama's avatar
v-tejrama
Community Support
2 months ago

Hi RajMamidalaQ ,

 

Thank you oussamahaimoud  for the response provided!

Has your issue been resolved? If the response provided by the community member addressed your query, could you please confirm? It helps us ensure that the solutions provided are effective and beneficial for everyone.

Thank you.

v-tejrama's avatar
v-tejrama
Community Support
2 months ago

Hi RajMamidalaQ ,

 

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.

Thank you.