Forum Discussion

RajMamidalaQ's avatar
RajMamidalaQ
Regular Visitor
3 months ago
Solved

Notebooks Not Reusing Spark Sessions Despite Same Session Tag

Scenario: We have a Master pipeline calling Validate pipeline followed by the Transform pipeline. The Validation pipeline calls the Validation notebook and then Notification notebook. The Transform ...
capacities
workspaces
  • oussamahaimoud's avatar
    oussamahaimoud
    3 months ago

    Hi RajMamidalaQ,

     

    Keep your 3 pipelines as-is for standalone/reusable use. In the Master pipeline specifically, instead of using "Invoke Pipeline" activity for Validation, inline the three notebook activities directly in the Master pipeline (not as child pipelines), all sharing the same session tag.

    Your child pipelines (Validate, Transform) remain intact for independent use, the Master pipeline becomes a flat orchestrator with direct notebook calls and a shared session tag.

     

    Hope this helps! Don't forget to accept as solution ✅ and thumbs up 👍 in order to keep helping others 🙂 

     

    Best regards,

    Oussama (Data Consultant - Expert Fabric & Power BI)

oussamahaimoud's avatar
oussamahaimoud
Memorable Member
3 months ago

Hi RajMamidalaQ,

 

Keep your 3 pipelines as-is for standalone/reusable use. In the Master pipeline specifically, instead of using "Invoke Pipeline" activity for Validation, inline the three notebook activities directly in the Master pipeline (not as child pipelines), all sharing the same session tag.

Your child pipelines (Validate, Transform) remain intact for independent use, the Master pipeline becomes a flat orchestrator with direct notebook calls and a shared session tag.

 

Hope this helps! Don't forget to accept as solution ✅ and thumbs up 👍 in order to keep helping others 🙂 

 

Best regards,

Oussama (Data Consultant - Expert Fabric & Power BI)