Notebooks Not Reusing Spark Sessions Despite Same Session Tag

Scenario:

We have a Master pipeline calling Validate pipeline followed by the Transform pipeline. The Validation pipeline calls the Validation notebook and then Notification notebook. The Transform pipeline calls the Transform notebook. All 3 notebooks, Validation, Notification and Transform, use the same session tag.

Issue:

The Validation notebook takes from 3 minutes to 5 minutes to run, and the Notification notebook then runs faster within 1 minute. The Notification notebook is using the same session as Validation notebook, so it runs faster however the Transform notebook, which is in a different pipeline doesn't reuse the same session and takes 3 minutes to 5 minutes although all notebooks are indirectly triggered by the Master pipeline. Our design with 3 pipelines is modular, reusable and maintainable, so we are not looking to compromise on the design.

Expected Solution:

Notification notebook and Transform notebook having same session tag as Validation notebook should reuse the Validation notebook's session, so that the overall Master pipeline completes in less time.