Forum Discussion
Data Pipeline : Invoke Pipeline Issue
Hi Jester_3 ,
The pipeline is failing because the child pipeline is taking longer to complete. To resolve this, I switched to using the Invoke Pipeline (Legacy) activity instead of the Invoke Pipeline (Preview) activity, and the pipeline succeeded.
Please use Invoke Pipeline (Legacy), as the Preview version relies on a linked service, and the access token used by that linked service is expiring during long-running executions.
If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it!
Thanks & Regards,
Rekha,
CustomerSupportTeam.
Hi Jester_3 ,
I have reproduced the same scenario. Please refer to the screenshot below for reference. In this case, Pipeline 5 serves as the main pipeline, while Pipeline 1 acts as a child pipeline, within which Notebook 1 is called. Both pipelines executed successfully after enabling 'Wait on completion'
Could you please share the screenshots showing how you configured both the child and main pipelines? This will help me suggest better solution.
Thanks & Regards,
Rekha,
CustomerSupportTeam.
Hi v-sathmakuri
This is the topology of my main pipeline :
This is the topology of my child pipeline :
The notebooks inside of the child pipeline have 2 code cells:
- The first one is a spark sql cell that deletes the old table from the lakehouse
- The second one is a pyspark cell that runs the python that gets the new data and creates a delta table in the lakehouse.
Hope this helps.
Hi Jester_3 ,
Thank you the quick response.
Could you please ensure that the 'Wait on completion' setting is enabled on the Invoke Pipeline activity, and also confirm that all notebooks in the child pipeline complete successfully? From my end, this setup appears to be working correctly.
Additionally, please check which specific activity or step is causing the failure in the parent pipeline. This will help isolate whether the error is coming from the child pipeline or the
invocation logic.
If the notebooks are intermittently failing on the first run and succeeding on retry, consider adding try-catch blocks within the notebooks to gracefully handle transient errors (like token expiration), so they don’t fail on the first attempt
If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it!
Thanks & Regards,
Rekha,
CustomerSupportTeam.
Hi v-sathmakuri
Wait on completion is enabled on all invoke-pipeline-activity nodes.
There is no error reported from the child pipeline :
There is an error reported from the initial invoke pipeline in the main pipeline :
This is the message that is displayed for that failure
"message": "Access token has expired, resubmit with a new access token".
I have no idea what access token its referencing.
Hope this helps narrow it down.
Like I said in my initial post i've ahd this set up for some time with no issue so i don't understand why its happening now, logically it shouldn't be.
Hi Jester_3 ,
The pipeline is failing because the child pipeline is taking longer to complete. To resolve this, I switched to using the Invoke Pipeline (Legacy) activity instead of the Invoke Pipeline (Preview) activity, and the pipeline succeeded.
Please use Invoke Pipeline (Legacy), as the Preview version relies on a linked service, and the access token used by that linked service is expiring during long-running executions.
If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it!
Thanks & Regards,
Rekha,
CustomerSupportTeam.