Forum Discussion

Jester_3's avatar
Jester_3
Frequent Visitor
1 year ago
Solved

Data Pipeline : Invoke Pipeline Issue

Hi,    In one of my workspaces I have 2 pipelines, one of which calls the other using the invoke pipeline utility.   The parent pipeline is successfully triggering the child pipeline which is the...
Need Help
Pipeline
  • v-sathmakuri's avatar
    v-sathmakuri
    1 year ago

    Hi Jester_3 ,

     

    The pipeline is failing because the child pipeline is taking longer to complete. To resolve this, I switched to using the Invoke Pipeline (Legacy) activity instead of the Invoke Pipeline (Preview) activity, and the pipeline succeeded.

    Please use Invoke Pipeline (Legacy), as the Preview version relies on a linked service, and the access token used by that linked service is expiring during long-running executions.

     

    If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

     

    Thanks & Regards,

    Rekha,

    CustomerSupportTeam.

v-sathmakuri's avatar
v-sathmakuri
Community Support
1 year ago

Hi Jester_3 ,

 

I have reproduced the same scenario. Please refer to the screenshot below for reference. In this case, Pipeline 5 serves as the main pipeline, while Pipeline 1 acts as a child pipeline, within which Notebook 1 is called. Both pipelines executed successfully after enabling 'Wait on completion'

 

 

Could you please share the screenshots showing how you configured both the child and main pipelines? This will help me suggest better solution.

 

Thanks & Regards,

Rekha,

CustomerSupportTeam.

Jester_3's avatar
Jester_3
Frequent Visitor
1 year ago

Hi v-sathmakuri 

 

This is the topology of my main pipeline : 

 

This is the topology of my child pipeline : 

 

The notebooks inside of the child pipeline have 2 code cells:

  • The first one is a spark sql cell that deletes the old table from the lakehouse
  • The second one is a pyspark cell that runs the python that gets the new data and creates a delta table in the lakehouse.

Hope this helps.