Forum Discussion

Jester_3's avatar
Jester_3
Frequent Visitor
1 year ago
Solved

Data Pipeline : Invoke Pipeline Issue

Hi,    In one of my workspaces I have 2 pipelines, one of which calls the other using the invoke pipeline utility.   The parent pipeline is successfully triggering the child pipeline which is the...
Need Help
Pipeline
  • v-sathmakuri's avatar
    v-sathmakuri
    1 year ago

    Hi Jester_3 ,

     

    The pipeline is failing because the child pipeline is taking longer to complete. To resolve this, I switched to using the Invoke Pipeline (Legacy) activity instead of the Invoke Pipeline (Preview) activity, and the pipeline succeeded.

    Please use Invoke Pipeline (Legacy), as the Preview version relies on a linked service, and the access token used by that linked service is expiring during long-running executions.

     

    If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

     

    Thanks & Regards,

    Rekha,

    CustomerSupportTeam.

Jester_3's avatar
Jester_3
Frequent Visitor
1 year ago

Hi v-sathmakuri 

 

I checked that the wait for completion setting was enabled on the invoke-pipeline-activity and it was.

 

Thanks for the suggestion.

v-sathmakuri's avatar
v-sathmakuri
Community Support
1 year ago

Hi Jester_3 ,

 

Could you please check whether the pipeline you're trying to run has succeeded? Also, please review the latest run of this pipeline using the Monitor option in Fabric

 

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it! 

 

Thanks & Regards,

Rekha,

CustomerSupportTeam.

  • Jester_3's avatar
    Jester_3
    Frequent Visitor
    1 year ago

    Hi v-sathmakuri 

     

    I have checked the monitoring hub and can see that the child pipeline is correctly completing but a failure is still being reported to the parent pipeline.

     

     

    In this instance the first fail from tickets is causing main pipeline to fail, but the fresh service pipeline runs until completion and has succeeded.

     

    Thanks

    • v-sathmakuri's avatar
      v-sathmakuri
      Community Support
      1 year ago

      Hi Jester_3 ,

       

      I have reproduced the same scenario. Please refer to the screenshot below for reference. In this case, Pipeline 5 serves as the main pipeline, while Pipeline 1 acts as a child pipeline, within which Notebook 1 is called. Both pipelines executed successfully after enabling 'Wait on completion'

       

       

      Could you please share the screenshots showing how you configured both the child and main pipelines? This will help me suggest better solution.

       

      Thanks & Regards,

      Rekha,

      CustomerSupportTeam.

      • Jester_3's avatar
        Jester_3
        Frequent Visitor
        1 year ago

        Hi v-sathmakuri 

         

        This is the topology of my main pipeline : 

         

        This is the topology of my child pipeline : 

         

        The notebooks inside of the child pipeline have 2 code cells:

        • The first one is a spark sql cell that deletes the old table from the lakehouse
        • The second one is a pyspark cell that runs the python that gets the new data and creates a delta table in the lakehouse.

        Hope this helps.