topic Re: Pipeline Activity with multiple conditions in Pipelines

Pipeline Activity with multiple conditions

JoachimSA — Wed, 03 Apr 2024 15:06:26 GMT

Hi folks,

i have a real basic question that I am not finding the answer to:

Will the "TMM Mapping " Notebook in the example above wait for both the "SOP Bucket Creator" and "Load TMM DB" before it starts to run ?

Thans for your help!

Joachim

Re: Pipeline Activity with multiple conditions

NandanHegde — Thu, 04 Apr 2024 03:26:02 GMT

yes, it would wait for both the previous activities to be completed and that too succesfully completed before it starts its execution.

The dependencies in data pipe lines acts as an AND expression.

Reference : https://datasharkx.wordpress.com/2024/02/17/error-logging-and-the-art-of-avoiding-redundant-activities-in-msft-fabric-data-pipelines/

Re: Pipeline Activity with multiple conditions

Anonymous — Thu, 04 Apr 2024 03:50:50 GMT

Hi JoachimSA
Thanks for using Fabric Community.
I have created a repro of your scenario:

1) Inititally the Dataflow1 and Dataflow2 will start running.
2) After this the Notebook1 starts running.
3) If both Notebook1 and Dataflow2 run successfully , then the Notebook2 starts running.

Hope this helps. Please let me know if you have any further questions.

Re: Pipeline Activity with multiple conditions

JoachimSA — Thu, 04 Apr 2024 06:33:10 GMT

Perfect thank you!

Just to confirm: all activities without a condition will be triggered on pipeline start ?

I have a scenario where i would want to switch off certain parts of the pipeline depending on which input data was changed in the background so I do not need to run through everything for every small update of some side info.

are there any ressources on how one would switch off certain branches of the tree of activities ?

If an activity is switched off but is still connected to the following node it will be taken out of the condition for the following node to run right ?

I just wasn able to find very good documentation on those very basic questions...do you have a hint for me ?

Thanks and best regards,

Joachim

Re: Pipeline Activity with multiple conditions

NandanHegde — Thu, 04 Apr 2024 06:39:34 GMT

Can you plz provide a normal PPT/ visual representaion of what you want to achieve?
To ans to your query, yes
all activities without any dependency would run paralley on pipeline start.

And you can switch / turn off activities based on multiple approaches like leveraging IF or Switch activity or via dependencies.

A visual representation of your ask would help us help you better

Re: Pipeline Activity with multiple conditions

JoachimSA — Thu, 04 Apr 2024 09:23:04 GMT

Hi Nandan,

so depending on which inputs changed i would like to easily exclude each of the 3 Highlighted Blocks individually from updating.

As you can See, at the moment i just deactivated some of them. Would that be the way to go ? Longer branches get increasingly cumbersome to switch on and of then.

Also: Each activity write a deltatable at the end which is the used by the next one. Is that the optimal way when I work in a Lakehouse ?

Re: Pipeline Activity with multiple conditions

NandanHegde — Thu, 04 Apr 2024 14:19:15 GMT

You can create a parameter at the pipeline level and then leverage IF activity to either enable that flows or ignore those and procced.
So rather than just disbaling the activities, you can pass the parameter at run time to either enable or skip the flows

Re: Pipeline Activity with multiple conditions

JoachimSA — Mon, 08 Apr 2024 06:16:56 GMT

thanks, i think i could figure something out 🙂
Are there any good examples / best practices / tutorials on this ? I feel like most stuff I find online are not covering these kinds of things