One Pipeline calling different notebooks Vs one pipleline to trigger each notebook

[fabricpribeiro]

Dears,

I have a medallion architecture with the following layers : Raw, Bronze, Silver and Gold

3 projects are different from all the rest,  as the users add files into the raw lakehouse and then we have an event trigger (one lake event) which , when someone adds a file, it triggers a pipeline in bronze that adds the information from the files into a table in bronze. 

Then, this infromation will need to be moved into silver and afetr into gold

I will have one notebook to do the necessary in each layer (one in silver an another in gold)

Onbce the infromation is added into bronze, it should be, on sucess , imediatly added to silver and after, imediately added to gold

I was thinking in having only one pipeline, the one which sits in Bronze that has the trigger and that same pipeline, calls the different notebooks on the different layers.

Do you think it makes sense?

Or should I have instead one pipeline for each notebook?

What I want is that once the trigger is fired, that everything moved till end

Thanks,

Pedro 

[nielsvdc · Accepted solution]

Hi @fabricpribeiro, I would choose one pipeline with all notebooks. When one of the layer processes fails, you can easily control the flow of the pipeline and divert it from processing any next layer.

[v-dineshya]

Hi @fabricpribeiro ,

Thank you for reaching out to the Microsoft Community Forum.

Hi @nielsvdc  and @tayloramy Thank you for your prompt responses.

Hi @fabricpribeiro  , could you please try the proposed solutions shared by @nielsvdc  and @tayloramy  ? Let us know if you’re still facing the same issue we’ll be happy to assist you further.

Regards,

Dinesh

[v-dineshya]

Hi @fabricpribeiro ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Regards,

Dinesh

[tayloramy · Accepted solution]

Hi @fabricpribeiro, 

I agree with @nielsvdc, doing it in one pipeline has less overhead, and allows you more granular control over what happens if there is a failure. 

Especially if it is only one notebook per layer, splitting this out into multiple multiple pipelines just adds complexity for no reason. 

The only time I start to split my layers into different pipelines is when they become more complex. 

[nielsvdc · Accepted solution]

Hi @fabricpribeiro, I would choose one pipeline with all notebooks. When one of the layer processes fails, you can easily control the flow of the pipeline and divert it from processing any next layer.