Making do without a container (and without excessive duplication)

[vivien57]

Hello everyone,

How do you develop your data pipelines without containers (as available on SSIS for example) and without duplicating things ?

For example, here I have a flow that calls two child pipelines and I want it to report an error if one of them is in error (so that the others can continue to load or run).

Here I have two child pipelines, so two failed components. But if I have to do this for 20 tables, I'll have to make 20 failed components...

Whereas with a container, you put the 20 components inside and if one is in error, the error will be reported but the other tables will continue to load anyway.

Thank in advance for your return,

Have a nice day,

Vivien

[NandanHegde · Accepted solution]

plz refer the below blog :

https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activiti...

wherein you can create a generic pipeline framework to capture the logs via 1 activity only rather than having it duplicated across every activity

[vivien57]

Thanks for your feedback.

It does work, but I don't think it's very clean.

Is a container planned in the roadmap?

[NandanHegde]

ADF has been in picture since last couple of years and Fabric data pipelines is hosted on top of it.
So I do not foresee the above request on any immediate basis , but you can still create a feedback for the product team to work on if need be.

[vivien57]

I've just found an idea that's already been submitted and has a planned status. Feel free to vote to make it a higher priority 🙂

https://community.fabric.microsoft.com/t5/Fabric-Ideas/Adding-a-Container-for-Logical-Grouping-of-Ac...