Simplify and improve visualisation for variable library importation within pipeline
Hi,
One idea would be to simplify the importation of variables from a variable library inside a pipeline.
Current state is when we add a variable from a library variable, we can only add once at a time.
If we have 4 variables libraries to import with 10 variables inside, we need to repeat this option 40 times. It can lead to human misclick.
Second improvment : When we have added all those variables it looks a bit messy in the library variables pane with lot of variables (1variable = 1 line). Maybe we can group them by variables libraries there ? Or find a way to make it more visual ?
Context:
I'm having a pipeline orchestrator, which is runing another pipeline 4 times to ingest data from 4 different datasources. The 4 uses the exact same pipeline, only changing the parameters with the library variables. I have 10 parameters for each libraries, which lead me to import 40 variables in the orchestrator to make it work. Tomorrow, I can add new sources again, and it would increase everytime by 10 the number of variables from libraries I need to import. Hence the difficulty with managing so many variables from the current library variables pane.
Thanks for reading!
2 Comments
- frithjof_vCommunity ChampionCurrent workaround for adding multiple at one time is to use the Edit JSON of the data pipeline
Recent ideas
OneLake Catalog full-text and content search
Why is it not possible to search OneLake by item description or even content? So I was thinking about how a user can find a table if they don't know in which lakehouse it is located? Why can't you se...TePe6 hours agoHelper IVPlanned1.1KViews21likes5Comments