Simplify and improve visualisation for variable library importation within pipeline

[LouisPaulVlx]

Hi, 

One idea would be to simplify the importation of variables from a variable library inside a pipeline. 

Current state is when we add a variable from a library variable, we can only add once at a time. 

If we have 4 variables libraries to import with 10 variables inside, we need to repeat this option 40 times. It can lead to human misclick. 

Second improvment : When we have added all those variables it looks a bit messy in the library variables pane with lot of variables (1variable = 1 line). Maybe we can group them by variables libraries there ? Or find a way to make it more visual ? 

Context: 
I'm having a pipeline orchestrator, which is runing another pipeline 4 times to ingest data from 4 different datasources. The 4 uses the exact same pipeline, only changing the parameters with the library variables. I have 10 parameters for each libraries, which lead me to import 40 variables in the orchestrator to make it work. Tomorrow, I can add new sources again, and it would increase everytime by 10 the number of variables from libraries I need to import. Hence the difficulty with managing so many variables from the current library variables pane. 

Thanks for reading!

[LouisPaulVlx]

Here is the related idea : https://community.fabric.microsoft.com/t5/Fabric-Ideas/Simplify-and-improve-visualisation-for-variab...

[LouisPaulVlx]

Hi @AntoineW , @tayloramy,

Thanks for your quick replies. Actually I thought to be in the idea section, following a call I had with Microsoft Support.. I'm new to this forum. Do you mind helping me creating this idea ? 

Also, that's a good workaround @tayloramy. However, I'm the only one with technical experience on this project, and it need to stay simple for the other users who might change those in the future.. That's why I also liked the reading clarity with the library variable feature.

[tayloramy]

Hi @LouisPaulVlx, 

You can submit an idea here: https://community.fabric.microsoft.com/t5/Fabric-Ideas/idb-p/fbc_ideas

 If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[tayloramy · Accepted solution]

Hi @LouisPaulVlx, 

As @AntoineW stated,  

Today the pipeline UI only lets you link library variables one at a time, and there is no grouping in the Variables pane. That is expected for the current preview.

To avoid the 40-click setup, I’d switch from “many variables per source” to a single config per source and parse it inside the pipeline: Put one JSON string per source in the Variable Library,

e.g. SRC_A_Config = {"Server":"...","Db":"...","Table":"...","Key":"..."}.

In your activity settings, read fields with the expression builder, for example: @json(pipeline().libraryVariables.SRC_A_Config).

This reduces each source from 10 linked variables to exactly 1, while still letting you parameterize everything. Variable Libraries and pipeline integration are documented here: Variable library integration with Data pipelines and the Variable Library overview/get-started docs here: Overview, Get started

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[AntoineW]

Hello @LouisPaulVlx,

According to Microsoft’s documentation, the Variable Library feature is still in preview, and currently does not support bulk importing of variables from variable libraries into pipelines — you need to add each library variable reference individually. 

Also, the UI doesn’t yet provide grouping or collapsing of variables by library in the pipeline’s variables pane. That kind of UI organization isn’t mentioned in the current docs.

So your suggestions (bulk import + better grouping) are valid feature requests. They’re not covered by what’s available today, but could improve usability a lot.

If you like, I can help you draft a request to Microsoft so these two features (bulk select/import & variable grouping) get considered.

Source : https://learn.microsoft.com/en-us/fabric/data-factory/variable-library-integration-with-data-pipelin...

I know some updates for Variable Library are coming soon — Microsoft shared at FabCon that improvements are rolling out (see the doc) which might bring a better experience for managing variables in pipelines

Hope it can help you ! 

Best regards,

Antoine