topic Re: dynamic library variables in Pipelines

dynamic library variables

krhacke — Tue, 22 Apr 2025 18:02:44 GMT

I'm trying to use the new library variables to dynamically call a workspace / notebook. The standard syntax to call dymanic library variable looks something like this, with the actual notebook name at the end:

@pipeline().libraryVariables.nb_table_transformation

I would like to pass my json variable to dynamically change the notebook name. I've tried a couple options:

1) Concatinating the base syntax with the dynamic value coming from json

@concat('pipeline().libraryVariables.', json(pipeline().parameters.TaskStep.Notebook).NotebookId)

2) Create a standard variable that is resolved in a previous step using the json and then concatenate it into the library variable syntax

@concat('pipeline().libraryVariables.', variables('v_notebook'))

Although both of these methods resolve the syntax correctly, the library variable does not resolve the underlying guid that I have created in my library variable set up.

Any ideas on how to make this work? Is this something that isn't yet available (being public preview) with library variables?

Any suggestions / info would be greatly appreciated!

Re: dynamic library variables

v-karpurapud — Wed, 23 Apr 2025 05:10:59 GMT

Hi krhacke

Thank you for reaching out to the Microsoft Fabric Community Forum.

We understand you are encountering an issue iwth dynamic libarary variables. As Library Variables are designed to store reusable configurations at the workspace level and are referenced using @pipeline().libraryVariables.<variable_name>.

Dynamic Concatenation (@concat('pipeline().libraryVariables.',json(pipeline().parameters.TaskStep.Notebook).NotebookId))may not work as expected because the expression engine treats it as a literal string or fails to resolve the GUID.

Library variables are resolved at runtime, but the dynamic construction of the variable name may not be fully supported in the current preview, as the engine might not dereference the concatenated string to the actual library variable.

Dynamic key resolution (e.g., pipeline().libraryVariables[variables('v_notebook')]) isn't supported in the current preview, requiring hardcoded keys instead.

To resolve this, consider a workaround that aligns with ADF’s current capabilities while achieve the goal of dynamically selecting a notebook based on a JSON parameter.

Using a Switch activity to map the JSON parameter to the corresponding library variable explicitly, avoiding dynamic key concatenation.
Set up library variables in the ADF workspace, mapping them to notebook paths or GUIDs (e.g., nb_table_transformation, nb_other_transformation).
Create a pipeline parameter TaskStep to include the notebook ID in JSON format (e.g., {"Notebook": {"NotebookId": "nb_table_transformation"}}).
Extract Notebook ID: Use a Set Variable activity to extract the notebook ID from the JSON parameter and store it in a variable (v_notebook_id).
Use a Switch Activity: Add a Switch activity to map the notebook ID to the corresponding library variable:

Case: nb_table_transformation: Set the notebook path to@pipeline().libraryVariables.nb_table_transformation.
Default Case: Add a Fail activity or a no-op activity with a message like "Invalid NotebookId".

This approach ensures explicit mapping of the notebook ID to the library variable, avoiding dynamic concatenation issues.

If this response resolves your query, kindly mark it as Accepted Solution to help other community members. A Kudos is also appreciated if you found the response helpful.

Thank You!

Re: dynamic library variables

krhacke — Wed, 23 Apr 2025 13:14:52 GMT

Hi v-karpurapud - thank you for your response!

You mentioned that dynamic construction of library variables isn't supported in the current preview. Is this something that will be included in the GA release?

We currently hardcode workspace and notebook guids in each of our source systems' notebook json workbenches. However, as these change from dev to test to prod workspaces, my client feels it's burdonsome to have to update these at each step in each workbench. It would be great to be able to set these name / guid values in a library variable once, then use just the name for each in the json.

Re: dynamic library variables

v-karpurapud — Mon, 28 Apr 2025 09:26:53 GMT

Hi krhacke

As of now, Microsoft has not officially announced that this feature will be included in the General Availability (GA) release either. Library variables must be referenced statically in the pipeline at design time, and expressions like @concat()or indexing into pipeline().library variables dynamically are not evaluated in a way that resolves the actual variable content.

If supporting dynamic referencing of library variables is important for your use case, it’s recommended to submit feedback via Microsoft Fabric Ideas so that it can be considered for future updates.

Thank You!