Transform Configuration Management with Fabric Variable Library

[Anusha_M]

 

The Variable Library is a groundbreaking feature in Microsoft fabric that is designed to empower users to define and manage variables at the workspace level. This tool allows seamless integration across various workspace items, including data pipelines, notebooks, and Lakehouse shortcuts. This feature addresses several pain points and enhances the overall user experience within Fabric.

Introduction and Purpose

Before the introduction of the Fabric Variable Library, managing configuration across workspace items was a significant challenge, especially in CI/CD processes. This created significant overhead and complexity.

• Managing configuration across workspaces was a significant challenge.

• CI/CD processes were less streamlined due to inefficient configuration handling.

• Configurations had to be manually managed in multiple workspaces, leading to increased overhead.

• Complexity arose from the lack of a unified system for managing configurations.

The Variable Library primary purpose is to provide a unified way to manage configuration across workspaces, streamline Continuous Integration and Continuous Deployment (CI/CD) processes by allowing configuration changes based on release stages. This feature aims to simplify the management of variables and sharing of configuration inside the workspace across different items.

 

Key Features and Functionality

Create a Variable Library

Variable Library as a new item in Fabric, which acts as a bucket of variables that can be consumed by other fabric items in the same workspace. Customers can create and manage variable libraries, which include various types of variables much like environment variables do in traditional development and deployment workflows. 

Users can also leverage the Variable Library to establish consistency across diverse components within a workspace. For instance, when multiple data pipelines depend on shared parameters, the library acts as a single source of truth, simplifying updates and reducing the risk of configuration errors. This cohesion facilitates smoother collaboration and streamlines operations, particularly critical in complex workflows.

Create a Variable Library by selecting New Item in workspace and name the variable library. This allows users to define and manage variables.

To define each variable, users need to specify its name, type, and default value. Additionally, there is a feature called “Add Value Set,” which allows users to create multiple value sets tailored for different stages such as development, testing, and production. This ensures flexibility in configuration management, enabling smoother transitions between stages without altering the core variable definitions.

Value Sets and Deployment

Value Sets within the Variable Library offer an enhanced way to manage configurations tailored to specific deployment scenarios, such as development, testing, and production. By creating multiple value sets, users can ensure flexibility and adaptability in their workflows. A unique feature allows one value set to be designated as active, which becomes the default configuration for the workspace. This is particularly useful for environments that require distinct setups based on the stage of the CI/CD pipeline.

Supported Fabric Items

The Variable Library supports various consumer items, including data pipelines, notebooks, and lake houses. Currently, Variable library and its integration with data pipelines is currently in public preview. The resolution of values depends on the item's behaviour and can occur during runtime.

 

Variable library with data pipelines:

1. To use a Variable library variable in your pipeline, create a new pipeline or navigate to an existing pipeline as below.
In your data pipeline, navigate to “Library variables(preview)” where you need to create a reference to your variable library in the bottom panel

2. Once you have added your library variable references, add your pipeline activities. In the activity settings, click Add dynamic content for the setting you want to parameterize. The expression builder opens.

3. If you don't see Library variables (preview), click the 3 dots next to Functions and select Library variables (preview).

4. Click on your Library variable reference to add a new expression to the expression builder. Then, click Ok to add your expression.

5. We will now see that the dynamic content is added to your activity setting.

6. Save and run your pipeline as you normally would. You'll see that the value passed is whatever value is set as Active in your Variable library.

Variable library with Notebooks:

Notebook will support Variable Library through Notebook utilities, allowing users to reference variables in their code. The user experience includes a help section explaining how to reference the Variable Library variables.

Git Integration

The Variable Library seamlessly integrates with Git, enabling users to manage Variable Library items effectively. Public API is available for CRUD operations of item definitions.

• Variable Listing: All variables are organized within a dedicated variable.json file for easy reference and management.

Valuesets: Each value set is stored as individual files in the “valueSets” folder. Only edited variables are listed under "variableOverrides".The active value set is not included in the core definition but can be accessed and controlled programmatically through the Public API.

 

Scenarios for Using the Variable Library

• Centralized Configuration Management: Teams working across multiple workspaces can use the Variable Library to maintain a unified configuration. This eliminates redundancy and reduces errors associated with manual configuration updates, ensuring consistency across data pipelines, notebooks, and other consumer items.
• Dynamic Parameterization: During runtime or specific events, the Variable Library supports dynamic resolution of values. This is particularly beneficial in scenarios where workflows need to adapt based on changing parameters without manual intervention.
• CICD – Stage specific Configurations: In development, testing, and production stages, the Variable Library enables seamless transitions by allowing distinct value sets for each stage. For instance, a testing stage might require mock database credentials, while the production stage mandates real credentials.
• Ease of Collaboration: With shared permissions, contributors, members, and admins can edit and manage values collaboratively. This shared control ensures that configurations stay relevant and up to date while maintaining workspace-specific security.

Roadmap Spotlight

The roadmap for the Variable Library is ambitious, promising exciting developments that will further enhance its usability. These advancements aim to make the library a versatile tool for configuration management across diverse workflows.

Moreover, the possibility of automating the Variable Library via REST APIs opens doors for streamlined operations, allowing users to dynamically control variables and their values programmatically. 

Upcoming Features of the Variable Library

• Introduction of item reference and connection reference variable types.
• Integration with Azure Key Vault.
• Support for additional fabric items such as:
  • Notebooks
  • Scheduler
  • Shortcuts for Lakehouse

In conclusion, the Variable Library represents a transformative addition to the Fabric suite. By addressing critical pain points and offering a robust solution for centralized configuration management, it empowers users to optimize their workflows. Coupled with its comprehensive features and planned enhancements, including automation possibilities through APIs, the Variable Library is set to be indispensable for Fabric users striving for seamless integration and scalability.

 

References

What is Fabric Variable library? To learn more, refer to the Variable library documentation.

REST APIs to automate and interact with the Variable Library - REST API (VariableLibrary)

Contributors

@Anusha_M @kranthimeda 

[mithunm]

Great Article @Anusha_M  , @kranthimeda !!