Manage once, reuse everywhere: Introducing variable library support for Eventstream (Preview)

Microsoft Employee

10 hours ago

Moving a real-time data pipeline from development to test to production should not mean reconfiguring every source and destination. Yet environment-specific connection details and destination references can make each release a manual, error-prone task. Variable library integration with Eventstream in Microsoft Fabric lets teams reuse the same eventstream definition across environments, with references that resolve to each environment's resources.

What's new?

Variable library integration with Eventstream is available in preview. For supported source connections and destination items, you can use variable library references instead of direct values. These references are preserved through Git integration and deployment pipelines, helping teams reuse an eventstream definition across workspaces with environment-specific configuration.

How variable library integration works

Variable library is an item type in Microsoft Fabric that provides a centralized way to define and manage configuration values at the workspace level. Instead of entering environment-specific connection details and item identifiers directly into an eventstream, a team creates variables in a variable library and references them from the eventstream's source and destination configuration.

The variable library supports two reference types relevant to Eventstream. A connection-reference variable points to a Fabric connection, such as the one used by an Azure Event Hubs or PostgreSQL CDC source. An item-reference variable points to a Fabric item, such as the KQL database for an Eventhouse destination or the Lakehouse that receives ingested data. Neither store credentials nor raw connection strings; both store pointers to managed Fabric resources.

A variable library can contain multiple value sets, each representing a different environment.

Figure: Variable library with item-reference and connection-reference variables configured across default and alternative value sets.

A value set named "Development" might point a connection reference to a development Event Hub and an item reference to a development KQL database, while a value set named "Production" points both to their production counterparts. Only one value set is active at a time, and the eventstream resolves its references against the active set.

Eventstream connects data sources to destinations inside Fabric. Each connection and destination is identified by environment-specific values: a connection ID, a workspace ID, an item ID. When an eventstream is promoted to the next environment, those values need to change.

Without variable library integration, teams handle this manually: reconfiguring each source and destination after deploying to a new workspace, maintaining parallel definitions per environment, or running find-and-replace scripts against exported definitions. These approaches introduce risks such as:

A missed replacement can route production data to a development database.
A manual edit during a release can introduce downtime.

Variable library integration moves environment-specific values into a centrally managed variable library. The eventstream definition stores symbolic references to those variables, allowing the same definition to be used across environments. During deployment, the target workspace’s variable library supplies the values for its local resources.

Portability depends on how the variable library is configured in the target workspace. The eventstream definition retains its symbolic references, while the target workspace’s variable library maps those references to the managed connections and items used in that environment. These can include an Event Hub connection, KQL database, or Lakehouse, without embedding credentials or raw connection strings in the eventstream definition.

Each environment provides its own values for the same variables. For example, SourceConnection can point to a development Event Hub connection in the development value set and a production Event Hub connection in the production value set.

These symbolic references are preserved when the eventstream definition moves through Git integration or a deployment pipeline. Before deploying and publishing the eventstream, the target workspace’s variable library and active value set must be configured so that the references resolve to the intended resources.

Example: Promoting a logistics eventstream

Consider an illustrative logistics scenario: GPS devices on fleet vehicles send location and status events to an event hub in Azure Event Hubs. An eventstream routes those events to a KQL database in an Eventhouse for live fleet dashboards. The engineering team maintains development, staging, and production environments, each with its own event hub and KQL database.

Before variable library integration, promoting the eventstream from development to staging required deploying the definition and then manually updating the Event Hub connection and KQL database reference in the staging workspace. A missed update during one release could cause staging events to flow into the development KQL database, corrupting test results and delaying the release.

With variable library integration, the team creates a variable library in each workspace containing a connection-reference variable for the Event Hub connection and an item-reference variable for the KQL database. In each workspace, the active value set points both variables to that environment's resources. The eventstream definition uses symbolic references to these variables rather than direct identifiers.

When the team promotes the eventstream through a deployment pipeline, the definition arrives in the staging workspace with its symbolic references intact. With the staging workspace's variable library configured for staging resources, those references resolve to the staging Event Hub connection and KQL database. The team does not need to edit those identifiers in the eventstream definition, reducing the risk of routing data to the wrong environment.

Configure an eventstream to use variable library references

Configuring an eventstream to use variable library references follows a short sequence.

Create a variable library in the workspace and add the variables the eventstream will reference.
In the eventstream's source configuration, select Use a variable in the connection field, then choose the connection-reference variable from the variable library. In the destination configuration, turn on Use variable for the destination item selection and choose the item-reference variable.
Publish the eventstream and confirm that the source and destination resolve correctly and show an active status.
To support multiple environments, add value sets to the variable library, one per environment, and configure each with the connection and item values for that environment. Set the appropriate value set as active in each workspace.
Before promoting the eventstream, ensure that the target environment's source connection and destination item exist and are accessible. Deploy the variable library to the target workspace, configure its values for those resources, and select the appropriate active value set. Then deploy the eventstream and verify that its symbolic references resolve against the target workspace's variable library.

For the complete setup procedure, including prerequisites and troubleshooting, refer to Variable library integration with Eventstreams.

Benefits

Variable library integration gives teams a structured way to manage the relationship between an eventstream and its environment-specific resources.

The most immediate benefit is a reduction in deployment risk. When connections and destinations are expressed as symbolic references, promoting an eventstream to a new workspace does not require anyone to find and replace environment-specific identifiers. The values change because the target workspace's variable library supplies them, not because someone edited the definition. This reduces the risk of deployment errors, such as a production eventstream routing data to a development database.

The integration also simplifies application lifecycle management. The eventstream definition committed to Git contains symbolic references rather than resolved identifiers, so the same definition works in every workspace with a properly configured variable library. Teams can keep symbolic references in the eventstream definition committed to Git and reuse that definition across workspaces with properly configured variable libraries. This simplifies application lifecycle management by reducing the need for environment-specific copies of the definition or scripts that replace connection and destination identifiers.

Teams can manage environment-specific configuration separately from the eventstream definition. Each target workspace's active value set supplies the resource references for development, test, or production when the eventstream is deployed and published, reducing repeated edits to source and destination identifiers.

Get started

Start with one eventstream source or destination: create a variable library, add the appropriate resource reference, and configure the eventstream to use it. Then verify that the reference resolves correctly. This gives your team a practical starting point for reusing an eventstream definition across environments without repeatedly editing resource identifiers.