Regular expression functions in Microsoft Fabric Data Warehouse (Preview)
Data warehouses routinely ingest valuable information embedded in free text, where inconsistent structure makes it difficult to query, connect, and analyze. Incoming values can also require quality checks before they reach curated tables. Microsoft Fabric Data Warehouse and the SQL analytics endpoint now support scalar regular expression functions in preview, so you can extract useful values, validate formats, standardize text, and mask sensitive data directly in T-SQL instead of adding a separate processing step. What’s new in preview Regular expressions provide a concise way to recognize, extract, and transform complex text patterns that would otherwise require multiple string operations. The preview introduces the following scalar functions: REGEXP_LIKE: checks whether text matches a pattern. REGEXP_REPLACE: replaces matching text to clean, standardize, or mask values. REGEXP_SUBSTR: extracts a substring that matches a pattern. REGEXP_INSTR: returns the position of a matching substring. REGEXP_COUNT: counts pattern occurrences in a string. Use these functions during refinement to turn identifiers embedded in free text into structured values and to flag malformed records before they reach downstream systems. The following scenarios demonstrate both uses. Extract useful values from raw text In this fictional example, inbound customer emails land in bronze.EmailMessages. The Sender column contains a value such as “Avery Smith”, and an order reference such as “ORD-10293” may appear in the Subject column. REGEXP_SUBSTR extracts the sender’s name, email address, company identifier, and order number as structured values that can be related to curated customer, company, and order data. For “Avery Smith ”, the query returns “Avery Smith” as the Name, “[email protected]” as the Email, and “contoso” as the Company. It also extracts the order number from the Subject column. Validate contact data before it reaches downstream systems Contact data in the silver.DimCustomer table can contain missing characters, extra spaces, or email addresses that do not follow the expected format. REGEXP_LIKE identifies records that fail the format check, so you can review, correct, or quarantine them before the dimension feeds reports, campaigns, and downstream applications. SELECT SourceCustomerId, EmailAddress FROM silver.DimCustomer WHERE NOT REGEXP_LIKE(EmailAddress, '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$'); The query isolates email addresses that do not match the expected format so they can be reviewed, corrected, or quarantined before the customer dimension is released to downstream consumers. Conclusion Regular expression functions help turn hard-to-use text into structured, trustworthy warehouse data. You can extract sender, company, and order details from inbound emails and validate contact values during refinement, all with concise T-SQL close to the source. The result is information that is easier to connect, analyze, and trust. Explore the preview and learn how to use regular expression functions in Fabric Data Warehouse in the documentation. Share your scenarios and feedback with the Microsoft Fabric community on Reddit.
Govern Fabric connections with admin APIs and data connection policies (Preview)
Co-author: Abhishek Narain Protect continuity, understand connection risk, and set tenant-wide guardrails Imagine that the person who built a business-critical pipeline has left your organization. The connection still exists, but nobody is sure who can manage it or whether its credentials will keep working. What begins as a handover problem quickly becomes a governance question: who owns your connections, where do they lead, and which identities should be allowed to use them? Microsoft Fabric brings these questions together through two complementary capabilities: Admin connection APIs give tenant administrators visibility and ownership-management tools across connection types. Data connection policies set preventive controls for cloud connections. Together, they support a practical sequence: restore responsibility, understand your connections, then enforce the authentication boundaries your organization needs. From recovery to prevention Keep work moving when a connection owner leaves A departure should not leave a connection without someone who can maintain it. When an existing connection has a single owner, a Fabric tenant administrator can use the admin connection APIs to grant the Owner role to a responsible successor, another administrator, or an appropriate service principal. This restores the ability to manage the connection and establishes clear accountability during the handover. Use the admin connection APIs to review current access, assign a responsible successor as an Owner, update an existing assignment, and remove access that is no longer appropriate. The role-assignment endpoints support GET, POST, PATCH, and DELETE operations; refer to Admin Connections REST API reference for endpoint syntax and examples. Ownership is only part of continuity. Granting Owner does not transfer or renew OAuth credentials, confer permissions on the source system, or guarantee that dependent workloads will run. The incoming owner must check the connection requirements, update or reauthenticate credentials where needed, and validate the workloads that depend on it. Treat that validation as part of offboarding, not a task to postpone until something fails. See your connections, endpoints, and access clearly A tenant-wide inventory helps administrators find ownership gaps, unexpected access, and connections that may require review before they become operational issues. The admin APIs provide visibility across all connection types, including connections you do not own or have permission to manage. By comparison, the standard Manage connections and gateways experience does not provide a tenant-wide inventory, even for administrators. Use the tenant-wide connections endpoint to collect the complete inventory, including all paginated results. The response includes connection names and IDs, connectivity and connector types, endpoint paths, privacy levels, and credential or authentication-type metadata—but not secret values. Combine this information with current role assignments to identify unexpected access and endpoints or domains that require closer review. Recency data can help prioritize the review. Created, last-linked, and last-credentials-used timestamps answer different questions: last linked indicates when the connection configuration was most recently associated with a workload, while last credentials used provides a runtime signal. These timestamps do not provide a complete historical audit log, an automatic dependency map, or proof that an apparently idle connection is safe to delete. Confirm critical dependencies with workload owners before making changes. This read-only demonstration uses a Python client to list tenant connections and inspect a connection role assignment. It does not change ownership. For the admin API endpoints and an example response, refer to List Connections (Admin) and List Connection Role Assignments (Admin). Try a read-only connection inventory app For a hands-on inventory, refer to the Fabric Connection Inventory sample app. It uses the admin connection APIs to list tenant connections and their Owner and PendingOwner assignments, with search and filters for reviewing connector types, endpoints, and ownership results. It does not change connections, assign owners, or enforce policies. This is a single-tenant deployment template, not a hosted app that readers can sign into. To use it, an organization must deploy its own instance in a workspace that supports Fabric Apps (Preview), configure a single-tenant Microsoft Entra app, grant delegated Tenant.Read.All consent, and sign in with a Fabric administrator account. See the repository’s setup instructions. Illustrative API examples To inventory connections across the tenant, an administrator can make a read-only request: GET https://api.fabric.microsoft.com/v1/admin/connections Authorization: Bearer <access-token> { "value": [ { "id": "11111111-1111-4111-8111-111111111111", "displayName": "ContosoConnection", "connectivityType": "ShareableCloud", "connectionDetails": { "type": "Web", "path": "https://www.contoso.com/" } } ] } If the response includes a continuationToken, use it to request the next page. For an ownership handoff, the following illustrative write request shows the payload for adding a user as an Owner. It was not part of the read-only GIF or the production test: POST https://api.fabric.microsoft.com/v1/admin/connections/11111111-1111-4111-8111-111111111111/roleAssignments Authorization: Bearer <access-token> Content-Type: application/json { "principal": { "id": "22222222-2222-4222-8222-222222222222", "type": "User" }, "role": "Owner" } An ownership assignment may require the incoming owner to reauthenticate. It does not transfer credentials or guarantee that dependent workloads will run. Make approved authentication methods the rule An inventory shows how connections authenticate today. An authentication allowlist turns your organization’s requirements into a tenant-wide boundary for cloud connections. Instead of relying on every creator to make the same choice, you define the methods that policy permits. An authentication allowlist helps administrators reduce inconsistent or noncompliant authentication choices across cloud connections. In the Fabric admin portal, select Data Policies in the left navigation, then open the Authentication allowlist tab. Add the approved authentication methods, turn on Restrict to allowlist, and select Save. Review the tenant-wide confirmation before committing. Select methods that meet your organization’s requirements and are supported by the relevant connectors; no single method is universally available or appropriate. When enforcement is enabled, methods outside the allowlist fail policy evaluation for existing as well as new cloud connections. Enable this carefully, because it can disrupt existing workflows. That makes the inventory essential preparation: identify affected connections, agree on supported alternatives with their owners, and plan credential changes before enabling the restriction. These controls govern cloud connections; they are not a promise of coverage for every gateway, desktop client, or data path. Define which Microsoft Entra tenants can authenticate connections An approved authentication method does not make every Microsoft Entra tenant acceptable. Organizations may also need to block connection authentication associated with unapproved tenants. The Tenant allowlist addresses this requirement by defining which Microsoft Entra tenants can be used for Entra-authenticated cloud connections, helping reduce the risk of authentication through unapproved tenant identities. On Tenant allowlist, select Add tenant ID and enter the approved tenant GUIDs. Validate each ID carefully: the experience does not look up friendly names, and the list supports up to 100 IDs. Turn on Restrict to allowlist, select Save, and review the tenant-wide confirmation. Once enforced, the restriction affects both existing and new Entra-authenticated cloud connections. Keep identity and destination controls distinct. The Authentication allowlist limits which methods cloud connections can use, while the Tenant allowlist limits which Microsoft Entra tenants they can authenticate with. Neither control approves destination endpoints nor acts as a firewall rule. Start with visibility, then enforce Before enabling enforcement, inventory your cloud connections, identify critical workloads and their owners, and confirm the authentication methods and Microsoft Entra tenant IDs they need. Review the Authentication allowlist and Tenant allowlist tabs together, then enable only the controls you intend to enforce. Each allowlist has its own Restrict to allowlist switch. Communicate the change and assess its effect on existing as well as new cloud connections; enforcement applies tenant-wide. Policy changes can take up to one hour to take effect. After enforcement begins, test affected cloud connections and dependent workloads. A noncompliant new connection is rejected; an existing connection may show Policy Evaluation Failure and go Offline. Open the connection details to identify the cause, then work with the connection owner and tenant admin to resolve the authentication-method or tenant-policy mismatch. Ownership transfer alone does not resolve a policy failure. Closing Start with a complete inventory and an ownership review of critical connections. Use that evidence to define and validate authentication-method and Microsoft Entra tenant allowlists. These capabilities cannot prevent every outage or misuse, but they can reduce unmanaged handovers, clarify decisions about access and endpoints, and apply preventive controls that reflect your organization’s requirements. To plan your implementation, review the guidance for data source management and data connection policies.Bring your Fabric data into your next app with connectors
Most business applications start with data that already exists. Customer orders may live in a Fabric SQL database. Inventory history may be in a Lakehouse. Sales performance may already be modeled in Power BI. The opportunity is not to move that data again, but to turn trusted data into an experience that helps people make decisions and take action. Connectors in Fabric Apps shorten that path. They give an app a consistent way to work with Fabric data, so developers can spend less time on integration plumbing and more time building the experience users need. Connect to the Fabric data you already trust. Build the experience your users need. Keep the application close to the data and governance foundation already in place. Instead of beginning with infrastructure, start with the customer's need: What should this app help someone understand, decide, or do? From data to an application Imagine a large telecommunications provider operating thousands of cell sites and serving millions of customers. Network telemetry may stream into a Lakehouse; service-impact and incident data may live in a Fabric SQL database; and trusted availability and capacity measures may already be modeled in Power BI. A report can show where the service is degrading. But a network operations engineer may need to go further: Which sites and customers are affected? Is the issue driven by congestion, equipment, power, or an upstream dependency? What maintenance work or recent changes are associated with the site? Can I create or update an incident and route the next action from the same experience? This is where a Fabric App adds value. Connectors let the app combine governed network and operational data in one task-oriented experience for investigation, triage, and action. Architecture: Network operations app Network telemetry remains in Fabric; operational records stay in their governed source; and trusted measures continue to come from the semantic model. The Fabric App brings those sources together so an engineer can move from a network signal to an informed operational action without reconstructing context across multiple tools. Choose the connector that matches the job Choose the connector that matches the Fabric item and the job the app needs to perform. Fabric Apps supports documented connector patterns for lakehouse SQL analytics endpoints, warehouses, SQL databases in Fabric, and Power BI semantic models. Fabric source Connector type Documented operations Lakehouse SQL analytics endpoint fabric-sqlanalytics Read Warehouse fabric-warehouse Read, create, update, delete SQL database in Fabric fabric-sqldatabase Read, create, update, delete Power BI semantic model fabric-semanticmodel executeQuery Request only the operations the scenario needs. For example, a read-only inventory experience should not request write access. An operational app that updates a warehouse or SQL database should make that requirement explicit. Three real-world patterns Incident triage A telecommunications provider can build a network operations app on a Lakehouse, Fabric SQL database, and semantic model. An engineer can search for a cell site; review latency, congestion, availability, and impacted services; inspect recent maintenance or configuration changes; and create or update the operational incident from one experience. Value: The engineer moves from detecting service degradation to understanding impact and coordinating the next action without switching across disconnected monitoring and case-management views. Customer 360 for sellers A sales team may have accounts and order data in a warehouse, product signals in a Lakehouse, and trusted revenue measures in a semantic model. A Fabric App can organize this information around the account and add a tailored experience for exploration, notes, recommended next steps, or AI-assisted questions. Value: The app is designed around the seller’s job, not around the boundaries between data systems. Retail store operations from a semantic model Imagine a retailer such as Nordstrom with trusted sales, inventory, fulfillment, and customer-service measures already defined in a Power BI semantic model. A Fabric App can use that model as the analytical foundation for a store operations experience where a manager reviews daily performance, identifies products at risk of stocking out, investigates order or pickup delays, and coordinates the next action with the store team. Value: Store teams reuse trusted business measures and move from insight to action without recreating logic or switching across multiple reports and operational tools. Start with one focused workflow A good first app starts with a narrow, valuable job. The builder journey is: Create or initialize a Fabric Apps project. Identify the Fabric item behind the workflow. Add a connector with only the operations the app needs. Test with the permissions and data boundaries expected for users. For example, a read-only warehouse connection can expose governed site and service-impact data to the app: npx rayfin connector add \ --type fabric-warehouse \ --workspace-id YOUR_WORKSPACE_ID \ --item-id YOUR_WAREHOUSE_ITEM_ID \ --name network-operations \ --operations read The connector is added to the project configuration and can then be used through the generated connector interface. For complete syntax, supported operations, prerequisites, and current limitations, see Connect Fabric Apps to Fabric data. Build from the data you already trust Your most valuable business data is already in Microsoft Fabric. Connectors help you turn that data into focused applications that do more than display insight by helping you complete real business tasks. Get started with Fabric App and connect to your data.Bring insights and action together with Fabric Apps in Org apps
Business users move from understanding what happened, to investigating why, to acting. Microsoft Fabric can bring that journey together. Fabric Apps provide purpose-built, data-driven experiences when users need more than a report. Org apps distribute them alongside Power BI reports and other supported Fabric content in a curated destination. Move from insight to action in one destination Power BI reports help users monitor performance, explore trends, and understand what’s happening in the business. Many insights, however, lead to a next step: submitting a request, updating a plan, investigating an exception, assigning an owner, or tracking a resolution. A Fabric App provides the custom interface and workflow for that next step. Including it in an org app places action beside the reports and operational content that provide context. Users enter one destination organized around their role, review the insight, open the Fabric App, and act. Understanding apps in Fabric Experience Purpose Value Fabric Apps Build for a specific task Create purpose-built applications connected to Fabric data, with managed APIs, authentication, hosting, and service integration. Org apps Distribute in business context Package Fabric Apps with Power BI reports, paginated reports, notebooks, maps, and real-time dashboards, with tailored navigation, branding, and audiences. Workspace apps Distribute Power BI content Package and publish reports and dashboards from a workspace. Does not support notebooks and real-time dashboards, automatically granting and revoking access to underlying items and support for Fabric Git integration What the combined experience looks like Consider a retailer that wants regional managers to identify store issues and coordinate a response. The Store Performance page displays a Power BI report, with other business experiences available through the org app navigation. Start with the insight A regional manager opens the Retail Operations Hub, an app designed around the work of running stores. The landing experience surfaces a Power BI report with revenue, margin, inventory, and customer-service trends. A real-time dashboard highlights emerging operational issues. ort with revenue, margin, inventory, and customer-service trends Continue into a Fabric App The report shows that a store is missing its sales target while carrying excess inventory. The manager opens the Store Action Planner, a Fabric App included in the same org app. The application brings the relevant context into a focused workflow where the manager can review the exception, choose a response, assign owners, and track the plan. Keep reporting and actions connected The org app keeps the solution together while tailoring content for regional and store managers. Business owners can update navigation and included items without rebuilding the Fabric App. Each team retains clear ownership: BI authors evolve reports, developers improve the Fabric App, and business owners curate the org app. Benefits One place to work: Move from Power BI insight to action without finding another tool. Role-based distribution: Give each audience the right mix of reports, applications, and operational content. Clear ownership: Let BI authors, developers, and business owners evolve their parts independently within one consistent destination. Business context: Organize navigation and branding around the process, not the underlying technology or workspace. Get started Build the tailored experience with Fabric Apps, then use the org app guidance to package it with Power BI reports and other supported Fabric content. Build the experience users need with Fabric Apps. Distribute it in the context of their work with org apps.
More ways to use scalar UDFs in Microsoft Fabric Data Warehouse (Preview)
Scalar user-defined functions (UDFs) let data teams define a business rule once and reuse it across workloads in both Warehouse and the SQL analytics endpoint in Microsoft Fabric. This helps teams maintain consistent logic without duplicating it across queries. Some business rules require more than a single expression. They use procedural logic such as conditional branches, multiple RETURN statements, or WHILE loops. Expression Block inlining compiles and executes this logic in a distributed manner, helping reusable functions scale across analytical workloads. Building on that foundation, this update brings several improvements: More natural loop control and broader query-shape support. Automatic adoption of newly supported capabilities by existing functions, without recreating or altering them. Inlineability validation for new or changed functions, with diagnostics that explain any gaps and an explicit WITH INLINE = AUTO override when a non-inlineable function is acceptable. Express loop intent with BREAK and CONTINUE Loops often need to skip the rest of the current iteration or stop processing entirely. Without CONTINUE and BREAK, developers typically rely on extra flags, nested IF blocks, or repeated conditions. Those workarounds make the control flow harder to read and more difficult to update safely. Consider a function that standardizes tracking codes received from carrier and warehouse systems. Formatting characters such as spaces, hyphens, and slashes should be ignored. A pipe character marks the beginning of an operational note, so the function should stop there. CREATE OR ALTER FUNCTION dbo.fn_NormalizeTrackingCode (@Input VARCHAR(MAX)) RETURNS VARCHAR(MAX) AS BEGIN IF @Input IS NULL RETURN NULL; DECLARE @Result VARCHAR(MAX) = ''; DECLARE @Character VARCHAR(4); DECLARE @Position INT = 1; WHILE @Position <= LEN(@Input) BEGIN SET @Character = SUBSTRING(@Input, @Position, 1); SET @Position = @Position + 1; IF @Character = '|' BREAK; IF @Character = ' ' OR @Character = '-' OR @Character = '/' CONTINUE; SET @Result = @Result + UPPER(@Character); END; RETURN @Result; END; In this example, CONTINUE skips formatting characters, while BREAK stops processing when the operational note begins. The function therefore follows the business rule without a separate stop flag or another level of nesting. For an input such as "northwind-1000 / 1234 | imported from partner API", the function returns "NORTHWIND10001234". Use large-value data types in more query shapes The example uses VARCHAR(MAX) for both the parameter and the return type. Previously, MAX data types relied on Scalar UDF inlining and were subject to its calling-query restrictions. Expression Block inlining now supports VARCHAR(MAX), NVARCHAR(MAX), and VARBINARY(MAX) parameters, local variables, and return values. The function can keep its MAX types and be called from various query shapes. The following CTE is one example: it standardizes each event, then summarizes the number of source formats and standardized codes for each carrier. WITH StandardizedEvents AS ( SELECT ShipmentId, CarrierCode, RawTrackingCode, dbo.fn_NormalizeTrackingCode(RawTrackingCode) AS StandardizedCode FROM dbo.TrackingEvents WHERE RawTrackingCode IS NOT NULL ) SELECT CarrierCode, COUNT_BIG(*) AS EventCount, COUNT(DISTINCT RawTrackingCode) AS RawCodeCount, COUNT(DISTINCT StandardizedCode) AS StandardizedCodeCount FROM StandardizedEvents GROUP BY CarrierCode ORDER BY CarrierCode; You no longer need to choose between retaining MAX parameters and return types and using the query structure that best expresses your intent. Call scalar UDFs in IF EXISTS queries Procedural batches frequently need to ask whether any row meets a reusable business rule before taking the next action. The query inside IF EXISTS can now call a scalar UDF that uses Expression Block inlining. The following statement checks whether the tracking event table contains a specific standardized code. The function owns the normalization rule, while the batch owns the decision about what to do when a matching row exists. IF EXISTS ( SELECT 1 FROM dbo.TrackingEvents WHERE dbo.fn_NormalizeTrackingCode(RawTrackingCode) = 'NORTHWIND10001234' ) BEGIN SELECT 'Matching shipment found' AS ValidationResult; END ELSE BEGIN SELECT 'No matching shipment found' AS ValidationResult; END; This pattern keeps the existence check set-based and keeps the reusable scalar logic in one function. It is useful for validation gates, conditional batch steps, data-quality checks, and orchestration procedures that need to branch on the presence of qualifying rows. Existing functions gain new capabilities automatically As inlining support expands, your existing UDFs can become inlineable, run more efficiently, or become available in additional query shapes without being recreated or altered. The function text remains the same, while its current inlineability is reflected in the sys.sql_modules catalog view. SELECT SCHEMA_NAME(o.schema_id) AS FunctionSchema, o.name AS FunctionName, m.is_inlineable AS IsInlineable, m.inline_eligibility_mask AS InlineEligibilityMask FROM sys.objects AS o INNER JOIN sys.sql_modules AS m ON o.object_id = m.object_id WHERE o.type = 'FN'; The is_inlineable value indicates whether the function can be inlined, and inline_eligibility_mask identifies the applicable inlining technique. Check these values to confirm current service behavior because a function’s eligibility can change as inlining support expands. Default inlineability checks with an explicit override CREATE OR ALTER FUNCTION now checks inlineability by default. When a definition does not meet the inlining requirements, the statement is rejected, and the error identifies what prevents inlining. This catches compatibility gaps when the function is defined, before it can be referenced by a query. When a non-inlineable function is acceptable given its business context, WITH INLINE = AUTO explicitly overrides the default creation requirement. For example, the following function reads a table and uses multiple RETURN statements. It can still be useful as a standalone existence check even though it is not inlineable. CREATE OR ALTER FUNCTION dbo.fn_HasTrackingEvents (@CarrierCode VARCHAR(20)) RETURNS BIT WITH INLINE = AUTO AS BEGIN IF EXISTS ( SELECT 1 FROM dbo.TrackingEvents WHERE CarrierCode = @CarrierCode ) RETURN 1; RETURN 0; END; WITH INLINE = AUTO does not make a function inlineable or change the execution behavior of a function that is already inlineable. It explicitly allows creation of an intentionally non-inlineable standalone function. What's next Existing functions gain these capabilities automatically. First, check their status in the system catalog, then test the new query shapes with your procedural scalar UDFs. For the current syntax, metadata, and inlining requirements, refer to CREATE FUNCTION for Fabric Data Warehouse. For a guided introduction, refer to Create scalar user-defined functions in Microsoft Fabric Data Warehouse. To share a specific scenario or feedback, email Fabric DW functions feedback.
Manage once, reuse everywhere: Introducing variable library support for Eventstream (Preview)
Moving a real-time data pipeline from development to test to production should not mean reconfiguring every source and destination. Yet environment-specific connection details and destination references can make each release a manual, error-prone task. Variable library integration with Eventstream in Microsoft Fabric lets teams reuse the same eventstream definition across environments, with references that resolve to each environment's resources. What's new? Variable library integration with Eventstream is available in preview. For supported source connections and destination items, you can use variable library references instead of direct values. These references are preserved through Git integration and deployment pipelines, helping teams reuse an eventstream definition across workspaces with environment-specific configuration. How variable library integration works Variable library is an item type in Microsoft Fabric that provides a centralized way to define and manage configuration values at the workspace level. Instead of entering environment-specific connection details and item identifiers directly into an eventstream, a team creates variables in a variable library and references them from the eventstream's source and destination configuration. The variable library supports two reference types relevant to Eventstream. A connection-reference variable points to a Fabric connection, such as the one used by an Azure Event Hubs or PostgreSQL CDC source. An item-reference variable points to a Fabric item, such as the KQL database for an Eventhouse destination or the Lakehouse that receives ingested data. Neither store credentials nor raw connection strings; both store pointers to managed Fabric resources. A variable library can contain multiple value sets, each representing a different environment. Figure: Variable library with item-reference and connection-reference variables configured across default and alternative value sets. A value set named "Development" might point a connection reference to a development Event Hub and an item reference to a development KQL database, while a value set named "Production" points both to their production counterparts. Only one value set is active at a time, and the eventstream resolves its references against the active set. Eventstream connects data sources to destinations inside Fabric. Each connection and destination is identified by environment-specific values: a connection ID, a workspace ID, an item ID. When an eventstream is promoted to the next environment, those values need to change. Without variable library integration, teams handle this manually: reconfiguring each source and destination after deploying to a new workspace, maintaining parallel definitions per environment, or running find-and-replace scripts against exported definitions. These approaches introduce risks such as: A missed replacement can route production data to a development database. A manual edit during a release can introduce downtime. Variable library integration moves environment-specific values into a centrally managed variable library. The eventstream definition stores symbolic references to those variables, allowing the same definition to be used across environments. During deployment, the target workspace’s variable library supplies the values for its local resources. Portability depends on how the variable library is configured in the target workspace. The eventstream definition retains its symbolic references, while the target workspace’s variable library maps those references to the managed connections and items used in that environment. These can include an Event Hub connection, KQL database, or Lakehouse, without embedding credentials or raw connection strings in the eventstream definition. Each environment provides its own values for the same variables. For example, SourceConnection can point to a development Event Hub connection in the development value set and a production Event Hub connection in the production value set. These symbolic references are preserved when the eventstream definition moves through Git integration or a deployment pipeline. Before deploying and publishing the eventstream, the target workspace’s variable library and active value set must be configured so that the references resolve to the intended resources. Example: Promoting a logistics eventstream Consider an illustrative logistics scenario: GPS devices on fleet vehicles send location and status events to an event hub in Azure Event Hubs. An eventstream routes those events to a KQL database in an Eventhouse for live fleet dashboards. The engineering team maintains development, staging, and production environments, each with its own event hub and KQL database. Before variable library integration, promoting the eventstream from development to staging required deploying the definition and then manually updating the Event Hub connection and KQL database reference in the staging workspace. A missed update during one release could cause staging events to flow into the development KQL database, corrupting test results and delaying the release. With variable library integration, the team creates a variable library in each workspace containing a connection-reference variable for the Event Hub connection and an item-reference variable for the KQL database. In each workspace, the active value set points both variables to that environment's resources. The eventstream definition uses symbolic references to these variables rather than direct identifiers. When the team promotes the eventstream through a deployment pipeline, the definition arrives in the staging workspace with its symbolic references intact. With the staging workspace's variable library configured for staging resources, those references resolve to the staging Event Hub connection and KQL database. The team does not need to edit those identifiers in the eventstream definition, reducing the risk of routing data to the wrong environment. Configure an eventstream to use variable library references Configuring an eventstream to use variable library references follows a short sequence. Create a variable library in the workspace and add the variables the eventstream will reference. In the eventstream's source configuration, select Use a variable in the connection field, then choose the connection-reference variable from the variable library. In the destination configuration, turn on Use variable for the destination item selection and choose the item-reference variable. Publish the eventstream and confirm that the source and destination resolve correctly and show an active status. To support multiple environments, add value sets to the variable library, one per environment, and configure each with the connection and item values for that environment. Set the appropriate value set as active in each workspace. Before promoting the eventstream, ensure that the target environment's source connection and destination item exist and are accessible. Deploy the variable library to the target workspace, configure its values for those resources, and select the appropriate active value set. Then deploy the eventstream and verify that its symbolic references resolve against the target workspace's variable library. For the complete setup procedure, including prerequisites and troubleshooting, refer to Variable library integration with Eventstreams. Benefits Variable library integration gives teams a structured way to manage the relationship between an eventstream and its environment-specific resources. The most immediate benefit is a reduction in deployment risk. When connections and destinations are expressed as symbolic references, promoting an eventstream to a new workspace does not require anyone to find and replace environment-specific identifiers. The values change because the target workspace's variable library supplies them, not because someone edited the definition. This reduces the risk of deployment errors, such as a production eventstream routing data to a development database. The integration also simplifies application lifecycle management. The eventstream definition committed to Git contains symbolic references rather than resolved identifiers, so the same definition works in every workspace with a properly configured variable library. Teams can keep symbolic references in the eventstream definition committed to Git and reuse that definition across workspaces with properly configured variable libraries. This simplifies application lifecycle management by reducing the need for environment-specific copies of the definition or scripts that replace connection and destination identifiers. Teams can manage environment-specific configuration separately from the eventstream definition. Each target workspace's active value set supplies the resource references for development, test, or production when the eventstream is deployed and published, reducing repeated edits to source and destination identifiers. Get started Start with one eventstream source or destination: create a variable library, add the appropriate resource reference, and configure the eventstream to use it. Then verify that the reference resolves correctly. This gives your team a practical starting point for reusing an eventstream definition across environments without repeatedly editing resource identifiers. Learn more For detailed guidance on configuration, prerequisites, and supported scenarios, see the following resources: Variable library integration with Eventstreams CI/CD in Microsoft Fabric Git integration in Microsoft Fabric We welcome your feedback. Leave a comment or submit ideas at Fabric Ideas.
Connect coding agents to Fabric notebooks with FNTK (Preview)
The Fabric Notebook Toolkit (Preview) helps developers shorten the feedback loop between writing code and validating it with live notebook results. It connects coding agents to Microsoft Fabric notebooks through a command-line interface (CLI) that runs in the terminal alongside tools such as Claude Code, GitHub Copilot CLI, and Codex. An agent can inspect the current code, run one cell, review its output, make a targeted change, and run it again without resubmitting the entire notebook—much as a human programmer working interactively. A coding agent uses FNTK from the terminal to interact with a Microsoft Fabric notebook. From code suggestions to notebook execution A coding agent can help write a transformation, but a proposed change is not the same as a working result. When execution happens elsewhere, developers must move code into a notebook, run it, and bring the results or errors back to the agent. FNTK connects those steps so the agent can work with a running Fabric notebook and its execution results instead of relying only on code shared in the conversation. Interactive development and debugging are the central use cases. You can ask an agent to inspect a notebook, investigate a failing transformation, or validate a change using actual execution output. The notebook remains in Fabric and uses Fabric capacity for distributed data processing, while FNTK provides the terminal commands the agent uses to interact with the live session. Run, inspect, and refine one cell at a time FNTK keeps the notebook session active while the agent works, so developers can explore and refine code without repeatedly restarting the workflow. For example, an agent can inspect a schema, adjust a conversion, check several records, and return to the same transformation as new information emerges. This mirrors the iterative way developers work in notebooks and keeps each change grounded in actual execution results. The toolkit brings the operations needed for that workflow into the terminal. An agent can discover the target notebook, read and update its cells, execute code, and retrieve results. Instead of treating every adjustment as a new whole-notebook submission, it can focus on the code being investigated. Find workspaces, notebooks, and lakehouses to establish the context for the task. Read, add, or update notebook cells as the work develops. Execute a cell's code and inspect its output before deciding what to change. Reuse the live session to refine and rerun the relevant code. For example, a transformation may fail because an input column has a different type than expected. An agent can inspect the schema, revise the affected code, and rerun it in the existing session to check the result. This avoids restarting the entire notebook merely to test that change and keeps the investigation focused on the failing operation. Give the agent the evidence to debug FNTK gives the agent the execution evidence needed to determine the source of a failure. It exposes cell output and Spark job state, along with driver logs, executor logs, and logs from Livy, the service that manages Spark sessions. With this context, the agent can investigate the likely source of a problem instead of relying on source code alone. The CLI can return structured JavaScript Object Notation (JSON) responses that identify results, errors, and suggested next actions. An agent can use this information to determine whether execution has finished, retrieve diagnostics, and choose what to investigate next. This shortens the feedback loop between writing code and understanding its behavior while reducing the need to copy outputs and logs manually between tools. FNTK in practice at Microsoft For our data engineering team at Microsoft, FNTK filled a gap between generating code and working with real data. Previously, we used an AI assistant to draft code locally, then copied it into Fabric notebooks to run it and brought the results back into the conversation. With FNTK, we can stay in GitHub Copilot, execute code in Fabric, and use the results to iterate. We started by installing the toolkit, connecting with the appropriate permissions, and asking questions about a data source. Exploratory work quickly became a valuable use case. We could ask for row counts, distributions, or samples, then follow up based on the actual results. From there, we built shared workflow skills and a developer plugin around FNTK, incorporated our engineering standards, and connected the plugin to Azure DevOps through the Model Context Protocol (MCP) to bring user-story context into the same workflow. In practice, this has sped up development and helped us work across multiple user stories, with more of our time spent reviewing changes rather than writing every line ourselves. Copilot handles the reasoning; FNTK provides the connection to live notebook execution. We can still inspect the runs in Fabric, and our existing source-control, quality, and human-review practices remain in place. In practice, this workflow has helped us work across multiple user stories and spend more time reviewing changes instead of writing every line ourselves. Copilot handles the reasoning; FNTK provides the connection to live notebook execution. We can still inspect the runs in Fabric, and our existing source-control, quality, and human-review practices remain in place. Figure: FNTK connects an AI developer agent to live Fabric notebook execution, alongside MCP tools for context and delivery tools for human-reviewed changes. Capabilities with FNTK commands Command or group Usage fntk help Show command guidance and examples. fntk init Install or update coding agent workflow skills. fntk discover Find workspaces, notebooks, lakehouses, and identity information. fntk notebook Manage notebooks, sessions, and default lakehouses. fntk cell Read, edit, and organize notebook cells. fntk code Execute code, retrieve results, and cancel runs. fntk monitor Inspect Spark progress, diagnostics, and resource usage. fntk logs Download driver, executor, Livy, and cell logs. fntk resource Manage notebook resource files and folders. fntk skill Read built-in coding and debugging guidance. fntk server Manage the local FNTK server. Get started To try the workflow, install the toolkit and connect your coding agent by following the FNTK installation and quick-start documentation, including fntk init for workflow skill setup. Then choose an existing development notebook and a focused task, such as inspecting one transformation. Run the code, review the output, and use the result to guide your next change.More expressive graph analysis: GQL improvements in Microsoft Fabric Graph
Coauthors: Idris Motiwala, and Justin Fine January–August 2026 updates expand how users find, inspect, compose, and analyze connections. Organizations use graph data to understand how information moves between systems, how services depend on one another, how conversations develop, and how people or organizations relate. Graph Query Language (GQL) in Microsoft Fabric Graph helps users explore and analyze those connections with standardized query patterns. From January through August 2026, new GQL capabilities made it easier to control traversals, inspect paths, compose multistage queries, summarize connected data, and work with complex values. These improvements help analysts express more graph questions in a single query and give developers finer control over query behavior. The following sections explain where each capability is useful and show the associated syntax. Find and control paths Shortest paths with ANY SHORTEST When several paths connect two entities, many applications need one shortest path rather than every possible path. ANY SHORTEST makes that intent explicit. This is useful for dependency analysis, recommendations, network routing, and investigations where fewer intermediate steps are preferred. MATCH ANY SHORTEST ({id: 1})->{,2}(d) RETURN d.id AS dst Quantified edge patterns Quantified edge patterns repeat a complete relationship pattern and can apply conditions to intermediate nodes. In the following example, each repeated step requires the source person to use Chrome. This gives developers more control than repeating only an edge pattern. MATCH p=(a:Person {id: 987})((x)-[k:knows]->() WHERE x.browserUsed = 'Chrome'){2}(b:Person) FOR n IN NODES(p) WITH OFFSET off RETURN n.browserUsed ORDER BY a.id, b.id, off LIMIT 12 Complete path modes Path modes define whether a traversal can revisit nodes or edges. WALK permits repetition. TRAIL prevents repeated edges. SIMPLE prevents repeated interior nodes. ACYCLIC prevents any repeated node. DIFFERENT EDGES prevent one edge from satisfying more than one edge variable in the match. These choices help query results reflect the meaning of a route, hierarchy, or dependency chain. MATCH WALK (:Person {firstName: "Juliana"})-[:studyAt]->(:University)<-[:studyAt]-(p:Person) RETURN p.firstName AS name MATCH TRAIL (:Person {firstName: "Juliana"})-[:studyAt]->(:University)<-[:studyAt]-(p:Person) RETURN p.firstName AS name MATCH SIMPLE (x)->{2}(y) RETURN COUNT(*) AS path_count MATCH ACYCLIC (x:`node` {id: 1})->(y)->(z) RETURN x.id, y.id, z.id ORDER BY x.id, y.id, z.id MATCH DIFFERENT EDGES (p:Person {name: 'Keanu Reeves'})-[e:ACTED_IN]->(m:Movie) RETURN m.title AS movie Unbounded variable-length path patterns Variable-length paths previously required a maximum hop count. Unbounded quantifiers allow the graph structure to determine the traversal depth when no meaningful fixed limit exists. The example follows one or more relationships, while ACYCLIC prevents the traversal from visiting the same node twice. Unbounded refers to the path expression, not unlimited compute, so use selective starting points and test performance with representative graph density. MATCH ACYCLIC ()-[]->{1,}() RETURN COUNT(*) AS total Inspect paths and graph elements NODES, EDGES, and INNER_NODES These functions expose the contents of a matched path. NODES returns all nodes in the path, EDGES returns all relationships in the path, and INNER_NODES returns the intermediate nodes without the endpoints. Applications can use these results to explain lineage, dependencies, and investigative findings instead of returning only a destination. MATCH p=({id: 1})-[]->{0,4}(z) RETURN NODES(p) AS res_nodes, INNER_NODES(p) AS res_inner_nodes ORDER BY z.id ELEMENT_ID ELEMENT_ID returns the identity of a graph element. It is useful when an application must compare, track, or reference a specific node or edge independently of its business properties. MATCH (x) RETURN ELEMENT_ID(x) AS vid, x AS actual_id ORDER BY x.id LIMIT 5 SOURCE and DESTINATION SOURCE and DESTINATION identify the endpoints of an edge. This makes directional checks and edge-centric analysis easier, especially when a query starts from a relationship rather than a node. MATCH (x)-[e]->() RETURN x.id, SOURCE(e).id AS s ORDER BY e.src, e.dst LIMIT 10 MATCH ()-[e]->(x) RETURN false IN COLLECT_LIST(x = DESTINATION(e)) AS no Compose larger queries NEXT NEXT passes the output of one query stage into the next, allowing analysts to express a multistep question in a single query. For example, they can calculate a measure, filter the result, traverse another relationship, and summarize the final result without moving intermediate data into application code. MATCH (m:Message)-[:hasCreator]-(p:Person) RETURN p, AVG(m.length) AS avg_msg_length GROUP BY p NEXT FILTER avg_msg_length > 100u MATCH (p)-[:knows]->(:Person) RETURN avg_msg_length, COUNT(*) AS connections GROUP BY p, avg_msg_length EXISTS subqueries EXISTS answers a yes-no question about another graph pattern. It can filter rows or participate inside an aggregate without requiring the outer query to return the matching details. MATCH (x) WHERE EXISTS { MATCH (y) WHERE y.id = 5 RETURN y } RETURN x.id AS xid ORDER BY xid MATCH (x) RETURN COUNT(*) FILTER ( WHERE EXISTS { MATCH (x)-[]->(y) RETURN y } ) AS count_with_out CALL subqueries CALL runs a contained query for each current outer result. In the example, it calculates each node’s outgoing degree and returns that value with the node identifier, keeping the intermediate calculation contained within the query. MATCH (x) CALL { MATCH (x)-[e]->(y) RETURN COUNT(e) AS out_degree } RETURN x.id AS xid, out_degree ORDER BY xid OPTIONAL CALL OPTIONAL CALL preserves the outer row when the contained query has no match. This supports enrichment scenarios where related information is useful but not required. MATCH (x {id: 1}) OPTIONAL CALL { MATCH (y)-[]->(x) RETURN y.id AS yid } RETURN x.id AS xid, yid ORDER BY yid UNION DISTINCT UNION DISTINCT combines compatible result sets and removes duplicate rows. It complements UNION ALL, which preserves duplicates, so the query can choose the correct set behavior. MATCH () LIMIT 3 RETURN 42 AS x UNION MATCH () LIMIT 3 RETURN 100u AS x Summarize connected data Filtered and paged aggregates Filtered aggregates calculate a summary over a selected subset. Paging controls such as LIMIT can be applied within the aggregate, giving developers precise control over which values contribute to the result. MATCH (m:Movie) RETURN COUNT(*) FILTER (WHERE m.released < 1980 LIMIT 10) AS filtered_count Nested aggregations Nested aggregation supports layered questions, such as finding the minimum of averages calculated across groups or paths. This reduces the need to move intermediate results into application code. MATCH (a:`node`)-[e:`edge`]->{1,2}(b:`node`) RETURN a.id AS id, MIN(AVG(e.weight)) AS min_average GROUP BY id COLLECT_ONE, COLLECT_LIST, and COLLECT_ELEMENTS Collection functions retain values or graph elements for evaluation as a group. COLLECT_ONE selects one value, COLLECT_LIST builds a list, and COLLECT_ELEMENTS gathers graph elements. They are useful for path-level checks, ordered output, and analysis across repeated relationships. MATCH (p:`node` {id: 1})-[e]->{2}(p1:`node` {id: 1}) RETURN (COLLECT_ONE(e.weight) IN [e[0].weight, e[1].weight]) AS contained MATCH ()-[e]->{1,2}() RETURN COLLECT_LIST(ALL e.weight) AS w ORDER BY w LIMIT 20 MATCH ()-[a:ACTED_IN]->{0,4}() WHERE SIZE(COLLECT_ELEMENTS(DISTINCT a.roles)) > 4 RETURN TO_JSON_STRING(a.roles) AS r Work with values and text Dynamic nested data Support for dynamic nested data lets queries work with richer values that do not fit a flat property model. This is useful when graph entities contain structured application payloads, nested attributes, or evolving schemas. CAST CAST converts a value to a supported target type. Explicit conversion makes comparisons and calculations more predictable when source data contains different numeric types. RETURN CAST(42 AS DOUBLE) AS double_res, CAST(42 AS UINT) AS uint_res, CAST(42.5d AS INT) AS int_res RANGE RANGE creates a numeric sequence. A generated sequence can support iteration, testing, bucketing, and list-oriented calculations within a query. RETURN RANGE(0, 4) AS res POWER POWER performs exponent calculations. It extends the numeric expressions available for scoring, weighting, and mathematical transformations. RETURN POWER(-2, 3) AS res NORMALIZE and IS NORMALIZED Unicode text can have visually identical but technically different encodings. NORMALIZE converts text to a normalization form, while IS NORMALIZED checks its current form. This supports more consistent matching and grouping across multilingual data. RETURN NORMALIZE('cafe\u0301') AS normalized_text RETURN 'cafe\u0301' IS NFKD NORMALIZED AS is_normalized Regular-expression functions The MSFT.REGEXP functions match, count, locate, extract, and replace text patterns. They help prepare identifiers, names, codes, and descriptions before those values are compared or summarized. RETURN MSFT.REGEXP_LIKE('café', 'caf.') AS x RETURN MSFT.REGEXP_COUNT('abcabcabc', 'abc') AS n RETURN MSFT.REGEXP_INSTR('hello world', 'world') AS p RETURN MSFT.REGEXP_REPLACE('a1b2c3', '[0-9]', '#') AS s RETURN MSFT.REGEXP_SUBSTR('abc123def', '[0-9]+') AS s Design guidance Use the narrowest pattern that answers the question. Start from selective nodes, specify relationship and node labels, and apply filters early. Use bounded traversal when the business question has a natural maximum. When depth is unknown, combine an unbounded path with an appropriate path mode and test it against representative graph density. Return only the properties or path elements the application requires. Feature availability can vary by environment and release stage. Confirm the release stage and regional availability of each feature before publication. If availability differs by feature, identify the applicable release stage in each section. Next steps Start with a connected-data scenario such as lineage, dependency analysis, recommendations, investigations, or knowledge discovery. Identify the relationships the question must follow, choose the path mode and traversal depth that match the scenario, and then use composition and aggregation features to shape the result. Graph in Microsoft Fabric overview GQL language guide for graph in Microsoft Fabric GQL graph patterns GQL standard conformance for graph in Microsoft Fabric Optimize GQL query performance for graph in Microsoft Fabric
Lineage-aware AI with the Fabric item relations API (Preview)
With two new REST operations, you can retrieve the upstream and downstream relations of any Fabric item directly from your own code — the dependency graph behind the portal's lineage view, now available as a supported, programmable surface. Each response includes related items, the typed relation edges that connect them, and the workspaces those items belong to. Lineage has always answered two human questions: “where does this data come from?” and “what breaks if I change it?” This API lets your tools — and your AI agents — ask those same questions programmatically. What’s new Until now, item lineage in Fabric was something you explored visually in the portal. You opened an item, looked at its lineage view, and traced dependencies by eye. That’s great for people, but it’s difficult to automate. With this update, lineage becomes an API. After authenticating to Fabric, you can programmatically: Get the downstream relations of an item — everything that depends on it (its consumers and impact radius). Get the upstream relations of an item — everything it depends on (its sources). Read the typed relation edges between items (for example, Shortcut, PushData, Orchestration). Resolve related items across workspaces, using the workspace list returned alongside the graph. Feed lineage into impact analysis, documentation, data catalogs, CI/CD checks, and AI agents. Why this matters If you only work in the Fabric portal, the lineage view already serves you well. The impact of this update shows up once you need to automate or scale — when understanding dependencies becomes part of a workflow rather than a manual click-through. A few examples: Before you delete or reroute a dataset, you call the downstream API to see every report, semantic model, and pipeline that would be affected. You build an internal catalog or documentation site that shows each item’s sources and consumers, kept fresh automatically. You add a CI/CD check that fails a deployment if a change would break a downstream dependency. You are an ISV building on Fabric, and you want lineage to be part of your product experience — not a side trip into the portal. That’s what a public API is for: turning a visual experience into a building block you can automate and compose. Lineage is context for AI AI agents are only as good as the context they are given. When an agent answers a question about a table, a report, or a metric, it benefits enormously from knowing where that data came from and what depends on it. Lineage is exactly that context. With the relations API, an agent can traverse an item’s dependency graph as part of its reasoning. Ask “is it safe to change this table?” and the agent can call the downstream API, enumerate the affected items, and ground its answer in the real graph instead of guessing. Ask “where does this number come from?” and it can walk upstream to the source. In other words, lineage helps ground AI responses in the actual dependency graph rather than inferred relationships. This is the same pattern we see across Fabric’s AI story: take governance metadata your organization already maintains — like sensitivity labels — and share it with AI so agents understand your data the way your organization does. Lineage joins that toolkit. It gives agents dependency awareness: the ability to reason about cause, effect, and blast radius, not just content. How it works At a high level, the API exposes both sides of the dependency graph: what an item depends on and what depends on it. The API adds two GET operations under the platform surface. Because the API is in preview, every call currently requires beta=true as a query parameter. GET /v1/workspaces/{workspaceId}/items/{itemId}/relations/downstream?beta=true GET /v1/workspaces/{workspaceId}/items/{itemId}/relations/upstream?beta=true The caller needs read permission on the item. Both user and service principal identities are supported. Each response is a small graph made of three lists: items — every item in the returned graph, including the item you queried, so each relation endpoint can be resolved (id, type, display name, and the workspace they belong to). relations — the edges, each with a source item, the item it depends on, and a relation type. workspaces — the workspaces referenced by those items, so you can resolve names across workspace boundaries. A downstream response for a semantic model consumed by a report looks like this: { "items": [ { "id": "3546052c-...", "type": "Report", "displayName": "Q4 Sales Dashboard", "workspaceId": "cfafbeb1-..." }, { "id": "9b218778-...", "type": "SemanticModel", "displayName": "Sales Semantic Model", "workspaceId": "cfafbeb1-..." } ], "relations": [ { "itemId": "3546052c-...", "dependentOnItemId": "9b218778-...", "relationType": "Association" } ], "workspaces": [ { "id": "cfafbeb1-...", "displayName": "Finance Analytics Workspace" } ] } The relation types describe how two items are connected. The set is extensible, so new types can be added over time: Relation type What it means Association The item consumes the dependency item — for example, a report built on a semantic model. Shortcut The item references data through a OneLake shortcut to another item. PushData The item writes or pushes data into the dependency item. Orchestration The item runs or manages execution of the dependency item (for example, a pipeline). Datasource The item reads from the dependency item as a data source — for example, a notebook reading a lakehouse. CascadeDelete A parent–child relationship; deleting the parent deletes the dependent. WeakAssociation A soft dependency that is removed if the dependent item is deleted. HiddenInWorkspace A dependency on an item that isn't surfaced in the workspace list, such as a staging artifact. Getting started A great first step is to pick a familiar item and explore its downstream relations to understand its impact radius. The following is the shape of a first call using the Azure CLI for authentication: 1. Authenticate to Fabric and get a token. $token = az account get-access-token ` --resource "https://api.fabric.microsoft.com" ` --query accessToken -o tsv 2. Call the downstream relations API for an item. GET https://api.fabric.microsoft.com/v1/workspaces/{workspaceId} /items/{itemId}/relations/downstream?beta=true Authorization: Bearer $token 3. Read the relations array to list what depends on the item, then walk upstream from any related item to trace it back to its sources. From there, wire the results into whatever needs dependency awareness — an impact-analysis check, a catalog page, or an AI agent’s context. You can find the operations in the Fabric REST API reference under the platform items surface: Items - Get Downstream Relations (beta) - REST API (Core) | Microsoft Learn Items - Get Upstream Relations (beta) - REST API (Core) | Microsoft Learn Exposing item lineage as a programmable API is a foundational step. It turns the dependency graph into something your tools, pipelines, and agents can read and reason over. We look forward to seeing what you build with it. Note: This API is in preview and provided for evaluation and development purposes. It may change based on feedback and is not recommended for production use.Use User Data Functions to securely call Fabric REST APIs
User Data Functions (UDFs) provide a secure, reusable logic layer for calling Microsoft Fabric REST APIs through a Fabric client. By centralizing authentication, authorization, and orchestration logic in a function, you can expose Fabric operations without embedding credentials or API logic throughout your applications. After a function authenticates with a service principal, it can perform only the operations allowed by that identity permissions and scopes. Combined with Azure Key Vault and Fabric generic connections, this approach helps keep secrets out of your code while maintaining explicit, governed access to Fabric resources. Overview This post explores a recommended pattern for calling Fabric REST APIs from a User Data Function, why service principal authentication matters, and how to securely manage credentials using Key Vault. While this post demonstrates triggering a pipeline, the same pattern can be used to: Create and manage Fabric items Manage workspaces Run jobs Perform capacity operations Call other supported Fabric REST APIs Before exploring implementation, we’ll review three key principles that make this pattern secure, reusable, and governable. Functions become a secure Fabric API gateway Instead of hardcoding orchestration logic across apps, notebooks, pipelines, or reports, you can centralize it in a function: The function decides what operation to run. The service principal provides controlled access to Fabric. The Fabric client calls the underlying REST API. The same pattern can call any supported Fabric operation The function becomes a reusable entry point for Fabric automation across Power BI, apps, notebooks, APIs, and other Fabric experiences. Run jobs, including pipelines. Read or update workspace metadata. Create, update, or manage Fabric items. Perform capacity read/write operations. Call other public APIs allowed by the service principal’s scopes. Permissions stay explicit and governed Because the function uses a service principal, access is governed by the permissions assigned to that identity. This gives you a clean separation: the function owns the business logic, the service principal controls access, and Fabric executes the operation. Trigger a pipeline with UDF Before you start, ensure you have: A Fabric workspace A User Data Function (UDF) A service principal with the required Fabric permissions An Azure Key Vault that stores the service principal secret A Fabric generic connection configured for Key Vault access Architecture pattern At a high level, the pattern works as follows: App or Fabric workload invokes the UDF. UDF retrieves the service principal secret from Azure Key Vault. UDF creates a Fabric client using the service principal (SPN) credentials. Fabric client calls the Job Scheduler API to trigger the pipeline. Results are returned to the caller. Sample code At a high level, the implementation uses three functions: a helper function to build a FabricClient using service principal credentials, a helper function to trigger a Fabric Data Pipeline through the SDK, and a main UDF that brings everything together by retrieving the service principal secret from Key Vault, creating the Fabric client, and invoking the pipeline. Checkout full sample for this scenario. def invoke_pipeline_with_spn( keyVaultClient: fn.FabricItem, workspaceid: str, pipelineid: str, tenantid: str, clientid: str, keyVaultUrl: str, spnSecretName: str, ) -> str: try: logging.info("Fetching secret from KV") # Step 1: Get the Key Vault credential from Fabric connection credential = keyVaultClient.get_access_token() client = SecretClient(vault_url=keyVaultUrl, credential=credential) spn_secret = client.get_secret(spnSecretName).value logging.info("Fetched secret from KV") # Step 2: Create FabricClient with SPN credentials fabric_client = _create_fabric_client( tenantid=tenantid, clientid=clientid, client_secret=spn_secret, ) logging.info("Created Fabric client") # Step 3: Trigger the pipeline using FabricClient SDK result = _run_fabric_pipeline( fabric_client=fabric_client, workspaceid=workspaceid, pipelineid=pipelineid ) logging.info("Ran Fabric pipeline") return json.dumps(result, indent=2) except Exception as e: logging.error(f"invoke_pipeline_with_spn failed: {str(e)}") return json.dumps({"status": "error", "message": str(e)}) Troubleshooting tip: If you run into issues, check if SPN has access to the workspace and if the Key Vault permissions are set correctly. Real-world example: Automate project workspace setup in Fabric For example, an internal project onboarding app can use a User Data Function to automate setup for a new project in Fabric. When a business user requests a project, the app invokes the function with the project name, owner, environment, capacity requirements, and project preferences. The function then uses Fabric REST APIs to spin up the project easily. The app invokes the User Data Function with project metadata, workspace configuration, and capacity target. The function retrieves the service principal secret from Key Vault and creates a permissioned Fabric client. The Fabric client creates the workspace and configures capacity. Once the workspace is provisioned and configured, project teams can immediately begin creating Fabric assets such as lakehouses, warehouses, notebooks, and pipelines. Next steps User Data Functions provide a secure and reusable way to expose Fabric automation behind governed APIs. By combining service principals, Azure Key Vault, and the Fabric SDK, you can centralize operational logic while keeping credentials protected. Ready to get started? Explore the User Data Functions documentation and adapt this pattern to your own Fabric automation scenarios.
