Your Data Is Everywhere, Now What? Building A Modern Data Platform.
Microsoft Fabric User Group Accra: A Ghanaian company has data in Excel, SQL databases, APIs, operational systems and applications. Different teams have different numbers. Reports take days. Nobody trusts the dashboard. As far as you make decisions based on data, this is for you. This session introduces the Fabric ecosystem and demonstrates how the different workloads work together. Attendees will build a simple end-to-end solution that takes raw data, stores and transforms it in a Lakehouse, and exposes it for analysis. Learning objectives By the end of the session, attendees will be able to: Explain what Microsoft Fabric is and where it fits in the Microsoft data ecosystem. Identify the major Fabric workloads. Understand the role of OneLake. Distinguish between Lakehouse, Warehouse, and other Fabric workloads. Create a Fabric workspace. Build a basic data-to-insight workflow.
SAP and Microsoft accelerate business insights and AI innovation with SAP Business Data Cloud Connect for Microsoft Fabric
Coauthors: Irfan Khan, President & Chief Product Officer, SAP Data & Analytics and Arun Ulag, President, Azure Data Today, at Microsoft Ignite in San Francisco, SAP and Microsoft unveiled plans to expand their longstanding partnership with the launch of SAP Business Data Cloud Connect for Microsoft Fabric. The new capability simplifies access to semantically rich SAP data products through bi-directional, zero-copy sharing with Microsoft Fabric, enabling enterprises to gain instant access to trusted, business-ready insights for advanced analytics and AI. “We are excited about continuing to strengthen our partnership with Microsoft to create more value for our customers,” said Muhammad Alam, member of the Executive Board of SAP SE, SAP Product & Engineering. “By bringing SAP Business Data Cloud and Microsoft Fabric closer together, our customers can seamlessly leverage the power of data to generate real business value through AI and analytics.” “Organizations across every industry are accelerating their AI transformation by bringing together data from operations, analytics, and applications,” said Scott Guthrie, Executive Vice President, Microsoft Cloud and AI. “With SAP Business Data Cloud and Microsoft Fabric, we’re delivering a trusted foundation for analytics and AI, and helping our customers move faster, make smarter decisions, and turn insight into real business outcomes.” Unlocking the True Potential of your Enterprise Data SAP BDC Connect for Microsoft Fabric empowers organizations to fully harness their data and applications by delivering secure, rapid access to SAP data products at scale—without the delays of data replication. Through bi-directional, zero-copy sharing between SAP Business Data Cloud and Microsoft Fabric, customers can realize use cases that previously required moving and managing copies of data. SAP data products will be seamlessly integrated into Microsoft OneLake, Microsoft Fabric’s AI ready data lake, and data sets shared from Microsoft OneLake will also be available in SAP Business Data Cloud to supplement intelligent applications. By utilizing Fabric’s data engineering, data warehousing, and Power BI capabilities, organizations can effectively integrate SAP data with their broader ecosystem, establishing a unified foundation for their enterprise data. OneLake integration into Microsoft AI Foundry can also help customers leverage their SAP data in building AI applications, and as OneLake is built into Microsoft 365, hundreds of millions of users can get secure access to their SAP data through the products they use every day such as Excel and Teams. Accelerating Business Insights & AI with SAP and Microsoft Fabric SAP BDC Connect for Microsoft Fabric enables a unified data foundation that helps organizations get insights faster and accelerate their AI strategy. Through this bi-directional integration, customers can: Build a semantically rich data foundation on harmonized SAP and non-SAP data Perform advanced analytics and interact with enterprise data in natural language with Copilot in Microsoft Power BI. Develop intelligent AI applications and agents grounded in mission-critical business data with Fabric data agents, Copilot studio, and AI Foundry. Enable multi-agent collaboration between M365 Copilot and SAP Joule, leveraging a unified enterprise data and productivity platform to provide a seamless experience to business users. Availability SAP Business Data Cloud Connect for Microsoft Fabric is planned to be generally available in Q3 2026.Fabric Data Warehouse best practices for medallion architectures
Part three of a series on medallion architecture with Fabric Data Warehouse. Good medallion architecture is mostly operational discipline. In part one of this series, we chose the pattern, and in part two, we filled in the Bronze, Silver, and Gold layers. Now comes the part that usually determines whether the architecture holds up in production: the operating rules. Many medallion architectures look great on paper but become difficult to maintain as data volumes, business requirements, and consumers grow. A medallion pipeline is easy to explain and easy to demo. It is more challenging to keep clean over time. The challenges usually start small: row-by-row loads, report-specific logic in the wrong place, transformations that cannot be safely rerun, or Gold tables that slowly become another staging layer. Why this matters Most medallion problems are not caused by the names Bronze, Silver, and Gold. They happen because the pipeline stops behaving like a pipeline. Bronze starts cleaning. Silver starts serving dashboards. Gold starts compensating for upstream data quality. Before long, nobody knows where a rule belongs, and every change feels risky. The goal of best practices is not to add ceremony. The goal is to make the pipeline predictable: predictable loads, repeatable transformations, trusted outputs, and clear places to look when something breaks. Best practice 1: Batch the writes Fabric DW is built for set-based work. Treat ingestion and transformations as batches, not as a stream of tiny row-by-row operations. In Bronze, that usually means using COPY INTO, Fabric Pipelines, or other bulk-loading patterns to land data in raw tables. If you are loading from files, aim for fewer well-sized files instead of many tiny ones. When practical, files in the 100 MB to 1 GB range are a healthier starting point than a long tail of small files. In Silver and Gold, the same idea applies: prefer set-based T-SQL transformations, CTAS, INSERT...SELECT, and MERGE patterns over procedural row-at-a-time logic. Do this well Load Bronze in batches, and avoid trickle inserts when the source can be staged first. Add ingestion metadata, such as source file name and load timestamp, so every batch is traceable. Keep operational logging lightweight. If you need very high-write audit events, do not turn the warehouse into a single-row logging engine. Let Bronze preserve the batch; let Silver decide what is valid. Rule of thumb: if a load pattern creates a large number of tiny writes, fix the load pattern before tuning the query. Best practice 2: Make Silver rerunnable Silver is where the pipeline earns trust. That means Silver transformations need to be repeatable, testable, and safe to rerun. If an upstream source reloads, or a cleansing rule changes, you should know how to rebuild the affected Silver tables without guessing which reports need to be patched. This is where idempotent design matters: a transformation should produce the same result when run again against the same inputs. In Fabric DW, use CTAS when you want to materialize a clean table from a query, INSERT...SELECT for controlled incremental loads, and MERGE when late-arriving or changed data needs to update existing Silver rows. Do this well Design transformations so they can run again without duplicating or corrupting data. Use staging tables when the logic is complex. A few clear steps are easier to operate than one unreadable query. Put quality gates in Silver: required fields, valid formats, duplicate handling, and reason codes for rejected records. Choose precise data types and lengths. Silver is the right place to turn loose source data into reliable analytical data. Rule of thumb: if a report needs to clean the data again, Silver did not finish its job. Best practice 3: Shape Gold for consumption Gold is not just “the final table.” Gold is the business-facing serving layer. It should be modeled around how people ask questions, not around how the source systems store data. For some workloads, that means a star schema with fact and dimension tables. For others, it means a data mart, a wide reporting table, or a pre-aggregated summary. The pattern matters less than the principle: Gold should make the common analytical path simple, fast, and trustworthy. This is also where you should be careful not to let Gold become a junk drawer. If Gold is full of one-off fixes, report-specific exceptions, and raw technical fields, the layer is doing too much. Do this well Model the grain explicitly. A fact table without a clear grain becomes hard to explain and harder to debug. Pre-aggregate where the business repeatedly asks the same question. Hide technical fields that helped the pipeline but do not help the consumer. Keep Gold dependent on Silver by default. Direct Gold-to-Bronze dependencies should be rare and deliberate. Rule of thumb: Gold should answer the business question quickly without making the report author rediscover the pipeline. Best practice 4: Use Fabric DW defaults, but do not fight the engine Fabric DW gives you a SQL warehouse over Delta data in OneLake. That means you get transactional behavior, optimized storage patterns, and a managed engine that handles many physical decisions for you. The practical advice: do not bring every habit from traditional data warehousing with you. You do not need to micromanage distribution or indexing the same way you would in older platforms. Focus first on healthy data layout, set-based transformations, good table design, and predictable query patterns. At the same time, do not ignore the basics. Query performance still benefits from clean data types, useful statistics, well-shaped Gold tables, and avoiding unnecessary scans. Do this well Keep V-Order and platform optimizations on unless you have a measured reason to change them. Use the performance guidance for Fabric Data Warehouse before inventing custom tuning patterns. Check query behavior when a Gold table becomes critical to many reports. Treat advanced exceptions as exceptions. Most teams should start with the defaults and tune only when evidence says to tune. Rule of thumb: tune from evidence, not from habit. Best practice 5: Monitor by layer A medallion pipeline should be observable at each layer. If a dashboard is wrong or slow, you should be able to tell whether the issue started in Bronze ingestion, Silver transformation, or Gold serving. In Fabric DW, use Query Insights and the warehouse monitoring views to understand query behavior, expensive operations, and refresh patterns. Pair that with pipeline-level monitoring so you can see not only whether a job failed, but where the failure happened. Measure the pipeline in terms the team can act on: Bronze load duration, Silver transformation duration, rejected-record counts, Gold refresh duration, and Gold query performance. Do this well Track load and refresh duration by layer. Log the number of records received, accepted, rejected, and published. Watch critical Gold queries after refresh, especially the ones that feed executive dashboards or widely used semantic models. Keep operational alerts tied to business impact. A failed Gold refresh matters differently from a delayed Bronze load. Rule of thumb: if you cannot tell which layer failed, your monitoring is not layer-aware enough. Before moving a medallion pipeline into production, use the following checklist to verify that each layer is operating as intended. The best-practice checklist Area Bronze Silver Gold Write pattern Batch ingest with metadata Set-based transformations Scheduled refreshes Quality rule Preserve what arrived Validate, conform, and flag Expose trusted fields only Performance focus Avoid tiny writes Keep logic rerunnable Shape for common queries Takeaway Part two of this series was about one job per layer. Part three is about operating each job like it matters. Batch the writes. Make Silver rerunnable. Shape Gold for consumption. Use Fabric DW’s managed engine instead of fighting it. Monitor the pipeline by layer so failures are easy to locate and fixes happen in the right place. If you follow those rules, your medallion architecture becomes less fragile over time, not more. This post is part of our Medallion Architecture on Fabric Data Warehouse series: Choosing your medallion pattern in Fabric Data Warehouse Building the Bronze → Silver → Gold layers Fabric DW best practices for medallion architectures Securing and governing your layers Performance tuning your medallion pipeline Ready to go deeper? Explore the Microsoft Fabric Data Warehouse performance guidelines and ingestion guidance, then stay tuned for Part four of this series, where we’ll cover securing and governing your layers.Manage Fabric connections at scale with connection recency in Fabric REST APIs
Understand Connection Recency Connections are shared infrastructure in Microsoft Fabric. Pipelines, dataflows, semantic models, and other Fabric items use them to reach data sources without storing connection details and credentials in every item. Connection Recency adds the context administrators need to understand how each connection is being used. If you go to “Manage Connections and Gateways” in Fabric settings and open the settings of any connection, the following information card will populate: The card provides three insights: Created on shows when the connection was created. Last Bound to items shows when an item was most recently associated with the connection. By any user reflects activity across the tenant, and By me is specific to the viewer. Last credentials used shows when a workload most recently used the connection credentials. It also separates activity By any user from activity By me, which is the current user. Linking and usage are different events. A connection can be linked but never used, or it could have been linked a long time ago to an item that still runs every day. Credential-use information can also be up to 30 minutes behind real time, so use it for governance and investigation rather than immediate usage monitoring. Learn more in the Data source management documentation. Why enterprise connection management becomes difficult Connections can grow exponentially across a large organization. Every team, project, environment, gateway, and data source can introduce more connections. Over time, names become inconsistent, employees move on, and multiple connections can point to the same endpoint. The Fabric interface is useful for inspecting one connection, but reviewing recency and ownership one connection at a time does not scale to an enterprise inventory. This creates three common problems: Stale connections: A connection may no longer support an active workload, but its name, creation date, and Last Bound Date do not prove that it is unused. Administrators need both Last Bound Date and Last Credential Use signals before deciding what to review or retire. Duplicate connections: Different display names can hide equivalent connection definitions. Duplicates increase credential maintenance, complicate troubleshooting, and make it harder to establish a preferred connection which dilutes the advantage of connection sharing and reuse across a tenant. Ownership risk: A connection with only one person as an owner can become orphaned when that person leaves the organization. The connection might then be unavailable for administration or no longer usable by dependent workloads. The REST APIs make it possible to evaluate these risks across every connection the caller can access. In the case of cloud connections, you can only see connections where you are the owner, but the admin for a gateway can see all connections on the gateway. This provides better scalability than the UI. Using the APIs to manage connections You can use the APIs to identify stale and duplicate connections for removal and to help mitigate the issue of single user ownership. Gateways in Fabric have a limit of 1,000 connections, so managing and governing your connections will help you stay under that limit. Now that we understand each of the three scenarios, let’s see what the API can provide to help us handle each scenario: Stale connections: LastBoundDateTime is the first of two signals we can use. If this is NULL, then it is currently not being used by a Fabric item and would be a candidate for removal. The one catch is that connections bound prior to recency being introduced will also show NULL, so we must filter out anything with a CreatedDateTime earlier than the introduction of the recency feature. Recency came out in preview at the end of March 2026, so my example code will filter anything created prior to May 1, 2026, to play it safe. The second signal we have is the LastCredentialUsedDateTime, which indicates the last time the connection was used by any item. You can change this in the example code, but I’m going to use 90 days. You might have connections that legitimately get used less frequently, such as year-end runs or only during the holiday season, so keep that in mind when reviewing and adjust accordingly. Duplicate connections: It is possible for two connections to have identical configurations except for the DisplayName. This defeats the purpose of having shareable connections, so we’ll need to use the LastCredentialUsedDateTime to determine which one was most recently used and flag any others to be considered for removal or consolidation. We’ll use ConnectionType, ConnectionPath, ConnectivityType, and GatewayID to identify duplicates. There are other columns you may want to add, such as CredentialType or ConnectionEncryption, depending on what you consider duplicate in your specific environment. Ownership risk: For ownership continuity, you want to identify connections whose only Owner role assignment is a User, then add another approved owner. This helps avoid having an orphaned connection if an owner gets deleted. Adding a Microsoft Entra group is preferred because group membership can be maintained as people join, leave, or change responsibilities. A second individual owner is better than a single owner, but it does not provide the same durable operating model as group ownership. MYTH: A common misconception is that a connection fails when its owner account is deleted. In reality, what matters is the credential being used for authentication. When you create a connection that uses OAuth, you become the owner and your OAuth credential is typically used. If your account is later deleted, the connection fails because it can no longer authenticate with that credential, not because ownership changed. If the connection instead uses an SPN and is shared with another user, it will continue to work. Explore the List Connections, List Connection Role Assignments, and Add Connection Role Assignment documentation for the complete request, pagination, identity, and permission details. Next steps Run the companion notebook first. The notebook is designed as a Python notebook to be run in the Fabric Python runtime and not the PySpark Python runtime. There will be a section for each of the previous scenarios. Stale and duplicate connections will be shown so you can review before deciding what to remove, but that part will be up to you. You can use the data frame and pass the connection ID values to the Delete Connection API. If you don’t know how to write that part, then it’s a great opportunity to use Copilot. The ownership section will show you all connections with a single owner where you will want to add another owner. As a best practice, we’ll assume you are adding a group, but that part will be commented out as you very likely do not want to blatantly add the same group to every connection across the board. You can use the Add Connection Role Assignment API to make those additions as you need. Connection recency data gives you the signals you need to manage connections at scale, but it should be the starting point for review rather than the basis for automatic removal. Use the companion notebook to identify potentially stale or duplicate connections and connections with a single owner, then apply your organization’s requirements before taking action. Regularly reviewing these signals can help you reduce unnecessary connections, improve ownership continuity, and establish a more manageable connection inventory.Modern Power BI architecture choices for reporting on Azure Databricks: A performance benchmark for Power BI storage modes
Many enterprise Power BI semantic models use Azure Databricks as a data source. When building these models, developers and architects face an early and consequential decision: which storage mode to use. Cost, security, and ease of development and tuning all factor in — but report performance is probably the most important of them, because reports that are slow to load are one of the most common causes of end-user dissatisfaction. In practice, that decision is often made on intuition rather than evidence. To help change that, we've published a new white paper, Modern Power BI Architecture Choices for Reporting on Azure Databricks, benchmarking four ways of serving the same Delta tables to a Power BI report: Direct Lake on OneLake — over Delta tables in a Fabric lakehouse or warehouse Direct Lake on mirrored Unity Catalog tables — shortcuts, no copy DirectQuery — on a Databricks SQL warehouse Composite Model on Databricks — DirectQuery combined with Import-mode aggregations Figure: The four Power BI storage modes benchmarked to evaluate their impact on report performance and scalability. What the results suggest: there's no universal winner — but there are clear patterns. Direct Lake on OneLake performed well across the widest range of situations in this benchmark. It's highly competitive at smaller and mid-size volumes, and for the typical Power BI workload — where reports are used repeatedly throughout the day — it delivers interactive performance without extra modeling effort. At the top end of the volume curve, the picture shifts. With billions of rows, a Composite Model with aggregations was the fastest and most consistent pattern, staying sub-100ms on queries the aggregation tables can resolve. The caveat is equally clear: that advantage doesn't extend to queries that fall through to the underlying DirectQuery source, so the payoff depends on how well your aggregations match real user behavior. These are just the headline findings — the detailed results vary considerably by data volume, cache state, filter scenario, and query type. We'd encourage you to read the white paper for the full picture before deciding on a pattern. One thing worth noting up front: this is a point-in-time comparison as of June/July 2026, and both platforms are moving quickly. It also measures the end-user query experience within Power BI, rather than raw database execution speed. Treat it as a guide for running your own testing, on your own data. Explore the white paper for a full walkthrough of through each pattern in detail — the test setup, what was measured, and how results break down by data volume, cache state, and query type. Visit the Power BI download center to access the paper and other related resources.Connect securely to Snowflake in Fabric Pipelines and Copy Jobs
Snowflake is a common source and sink for enterprise data movement, and Microsoft Fabric Data Factory supports it in both pipelines and Copy Jobs. For organizations that restrict public network access, some additional considerations are needed to meet that requirement. You can have private inbound and outbound access to Fabric, a private staging storage account, and a private Snowflake instance. By combining Workspace-level Private Link, a private Azure Storage account, and Snowflake's storage integration with USE_PRIVATELINK_ENDPOINT, you can keep the path entirely private. This post explains how those controls fit together for Fabric Data Factory pipelines and Copy Jobs. When this might apply Your organization might require private networking, where every hop needs to be inspectable and privately routed. Regulated workloads in financial services, healthcare, and public sector regularly require that analytics platforms have no public endpoints, that inbound and outbound traffic to storage stays inside a VNet, and that any third-party SaaS integration uses private links where supported. Snowflake supports Azure Private Link for storage endpoints and Fabric supports Workspace-level Private Link for inbound workspace access which can be combined to achieve a private path. Using each control for its intended direction avoids public exposure. The challenge The Snowflake connector in Fabric Data Factory uses Snowflake's native COPY command under the hood and requires an interim storage account for data staging. Currently, staging is not supported by Onelake when using Workspace-level Private Link, so an external Azure Storage account can be used instead. Staged copy supports source or destination shapes that aren't natively compatible with Snowflake's COPY command and can improve throughput. The catch is the Azure Storage account sits between two systems that each have their own network story: Fabric on one side and Snowflake on the other. Locking down the copy path end-to-end means securing all three surfaces — the Fabric workspace, Azure Storage staging account, and Snowflake. The core model Think of the topology as two independent network paths that meet at the staging account. Figure: Private Path for Snowflake in Fabric Pipelines and Copy Jobs. Inbound Access to the Fabric Workspace — Workspace-level Private Link maps the workspace to an approved VNet and lets you deny public inbound access. It doesn't place the Fabric workspace inside your VNet or govern outbound data-source traffic. Azure Blob Staging — Trusted Workspace Access provides a private connection from the Fabric Workspace to your storage account. Not only is it private, but it uses a Workspace Identity to authenticate and a Resource Instance rule to only allow specific workspaces. Snowflake to Staging Storage — Snowflake provisions a private endpoint into your Azure tenant against the same staging account. The Snowflake storage integration is created with USE_PRIVATELINK_ENDPOINT = TRUE so Snowflake's own copy operations honor the private path. Public Internet Access — Public network access on the storage account and Fabric Workspace is blocked. Network access and authorization are separate. Snowflake's storage integration provisions a service principal in Azure. This SPN will need Storage Blob Data Contributor permission on the staging storage account. The Fabric Azure Blob connection uses the authentication configured for that connection, such as workspace identity. That identity will need Storage Blob Data Reader on the staging storage account. Setup The full configuration breaks into five phases: Build the Network Foundation. Create a workspace, Azure private link service, virtual network, virtual machine, private endpoint from the workspace to the private link service and finally turn off public networking to the workspace. Complete instructions for all these items can be found in the Set up and use Workspace-level Private Links documentation. Create Staging Storage Account. Use these instructions to create an Azure storage account. Once you have an account you need to configure Trusted Workspace Access for private networking between your Fabric Workspace and the staging storage account. Configure the Snowflake side. In Snowflake, create a private connectivity endpoint that points at the external staging container and set USE_PRIVATELINK_ENDPOINT = TRUE. Snowflake provisions a service principal in your Azure tenant on your behalf. Grant that service principal the Storage Blob Data Contributor role on the staging account. This lets Snowflake read from and write to staging over the private endpoint using its own identity. Close the public paths. You should have already denied public inbound access to the Fabric workspace as the last part of the instructions provided in step 1. You also need to disable public network access to the staging account by setting the default public network access rule to disabled. Create Copy Activity or Copy Job. Create your pipeline with Copy Activity or, even better, create a Copy Job. Then select Snowflake as the source or destination, enable external staging, select the staging storage account, enter the required storage path, choose a destination, and run the item. Boundaries and things to check A few constraints are worth calling out before you plan a rollout: Workspace-level Private Link prerequisites. Confirm that your Fabric capacity SKU and region support Workspace-level Private Link, and that your tenant admin has enabled the required tenant settings. Staging is Azure Blob. External staging for Snowflake in the Copy Activity uses an Azure Blob container. You don't need to enable hierarchical namespace on the account for staging to function. Snowflake USE_PRIVATELINK_ENDPOINT requires a supported Snowflake account. Snowflake-managed outbound private connectivity endpoints require Business Critical edition or higher. Confirm that the feature is enabled for your Snowflake account before provisioning the endpoint. Private Endpoint Approval. The private endpoint from Snowflake into your storage account will be in a pending state and must be approved on the storage account. Next steps If you're evaluating Snowflake as a source or sink in Fabric today, deploy this topology end-to-end in a non-production workspace before you promote it. The moving parts are simple individually, but each one must be in place for the private path to hold. Learn more from the following documentation: Set up and use Workspace-level Private Links in Microsoft Fabric Configure Snowflake in a copy activity in Fabric Data Factory Use workspace identity in Fabric Data Factory Configure trusted workspace access in Microsoft Fabric Snowflake: Configuring an Azure container for loading data Snowflake: Managing Azure Private Link endpointsChoosing your medallion pattern in Fabric Data Warehouse
Coauthor: Artur Vieira Part one of a series on medallion architecture with Fabric Data Warehouse. Medallion architecture is one of the most common patterns for organizing data in Microsoft Fabric, but successful implementations require a series of design decisions — from choosing the right architecture pattern to securing, governing, and optimizing your workloads. In this five-part series, we'll walk through the practical choices that shape a modern medallion implementation in Fabric Data Warehouse (DW), sharing recommendations, tradeoffs, and real-world guidance along the way. In this installment, we'll focus on the first and most important decision: choosing the right medallion pattern for your workload. Why this matters Most teams designing a medallion architecture in Microsoft Fabric start with the wrong question: "Should I use a Lakehouse or a Warehouse?" The better question is, "How much Spark do I actually need?" Your answer will shape everything from development workflows to security and long-term maintenance. Quick level-set: medallion organizes data into three layers — Bronze (raw), Silver (enriched), and Gold (curated) — to progressively improve data quality and structure. That’s the whole definition you need. The interesting part is how you map those layers onto Fabric. The key decision: Data Warehouse, Lakehouse, or both? Fabric gives you two major analytics storage options on OneLake: Lakehouse and Data Warehouse. Because Fabric DW is an enterprise-scale SQL warehouse built on the open Delta Lake format in OneLake — combining a relational SQL engine with lake storage, so data is stored as Delta Parquet with ACID transactions and time travel — you don’t have to choose between “warehouse” and “lake.” You’re really choosing how much of the pipeline you run in T-SQL versus Spark. That leads to two patterns worth recommending: an all-in-one or hybrid approach. Pattern A: All-in-One Data Warehouse Use Fabric DW for Bronze, Silver, and Gold, separated by schemas (for example Bronze.*, Silver.*, Gold.*) or by separate warehouses, with data flowing raw → curated entirely within the warehouse using T-SQL or Data Factory pipelines. Best when: most of your data is structured (or can be structured on load) and your team prefers SQL-centric development. Why it’s nice: one engine, one skill set, transactions and views for every hop. Pattern B: Lakehouse + Warehouse Hybrid Land raw data in a Lakehouse for Bronze (and optionally Silver), where you can use Spark for complex prep, then implement Gold in the Warehouse as the SQL serving layer for BI. This works seamlessly because OneLake underpins both — Silver or Gold data can be materialized to Delta tables and queried by the Warehouse’s SQL endpoint without copying. Best when: You have unstructured or semi-structured data, or need heavy data engineering in Python or Scala. Why it’s nice: The right engine for each layer — Spark for heavy prep, T-SQL for serving — with no data copying, since OneLake underpins both Which should you pick? Here’s the rule I’d give a customer: Rule of thumb: For most SQL-focused analytics teams, an all-in-one Fabric Data Warehouse is the pattern I’d recommend starting with. If you have a mix of unstructured data or need heavy data engineering, land and refine raw data in a Lakehouse, then serve the final Gold layer from the Warehouse. Fabric’s architecture lets you evolve, so you’re not locked in: start all-in-warehouse and add a Lakehouse later when a new source needs Spark, or vice versa. All-in-One DW Lakehouse + DW Hybrid Primary skill T-SQL Spark / Python + T-SQL Best for data Structured / relational Unstructured, semi-structured, streaming Bronze lives in Warehouse staging tables Lakehouse (raw files, delta tables or both) Gold lives in Warehouse Warehouse Pick it for Simplicity, one engine Flexibility, heavy engineering One layout rule, whichever pattern you choose Keep layer separation clear. Microsoft recommends isolating layers into different workspaces, or at least different Fabric items, for better control and governance. The payoff is real: separate items give you finer security control — only data engineers touch Bronze while analysts see only Gold — and clearer isolation, so an accidental change in Gold can’t affect Bronze. Takeaway Don’t frame this as “Warehouse vs. Lakehouse.” Frame it as “How much Spark do I need?” If your workloads are structured and SQL-first, choose an all-in-one DW. If they’re unstructured or engineering-heavy, choose a hybrid approach, with Gold always served from the Warehouse. As your implementation grows, consider separating Bronze, Silver, and Gold into distinct Fabric items or workspaces to simplify governance and security. Ready to go deeper? Explore the Microsoft Fabric documentation for Data Warehouse and OneLake, then stay tuned for Part 2 of this series, where we'll walk through howBronze, Silver, and Gold layers are implemented in practice. This post is part of our Medallion Architecture on Fabric Data Warehouse series: Choosing your medallion pattern in Fabric Data Warehouse Building the Bronze → Silver → Gold layers Fabric DW best practices for medallion architectures Securing and governing your layers Performance tuning your medallion pipeline In the next post, we'll explore what Bronze, Silver, and Gold layers actually look like in Fabric Data Warehouse and how data moves between them.
Fabric Data Agents in Microsoft Copilot Studio (Generally Available)
Organizations can now bring governed business data from Microsoft Fabric into Copilot Studio agents, enabling agents to answer questions, generate insights, and support business processes using trusted enterprise data. Fabric data agents make it easier to ground experiences in the data your organization already relies on. Since we announced this integration in preview, both Fabric and Copilot Studio have evolved significantly. General availability provides a more integrated experience, broader publishing options, and a stronger foundation for deploying data-powered agents across the Microsoft ecosystem. Fabric Data Agents are now integrated as tools Previously, you added a Fabric data agent from the Agents category as a connected agent. Now, Fabric data agents integrate through the new tool-based experience in Copilot Studio. Simply select Add a tool, search for Fabric, and select Fabric IQ Data MCP. Your Copilot Studio agent can then invoke the Fabric data agent the same way it calls any other tool. The orchestrator decides when to reach for enterprise data and when to rely on its other knowledge sources and tools. The Fabric data agent continues to execute in Fabric, respect permissions on underlying data sources, and return answers grounded in governed enterprise data. The benefit is a more composable architecture. A Fabric data agent becomes one capability among the connectors, workflows, and other tools available to your Copilot Studio agent. You can combine multiple Fabric data agents with other business systems to create richer, more capable solutions. Built on the new Copilot Studio experience and runtime Copilot Studio introduced a redesigned authoring surface and runtime powered by the GitHub Copilot harness. You can build, test, evaluate, publish, and monitor agents from a unified environment while using natural language to describe agent behavior and capabilities. For Fabric customers, the biggest benefit is improved orchestration. The harness uses an improved reasoning model that is better at deciding which tool to call and how to combine what comes back, which matters when a Fabric data agent sits next to SharePoint knowledge, connectors, and other tools in the same agent. You choose the harness when you create the agent. Agents built with the standard harness remain fully supported. Instead of building isolated data experiences, organizations can create agents that reason across multiple sources of enterprise knowledge while using Fabric as their trusted data foundation. Publish to Microsoft Teams and Microsoft 365 Copilot Another improvement since preview is broader reach. You can now publish your Copilot Studio agent to Microsoft Teams and Microsoft 365 Copilot, bringing governed, data-grounded experiences directly into the tools people use every day. For example, a business user can ask about last quarter's regional performance in Microsoft 365 Copilot and get an answer from governed Fabric data without leaving the conversation they are already in. As always, permissions continue to apply. Users must have access to the Fabric data agent and its underlying data sources. d GIF - Add Fabric data agent as a tool to your Copilot Studio Agent. Why this matters The rise of AI agents is creating a new challenge for organizations: how to ensure agents are grounded in trusted business data rather than disconnected information sources. Fabric and Copilot Studio solve different parts of that challenge. Fabric provides the intelligence layer. It is where organizations bring together, govern, and publish data expertise through data agents. Copilot Studio provides the orchestration layer. It is where organizations build agents, workflows, and business processes that put that expertise to work inside a broader business process. Adding a Fabric data agent means everything you compose there can reason over governed enterprise data. Now, Fabric data agents and Copilot Studio (Generally Available) makes it easier than ever to bring governed data, business context, and intelligent action together in a single agent experience. Together, they enable organizations to: Build multi-agent solutions. Fabric data agent supplies the analysis while other agents and tools carry the rest of the process, such as drafting the document, updating the system of record, or routing an approval. Bring data together with everything else your agent knows. A Copilot Studio agent can pull a number from Fabric and a policy from SharePoint in the same answer. Meet users where they work. Publish to Teams and Microsoft 365 Copilot so business users get data-grounded answers in the flow of their work. Keep one definition of your data. Fabric data agent stays the single source of truth for every Copilot Studio agent that consumes it, so answers do not drift between teams. Build it once in Fabric and reuse it across departments. Getting started Getting started with Fabric data agent integration with Copilot Studio (Generally Available) is easy: Build and publish a Fabric data agent with a clear, detailed description. Create or open an agent in Copilot Studio. Under Tools, select Add a tool, search for Fabric, and add Fabric IQ Data MCP. Test your agent in Preview. Publish to Microsoft Teams or Microsoft 365 Copilot. To learn more, refer to the Add a Fabric data agent as a tool in Microsoft Copilot Studio documentation for the full setup, and join the community discussion to share feedback and report issues.New CI/CD resources for Microsoft Fabric: from concepts to end-to-end automation
Introducing a new set of resources that make it easier to build continuous integration and continuous deployment (CI/CD) into your Microsoft Fabric data projects. A Fabric solution is rarely a single artifact. It is a composition of workspace items, and with more than 70 item types available today (and the number continuing to grow), moving a solution from development to production takes a deliberate plan. These new resources connect the platform capabilities into one coherent workflow and form a path you can follow end to end: understand the platform, plan your approach, get hands-on with guided tutorials, and reach for advanced options when your scenario calls for them. Get started: Introduction to CI/CD in Microsoft Fabric The new Introduction to CI/CD in Microsoft Fabric is the main landing page for these resources. It explains the platform layer by layer, from the Fabric REST API foundation through Git integration, deployment pipelines, the Variable library, the Fabric CLI, infrastructure as code with Terraform, and the fabric-cicd library. It also includes an enterprise reference architecture that shows how source control, CI automation, capacities, and workspaces fit together. If you are new to Fabric CI/CD, or want to understand how the pieces relate, start here. Plan with confidence: Concepts and best practices guide The Fabric CI/CD concepts and best practices guide moves from foundational concepts to platform capabilities, automation tooling, and a practical checklist you can apply to your projects. It explains what constitutes a Fabric solution, how a CI/CD project moves that solution across isolated dev, test, and prod environments, and which design decisions make promotion safe and repeatable. Use the best practices checklist when planning your next project. Get hands-on: New step-by-step tutorials For teams that learn by doing, new guided tutorials cover the full journey, from an automated end-to-end pipeline to focused walkthroughs of each approach: Automate an end-to-end CI/CD workflow A full walkthrough that uses Terraform to provision the dev, test, and prod workspaces and the fabric-cicd library to deploy content between them, tying the pieces into one automated flow from source control to production. To learn more, refer to the Automate an end-to-end CI/CD workflow tutorial. Deploy locally with the fabric-cicd library or the Fabric CLI Set up a basic development environment and deploy a lakehouse and notebook from your local workstation to a Fabric workspace by using either the fabric-cicd Python SDK or the fab deploy command. This is the fastest way to see the deployment process in action. To learn more, refer to the Deploy locally with the fabric-cicd library or the Fabric CLI tutorial. Build a CI/CD pipeline with Azure DevOps and the fabric-cicd library Promote changed items from dev to test to prod with approval gates, automatic replacement of environment-specific GUIDs through parameter files, and secure credentials in Azure Key Vault and a service principal. To learn more, refer to the Build a CI/CD pipeline with Azure DevOps and the fabric-cicd library tutorial. Deploy from Git with the Bulk Import and Export API A build-environment pattern in which an Azure DevOps pipeline reads Fabric item definitions from a Git folder and deploys them to a target workspace that is not connected to Git. The Bulk Import API creates new items and updates existing ones in place, relying on Fabric's built-in dependency handling to deploy items in the correct order. This approach is a strong fit when you want to treat item definitions as code and promote them through a structured release flow. To learn more, refer to the Deploy from Git with the Bulk Import and Export API tutorial. Go deeper: Dependency binding across workspaces When you promote a solution, the references between items can break. The new Understand dependency binding in cross-workspace deployment article explains why: some items reference their dependencies with portable logical IDs that bind automatically to the matching item in the target workspace, while others use workspace-specific object IDs that break unless you parameterize them. It maps which item types bind automatically and which need manual handling. For now, this guidance covers deployment through Git integration and the Bulk Import API. Bring your own Git provider: open-source generic Git sample Some teams promote content through a Git provider that Fabric integration does not natively support or run their environments in different Microsoft Entra tenants. The open-source fabric-cicd-generic-git sample addresses exactly these cases. It uses the Bulk Export and Import APIs, with portable Bash scripts that work the same across providers such as GitLab, Bitbucket, GitHub Enterprise Server, and Azure DevOps Server, and includes first-class support for service principals and multi-tenant deployments. If your setup falls outside native Git integration, start with this sample. Next steps Start with the Introduction to CI/CD in Microsoft Fabric to build a mental model. Use the concepts and best practices guide to plan your approach, then choose the tutorial that best matches your environment and build your first CI/CD process. Share your feedback through the Fabric Community.Five Business Event scenarios and the pattern they share
Welcome to the sixth post in our Business Events, Fabric Events, and Azure Events series for Microsoft Fabric. This series takes you from foundational event-driven concepts to practical implementation patterns that help teams turn meaningful business moments into trusted signals, decisions, and actions across Fabric.
