From Fabric Data to Applications: Getting Started with Rayfin
Many organizations have invested in bringing data together in Microsoft Fabric. The next challenge is turning that data into applications people can use—not just reports they can view. Building those applications often means assembling a backend, authentication, APIs, storage, and hosting before a team can test its idea. Rayfin offers a different starting point. It is a backend-as-a-service built on Microsoft Fabric. Developers define an application’s data model using TypeScript decorators, while Rayfin provisions and manages backend capabilities including the database, authentication, data APIs, storage, and hosting. Where Rayfin fits Think of Rayfin as an application-building layer, not a replacement for Fabric analytics. Fabric provides the data platform and governance foundation; Rayfin helps developers build and deploy an application backend on that foundation. This can be useful when a team wants to move from an analytical insight to a workflow or app that people interact with. For example, a team might begin with governed data in Fabric and prototype an application that lets users review records, take action, and track outcomes. Rayfin reduces the amount of backend setup needed to get that prototype running, leaving the team more room to focus on the user experience and business process. A practical way to start Choose one narrow scenario. Identify the users, the decision or action they need to take, and the data the app requires. Define the data model. Start with the entities and relationships the application needs rather than attempting to expose an entire data estate. Build a small working app. Use the Rayfin starter project to explore its generated backend capabilities and deployment workflow. Validate the enterprise requirements. Review access, data handling, integration boundaries, and operational needs before expanding beyond a pilot. The most valuable first outcome is not a large application. It is a working, governed path from data to a useful action—one that developers and business stakeholders can evaluate together. Rayfin, Fabric IQ, and data agents: related, but distinct These capabilities address different parts of an intelligent application. Rayfin helps build the application and its backend. Fabric IQ provides business context for data through concepts such as entities and relationships. Fabric data agents support conversational questions over Fabric data. A solution may use more than one of them, but they should not be described as interchangeable. The takeaway Rayfin is worth exploring when the barrier to a Fabric-based application is the time and complexity of assembling its backend. Start with a focused use case, build a pilot, and assess whether the resulting application meets your organization’s security, governance, and operational requirements.From API to Action: A Disruption-Response Web App on Fabric's SQL Database
Part 3 of a Series: Putting the Data to Work When a supplier location goes dark, the first question is always the same: who else is in that zone, and how much do we depend on them? The data to answer it already exists. The problem is reaching it fast enough, without opening a report or writing a query. In Part 1 I built the foundation for a Supply Chain Disruption Response App inside Microsoft Fabric: a SQL database, supplier data from an external OData feed, layered analytical views, and a first Power BI report. In Part 2 I opened that foundation up to the team, with a T-SQL notebook for the analyst and Git integration with Azure DevOps for the engineer. This final part is for the third person on that list: the developer. The goal is a small web app where a disruption-response team types in an affected location ID and instantly sees every supplier in that location and how many products each one provides. It takes two pieces: An API for GraphQL over one of the views from Part 1, so the app never needs a connection string. An ASP.NET Core web app that calls that API with the user's own sign-in and renders the answer. Same database, same workspace, still no new infrastructure to host for the data layer. Finishing the API Part 2 Promised Part 2 ended with "the API is ready." Here's what that took, because it's the quickest step in the whole series. In the workspace I selected + New item > API for GraphQL and named it supplier_impact_gql. For the data source I chose single sign-on (SSO) authentication, picked supply_chain_analytics_database from the OneLake catalog, and loaded one object: the SupplyChain.vProductsbySuppliers view, which counts products per supplier and supplier location. That was the whole build. Fabric generated the schema, the query types, and a filter for every column. Before writing any app code, I tested the exact question the app would ask in the built-in editor: query { vProductsbySuppliers(filter: { SupplierLocationID: { eq: 7 } }) { items { CompanyName SupplierLocationID ProductCount } } } The JSON came back with every supplier at location 7 and its product count. The SSO choice matters more than it looks. There's no shared key or service password baked into the API. Every caller is checked against the database with their own identity, so the API can never show someone more than the database would. The last step on the Fabric side was Copy endpoint in the ribbon. That URL is the only thing the app needs to know about Fabric. Scaffolding the App The app is a plain ASP.NET Core project, built in Visual Studio Code with the .NET SDK (version 8 or later). After installing the SDK from dotnet.microsoft.com/download and checking it with dotnet --version, I created the project from the VS Code terminal: dotnet new webapp -n GraphQLWebApp cd GraphQLWebApp The package install failed on my first try, because nuget.org wasn't registered as a source on that machine. If dotnet nuget list source doesn't show it, one command fixes it: dotnet nuget add source https://api.nuget.org/v3/index.json -n nuget.org Then three packages, each with one job: dotnet add package Azure.Identity dotnet add package GraphQL.Client dotnet add package GraphQL.Client.Serializer.Newtonsoft Package Role Azure.Identity Gets a Microsoft Entra token for whoever is signed in GraphQL.Client Sends the query to the Fabric endpoint GraphQL.Client.Serializer.Newtonsoft Turns the JSON response into objects For sign-in during development, the app reuses the Azure CLI session. I installed the CLI (winget install -e --id Microsoft.AzureCLI on Windows; brew install azure-cli on macOS), restarted VS Code so the terminal could find the az command, and signed in with the same work account I use for Fabric: One File, Two Pages The whole app lives in Program.cs: a home page with a single input, and a results page that calls the API and renders a table. I replaced the generated file with the code below. The only line to change is the endpoint you copied from Fabric. using System.Net; using System.Net.Http.Headers; using System.Text; using Azure.Core; using Azure.Identity; using GraphQL; using GraphQL.Client.Http; using GraphQL.Client.Serializer.Newtonsoft; var builder = WebApplication.CreateBuilder(args); var app = builder.Build(); // The endpoint you copied from the API for GraphQL item const string graphQLEndpoint = "ReplaceWithYourGraphQLEndpointAddress"; // Sign-in: uses your Azure CLI login locally, a managed identity when hosted var credential = new DefaultAzureCredential(); var tokenContext = new TokenRequestContext( new[] { "https://analysis.windows.net/powerbi/api/.default" }); const string pageHead = """ <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <title>Supplier Impact Finder</title> <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/css/bootstrap.min.css"> </head> <body><div class="container mt-5"> <h3>Supplier Impact Finder</h3> """; const string pageFoot = "</div></body></html>"; // Home page: ask for the impacted location app.MapGet("/", async context => { context.Response.ContentType = "text/html"; await context.Response.WriteAsync(pageHead + """ <p>Enter the location ID facing an outage. You will see every supplier in that impact zone and how many products each one supplies.</p> <form method="get" action="/graphql" class="form-inline"> <label for="locationId" class="mr-2">Location ID:</label> <input type="number" min="1" class="form-control mr-2" id="locationId" name="locationId" required> <button type="submit" class="btn btn-primary">Search</button> </form> """ + pageFoot); }); // Results page: query the GraphQL API and render a table app.MapGet("/graphql", async context => { context.Response.ContentType = "text/html"; if (!int.TryParse(context.Request.Query["locationId"], out var locationId)) { await context.Response.WriteAsync(pageHead + "<p>Please enter a numeric location ID.</p><a href='/'>Back</a>" + pageFoot); return; } // Get a fresh token per request; tokens expire after about an hour var token = await credential.GetTokenAsync(tokenContext); using var client = new GraphQLHttpClient( new GraphQLHttpClientOptions { EndPoint = new Uri(graphQLEndpoint) }, new NewtonsoftJsonSerializer()); client.HttpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token.Token); // Pass the location as a variable rather than building the query string var request = new GraphQLRequest { Query = """ query ($locationId: Int!) { vProductsbySuppliers(filter: { SupplierLocationID: { eq: $locationId } }) { items { CompanyName SupplierLocationID ProductCount } } } """, Variables = new { locationId } }; var response = await client.SendQueryAsync<dynamic>(request); if (response.Errors?.Length > 0) { await context.Response.WriteAsync(pageHead + "<p>Query failed: " + WebUtility.HtmlEncode(response.Errors[0].Message) + "</p><a href='/'>Back</a>" + pageFoot); return; } var rows = new StringBuilder(); var count = 0; foreach (var item in response.Data.vProductsbySuppliers.items) { count++; rows.Append($"<tr><td>{WebUtility.HtmlEncode((string)item.CompanyName)}</td>" + $"<td>{item.SupplierLocationID}</td><td>{item.ProductCount}</td></tr>"); } var body = count == 0 ? $"<p>No suppliers found at location {locationId}.</p>" : $""" <p>{count} supplier(s) affected at location {locationId}.</p> <table class="table table-striped"> <thead><tr><th>Company</th><th>Location ID</th><th>Product count</th></tr></thead> <tbody>{rows}</tbody> </table> """; await context.Response.WriteAsync(pageHead + body + "<a href='/' class='btn btn-link'>New search</a>" + pageFoot); }); app.Run(); Three decisions in that file are worth calling out, because they're the difference between a demo and something a team could keep running: A fresh token on every request. Entra tokens expire after about an hour. Fetching one at startup works in a quick test and then fails quietly the next morning. GraphQL variables, not string building. The location ID travels as $locationId, so the API checks that it's an integer and nothing typed into the box can change the shape of the query. One credential for every environment. DefaultAzureCredential picks up the Azure CLI sign-in on my laptop and a managed identity once the app is hosted. The code doesn't change between the two. Within a second the page listed every supplier operating at location 7, with the number of products each one provides to manufacturing. That table is where a response starts: which suppliers to call first, which products are exposed, and where to look for alternatives. Because the API reads the view directly, the answer reflects the database as it stands right now, including any schema change that came through Git in Part 2. Why This Pattern Matters A custom API for this app would normally mean another project: a service layer to write, a host to run it on, connection secrets to store and rotate, and an authorization model to keep in step with the database. Here, the API was an item in the same workspace, the security came from the database itself, and the app is one file. For disruption response, that cuts the path from question to answer down to a text box. The person handling the crisis doesn't need Power BI open, a SQL login, or an analyst on call. They need a location ID. Looking Back Across the Series Three posts, one SQL database, one workspace: Part Built Fabric capability For 1 Database, external supplier data, layered views, Power BI report SQL database, Dataflow Gen2, SQL analytics endpoint, semantic model The business 2 Exploratory analysis and version-controlled schema T-SQL notebooks, Git integration with Azure DevOps The analyst and the engineer 3 Live API and a disruption-response web app API for GraphQL, Entra sign-in The developer and the response team The thread through all of it is a single definition of the data. The report, the notebook, the repository, and the app all read the same views. When a supplier table changes, every one of them sees the change, and nobody is reconciling copies in the middle of an incident. If you want to keep building, three natural next steps: Host the app in Azure App Service with a managed identity. DefaultAzureCredential picks it up with no code change. Add a GraphQL mutation so responders can log an incident against a location. Expose vSalesByDate through the same API to show the revenue at risk next to each affected supplier. If you missed the earlier parts: Part 1: From Chaos to Clarity: A Supply Chain Response App with Fabric's SQL Database | Microsoft Fabric Community Part 2: From Solo Build to Team Asset: Notebooks and Source Control on Fabric's SQL Database | Microsoft Fabric Community Thanks for following the series. If you take this pattern into your own data, I'd like to hear what you build. Happy Reading and share your thoughtsFrom Solo Build to Team Asset: Notebooks and Source Control on Fabric's SQL Database
A database nobody can explore, change safely, or call from an application is just a well-organized archive. In Part 1 I built the data foundation for a Supply Chain Disruption Response App inside Microsoft Fabric: a workspace, a SQL database, supplier data pulled in from an external OData feed, a layered set of analytical views, and a first Power BI report. That was the foundation. This post is about opening it up in three directions, each aimed at a different person on the team: The analyst, who wants to poke at the data interactively and leave notes behind, using a Fabric Notebook. The engineer, who wants every schema change tracked and reviewable, using Azure DevOps source control. The developer, who wants a clean API instead of a connection string, using Fabric's API for GraphQL. Same database, same workspace, no new infrastructure. Here's how each piece came together. A Notebook That Speaks T-SQL Mention notebooks and most people picture PySpark and data frames. Fabric notebooks default that way too, but the language is a setting, not a commitment. Switching the notebook from PySpark (Python) to T-SQL in the toolbar, and again on the first code cell, turns it into something a SQL-first team can use on day one. From there, I attached a data source through Add data items > From OneLake catalog and picked the SQL analytics endpoint for supply_chain_analytics_database. The whole object tree from Part 1 showed up in the side panel: schemas, tables, and the three SupplyChain views. I didn't type the first query. Clicking the ellipsis next to vProductsBySupplier and choosing SELECT TOP 100 generated a ready-to-run cell: SELECT TOP (100) [ProductID], [CompanyName], [TotalOrderQty] FROM [supply_chain_analytics_database].[SupplyChain].[vProductsBySupplier] The result grid is where the notebook earns its place over the query editor. Alongside the rows there are buttons to chart the output, save it as a table, or download it. A side pane profiles each column automatically: minimum and maximum values, missing data, unique counts. For a disruption-response analyst, that's a fast sanity check on whether a supplier's order volume is an outlier or business as usual, with no extra query. The other half of the value is the part that isn't code. Hovering between cells offers + Markdown, and a short note explaining what a query shows and why it matters turns a throwaway exploration into something a colleague can pick up next week. I saved mine as products_by_suppliers_notebook in the same workspace, next to the database and the report. A query in the editor answers a question once. A notebook keeps the question, the answer, and the reasoning together. Putting the Workspace Under Source Control Everything so far lived only in the Fabric portal, which is fine for one person and risky for a team. If someone changes a column on a table that feeds a disruption dashboard, there should be a record of who did it, when, and why. Fabric's answer is Git integration at the workspace level. The setup has two halves. In Azure DevOps, I created a new Git repository named SupplyChainAnalytics inside my project, with a README so the main branch exists from the start. In Fabric, under Workspace settings > Git integration, I chose Azure DevOps as the provider and filled in five values: Setting Value Organization My Azure DevOps organization Project Supply-Chain-Disruption-App Git repository SupplyChainAnalytics Branch main Git folder SupplyWorkload The folder didn't exist yet, so Fabric offered to create it. One click on Create and sync, and after a few minutes every item in the workspace showed a Synced status. What landed in the repository is the interesting part. The SQL database isn't stored as an opaque binary or a backup file. It arrives as a folder, supply_chain_analytics_database.SQLDatabase, organized by schema and object type, with one .sql file per object. Opening dbo/Tables/Suppliers.sql shows a plain CREATE TABLE statement for the supplier table that Dataflow Gen2 created in Part 1. That's the shift worth noticing: the database schema is now text. It can be diffed, reviewed in a pull request, and rolled back like any other code. Schema Changes in Both Directions A sync that only runs one way is a backup. What makes this real source control is that changes travel in both directions, and I tested each. In Azure DevOps, I edited Suppliers.sql directly and changed the Fax column definition to: [Fax] NVARCHAR (255) NULL, [Phone] NVARCHAR (255) NULL, I committed it with a descriptive message, Updated Suppliers.sql [Fax] column. Back in Fabric, the Source control button showed one pending update. Selecting Update all applied it to the live database, and a quick check confirmed the change had landed: SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'Suppliers'; No deployment script, no manual ALTER TABLE. The definition in the repository changed, and Fabric worked out what the database needed to match it. From the database to the repository Then the reverse. In the Fabric query editor I added a column the way a developer in a hurry would: ALTER TABLE Suppliers ADD Notes NVARCHAR(MAX); This time Source control showed an uncommitted change rather than an incoming update. I selected the object and committed it from Fabric, and the new Notes column appeared in Suppliers.sql in the repository. In production, the first direction is the one to standardize on: change the definition in a branch, review it, merge, then update the workspace. But the second direction matters too, because it means a quick fix made in the portal doesn't silently drift away from what the repository says. Fabric notices and asks you to commit it. One caveat: I edited the file in the DevOps web editor to keep the demo short. For real work, the same files open as a SQL project in Visual Studio Code, which is where schema changes belong. Why This Pattern Matters Part 1 showed that Fabric collapses the database, the pipeline, and the BI layer into one environment. Part 2 shows the same thing happening to the work around the database. Traditionally, each of these three steps means another tool and another handoff: a separate notebook environment with its own connection setup, a schema-compare tool or hand-written migration scripts, and a custom API layer someone has to build, host, and secure. Here, each one was an item added to the same workspace, pointed at the same database, using the same sign-in. For a disruption-response system, that matters in a specific way. The analyst's notebook, the engineer's repository, and the developer's API all read from one definition of the data. When a supplier table changes, the change is tracked in Git, visible in the notebook, and reflected in the API. Nobody is reconciling three copies of the truth in the middle of a crisis. What's Coming Next in This Series The database can now be explored, versioned, and called. One piece is left: Part 3: The Application. Wiring the GraphQL endpoint into a working ASP.NET web app, where a disruption-response team types in an affected location ID and instantly sees every supplier in that location and how many products each one provides. If you missed it, Part 1 covers the data foundation this post builds on: From Chaos to Clarity: A Supply Chain Response App with Fabric's SQL Database. Stay tuned. The API is ready; next up is the app that calls itMicrosoft Fabric Database Hub: A Unified Control Plane for the Modern Database Estate
Introduction Organizations today manage an increasingly complex database landscape. Mission-critical data lives across Azure SQL Database, SQL Server, Azure Database for PostgreSQL, Azure Cosmos DB, Fabric databases, and various on-premises environments. While each platform serves important business needs, managing them often requires separate monitoring tools, governance frameworks, operational processes, and administrative experiences. As enterprises accelerate their AI and data transformation initiatives, fragmented database management becomes a significant challenge. Database administrators, platform teams, and data leaders need a unified way to understand the health, security, performance, and operational posture of their entire database estate. Microsoft Fabric Database Hub addresses this challenge by introducing a centralized database management experience inside Microsoft Fabric. Rather than managing databases through multiple portals and disconnected tools, organizations gain a single operational experience to discover, observe, govern, and optimize databases across cloud, on-premises, and Fabric environments. What is Microsoft Fabric Database Hub? Database Hub is a unified operational experience within Microsoft Fabric that provides centralized visibility into an organization's database estate. The Database Hub enables customers to: Discover databases across multiple platforms Monitor performance and health Identify operational risks Improve governance and compliance Receive AI-assisted recommendations Investigate issues across environments Prioritize remediation activities Instead of jumping between Azure Portal, SQL Management tools, monitoring platforms, and Fabric workspaces, administrators gain a single pane of glass for database operations. The vision is simple: One place to see the estate, understand what matters, and take the next best action. Why Database Hub Matters The Challenge of Database Fragmentation Most enterprises operate hundreds or even thousands of databases across multiple technologies: Azure SQL Databases Azure SQL Managed Instances SQL Server Azure Database for PostgreSQL Azure Cosmos DB SQL Database in Fabric Hybrid and Arc-enabled deployments Each platform typically has: Separate monitoring tools Separate security dashboards Different governance processes Different performance views Different troubleshooting experiences As AI initiatives expand, organizations require broader visibility across operational and analytical systems. Database Hub brings these experiences together. A Strategic Shift Historically, organizations focused on managing individual databases. Database Hub introduces a new operational model: Manage the entire database fleet rather than managing databases one at a time. This estate-first approach enables platform teams to: Identify systemic issues Prioritize risks Establish governance standards Improve operational consistency Scale DBA operations efficiently Supported Database Platforms Database Hub provides a unified view across Microsoft database technologies including: Azure SQL Database Azure SQL Elastic Pools Azure SQL Managed Instance SQL Database in Fabric Azure Cosmos DB Azure Database for PostgreSQL SQL Server enabled by Azure Arc SQL Server running on Azure Virtual Machines This broad support allows customers to modernize at their own pace while maintaining visibility into legacy and modern environments. Key Capabilities 1. Estate-Level Visibility One of the most powerful capabilities of Database Hub is estate-wide visibility. Database administrators can: View all databases in a single inventory Search and filter resources Group databases by platform Understand deployment distribution Assess operational health Instead of reviewing systems individually, administrators gain immediate visibility across their entire estate. 2. Centralized Monitoring Database Hub introduces a consolidated monitoring experience. Teams can analyze: Database health Availability indicators Capacity trends Utilization metrics Resource consumption Performance patterns The platform helps identify emerging issues before they become business-impacting incidents. 3. Unified Performance Insights Performance data is often scattered across multiple monitoring tools. Database Hub provides: Estate-wide performance visibility Cross-database trend analysis Real-time monitoring Historical performance analysis Administrators can quickly understand: Which databases require attention Performance degradation trends Resource bottlenecks Optimization opportunities 4. Issues and Recommendations Database Hub automatically surfaces: Issues Issues represent conditions requiring immediate attention: Security concerns Configuration problems Performance degradation Availability risks Operational anomalies Suggestions Suggestions identify opportunities for improvement: Performance tuning Cost optimization Security enhancements Utilization improvements Best-practice recommendations This helps teams move from reactive operations to proactive optimization. 5. AI-Assisted Database Operations One of the most exciting innovations is the integration of AI-driven assistance. Database Hub leverages intelligent database agents and Copilot-powered experiences to help users: Understand operational changes Investigate risks Diagnose issues Recommend corrective actions Identify optimization opportunities Rather than simply displaying metrics, the platform helps explain: What changed Why it matters What action should be taken This dramatically reduces time-to-resolution and accelerates troubleshooting. 6. Governance and Compliance Visibility Governance remains a top concern for regulated organizations. Database Hub supports: Centralized governance visibility Security posture monitoring Policy reporting Compliance tracking Risk identification Importantly, organizations can maintain their existing governance and operational models while benefiting from centralized observability. This is particularly valuable for industries such as: Oil & Gas Energy Financial Services Healthcare Government Manufacturing 7. Hybrid and Multicloud Awareness Many organizations continue operating hybrid environments. Database Hub acknowledges this reality by providing visibility across: Cloud databases On-premises databases Arc-enabled environments Fabric-native databases Customers gain a consistent operational experience without requiring workload migration. How Database Hub Fits into the Fabric Vision Microsoft Fabric is evolving into a unified data platform that combines: Data Engineering Data Factory Data Science Real-Time Intelligence Power BI OneLake Fabric Databases Fabric IQ Database Hub extends this strategy by integrating operational databases into the broader Fabric ecosystem. The result is a platform that spans: Operational Data Transaction processing Business applications Line-of-business systems Analytical Data Warehouses Lakehouses Power BI models AI Workloads Intelligent applications Retrieval-Augmented Generation (RAG) Agentic AI solutions Copilot experiences Database Hub serves as the operational bridge connecting these worlds. Benefits for Customers For Database Administrators Single management experience Faster troubleshooting Estate-wide visibility Reduced tool sprawl For Platform Teams Centralized governance Consistent operational standards Improved fleet management For Executives Better operational risk visibility Improved compliance posture Greater operational efficiency AI-ready database strategy For Data and AI Teams Better connection between operational and analytical systems Easier integration with OneLake Faster access to enterprise data Improved AI readiness Real-World Customer Scenario Consider a global energy company running: SQL Server on-premises Azure SQL Managed Instance Azure Database for PostgreSQL Azure Cosmos DB Fabric Databases Traditionally, each environment requires separate management processes. With Database Hub, the organization can: Discover all databases from one location. Monitor estate-wide health and performance. Identify security and compliance gaps. Receive AI-generated recommendations. Investigate issues before they impact production. Prioritize remediation activities across the entire portfolio. This shifts operations from reactive administration to intelligent estate management. The Future of Database Operations The database landscape is becoming more distributed and complex. At the same time, organizations expect: Greater reliability Lower operational costs Stronger governance Faster innovation AI-driven insights Database Hub represents Microsoft's vision for the future of database operations: A unified, AI-powered control plane capable of managing the complete database estate regardless of where workloads run. As organizations continue their modernization journey, solutions like Database Hub will become increasingly important for maintaining visibility, governance, and operational excellence. Final Thoughts Microsoft Fabric Database Hub is more than another management portal. It is a strategic evolution in how organizations operate, govern, and optimize their database estates. By bringing Azure, Fabric, SQL Server, PostgreSQL, Cosmos DB, and hybrid environments into a single operational experience, Database Hub empowers teams to move from fragmented database administration to intelligent estate-wide operations. For organizations pursuing data modernization and AI transformation, Database Hub provides a critical foundation: visibility, governance, observability, and actionability across the entire database landscape. In the era of AI, the winners will not simply be the organizations with the most data. They will be the organizations that can understand, govern, and operationalize their data estate most effectively. Database Hub in Microsoft Fabric is designed to help them do exactly that.How to Mirror Google BigQuery Data into Microsoft Fabric | Step-by-Step Tutorial
In this video, I walk through the complete process of mirroring Google BigQuery tables into Microsoft Fabric using the Mirrored Google BigQuery (Preview) feature. This allows seamless real-time replication of your data from GCP into Microsoft Fabric’s OneLake—without writing ETL pipelines. What You’ll Learn How to set up BigQuery mirroring in Microsoft Fabric How to connect BigQuery using a Service Account How automatic table synchronization works How to query mirrored tables using SQL in Fabric How new tables in BigQuery are automatically replicated Why Use BigQuery Mirroring? No manual data movement or pipelines required Unified analytics across cloud platforms Near real-time data freshness via CDC Query directly in Power BI, Notebooks, and SQL Analytics Endpoint If you found this helpful, please like, share, and subscribe for more Fabric & Data Engineering tutorials Comment below #MicrosoftFabric #GoogleBigQuery #BigQueryMirroring #FabricMirroring #OneLake #CloudAnalytics #DataEngineering #ModernDataStack #RealTimeDataSync #PowerBI #SQLAnalyticsEndpoint #GCP #Azure #DataWarehouse #AnalyticsEngineering #NoETL #CDCReplication #DataIntegration #EnterpriseData #DataPlatform #FabricWorkspace #GoogleCloud #MicrosoftAzure #DataAnalytics watch?v=Pqllan2Rc4oOpen Mirroring in Microsoft Fabric | Step-by-Step Hands-On Demo
In this video, we explore Open Mirroring in Microsoft Fabric, a powerful new capability that enables near real-time data ingestion into OneLake without building complex data pipelines. This session is a detailed, hands-on walkthrough covering how to create a mirrored database, perform initial data loads, handle incremental updates, manage schema changes, and query data using the SQL Analytics Endpoint. Don’t forget to Like, Share, and Subscribe for more community-driven data and Fabric content. watch?v=CtoDdjpdjT4From Chaos to Clarity: A Supply Chain Response App with Fabric's SQL Database
Supply chains break in ways that rarely announce themselves in advance. A supplier goes dark, a shipment stalls at a border, a single component runs low in a warehouse halfway across the world — and suddenly a business is scrambling to figure out what's actually happening and what to do about it. The businesses that recover fastest aren't the ones with the fewest disruptions; they're the ones who can see the problem the moment it appears. That's the premise behind a hands-on project I've been working through: building a Supply Chain Disruption Response App entirely inside Microsoft Fabric, using its native SQL database capability. No separate database server to provision, no disconnected BI layer bolted on afterward — just one unified platform taking raw supplier and warehouse data and turning it into something a team can actually act on. It's a big enough build that I'm splitting it into a short series. This first post covers the full data foundation — standing up the workspace and database, loading and shaping the data, layering analytical views on the SQL analytics endpoint, and turning it all into a live Power BI report. Later posts will pick up from here and cover notebooks, DevOps, a GraphQL API, and finally a real web application. Here's how the foundation came together, and what stood out along the way. Starting with a Workspace, Not a Server The first thing that struck me about Fabric is how little "infrastructure thinking" it requires. There's no spinning up a VM, no configuring a database engine from scratch. Everything begins with a workspace — essentially a shared home for every item in the project: databases, pipelines, notebooks, and reports all live side by side. Setting one up is a matter of naming it, attaching it to a Fabric capacity, and deciding on a semantic model storage format. Ten minutes in, I had a dedicated space — Supply Chain Analytics Workspace — ready to hold the entire project without a single line of provisioning script. Spinning Up the Database From there, creating the actual SQL database is almost anticlimactic in its simplicity: a name, a click, and Fabric provisions a fully managed SQL database behind the scenes. I called mine supply_chain_analytics_database, and within moments it was ready to accept data. Fabric even ships with sample retail data (the familiar SalesLT schema — customers, products, orders) that loads in with one click. It's a small touch, but it meant I could start writing real queries immediately instead of wrestling with seed data. Teaching the Database to Think About Supply Chains Sample sales data is useful, but it doesn't know anything about supply chains. So the next step was to give the database a vocabulary for the problem at hand: a dedicated SupplyChain schema, and inside it, a Warehouse table modeling the essentials — which product ties to which component, which supplier provides it, where that supplier is located, and how much stock is currently on hand. Rather than hand-typing rows, I populated the table directly from the existing product catalog, using T-SQL to generate plausible component, supplier, and location IDs on the fly. In a few lines of script, the database went from "generic retail demo" to "a working model of a multi-tier supply network" — the exact structure you'd need to trace a disruption back to its source. One small but genuinely useful discovery here: Fabric's query editor has a Copilot-style assist built in. Typing a plain-English comment like "show the total number of customers" and pressing Tab generates the T-SQL for you — and there's an "Explain query" option that annotates existing code step by step. For anyone who's ever inherited a gnarly SQL script with zero comments, this alone is worth the price of admission. Bringing in the Outside World Real supply chains don't live in one database — they're distributed across vendors, partners, and legacy systems that were never designed to talk to each other. To simulate that, I used a Fabric Data Pipeline with a Dataflow Gen2 to pull supplier data from an external OData feed (in this case, the Northwind Traders sample service), representing a partner organization sharing the same supplier network. https://services.odata.org/v4/northwind/northwind.svc/ What made this step interesting wasn't the mechanics — point, click, choose a table, run — but what it represents: Fabric's pipelines are built to treat "connect to an external partner's data" as a first-class, repeatable operation rather than a one-off script someone writes and forgets. Once the dataflow published and the database refreshed, a new Suppliers table appeared right alongside the internally generated data, ready to be joined. Turning Rows into Answers With warehouse and supplier data now living in the same database, the real payoff arrives: queries that answer questions a disruption-response team would actually ask. A simple ranking query — which products are moving the most — is useful on its own. But the more important artifact is a view: a saved query, vProductsbySuppliers, that joins warehouse inventory against supplier records and groups the results by supplier location. That view becomes a reusable lens on the data — the kind of object that can sit quietly behind a dashboard, or be exposed through Fabric's SQL analytics endpoint and GraphQL API for a front-end app to query directly. It's the difference between running a report once and building infrastructure a team can lean on every day. The Parts You Don't Have to Build Perhaps the most understated part of the whole exercise was everything I didn't have to configure. Fabric's SQL database comes with a built-in Performance Dashboard tracking CPU consumption, connection counts, and allocated storage, plus automatic indexing that tunes itself based on query patterns over time. Backups happen on a schedule without anyone setting up a maintenance window. For a demo project this is a nice-to-have; for a production disruption-response system running around the clock, it's the difference between a database that needs a dedicated administrator and one that mostly looks after itself. Querying Through the SQL Analytics Endpoint Every SQL database in Fabric quietly maintains a second identity: the SQL analytics endpoint. It's a read-only, automatically synchronized mirror of the same data, purpose-built for analytical querying rather than transactional writes. Practically speaking, that means I could point ordinary T-SQL at it — no different syntax, no separate connection ceremony — and start layering real analytical logic on top of the operational tables without ever putting extra query load on the transactional database itself. That separation turned out to be the right place to build out the analytical model properly. Rather than one flat query, I created three views, each with a distinct job: vProductsBySupplier rolls up order quantities by product and supplier; vSalesByDate breaks sales volume down by year and month; and vTotalProductsByVendorLocation sits on top of the first view and groups everything by supplier location — the exact cut a disruption-response team needs when a single region goes offline. Building views on top of other views felt like the right instinct here rather than a shortcut. vTotalProductsByVendorLocation didn't need to know anything about sales order headers or line-item details — it just needed the already-summarized product-and-supplier numbers, joined once more against the warehouse table to bring location into the picture. Each layer stays simple, and each one is independently reusable. CREATE VIEW SupplyChain.vProductsBySupplier AS -- View for total products each supplier SELECT sod.ProductID , sup.CompanyName , SUM(sod.OrderQty) AS TotalOrderQty FROM SalesLT.SalesOrderHeader AS soh INNER JOIN SalesLT.SalesOrderDetail AS sod ON soh.SalesOrderID = sod.SalesOrderID INNER JOIN SupplyChain.Warehouse AS sc ON sod.ProductID = sc.ProductID INNER JOIN dbo.Suppliers AS sup ON sc.SupplierID = sup.SupplierID GROUP BY sup.CompanyName, sod.ProductID; GO CREATE VIEW SupplyChain.vSalesByDate AS -- Product Sales by date and month SELECT YEAR(OrderDate) AS SalesYear , MONTH(OrderDate) AS SalesMonth , ProductID , SUM(OrderQty) AS TotalQuantity FROM SalesLT.SalesOrderDetail AS SOD INNER JOIN SalesLT.SalesOrderHeader AS SOH ON SOD.SalesOrderID = SOH.SalesOrderID GROUP BY YEAR(OrderDate), MONTH(OrderDate), ProductID; GO CREATE VIEW SupplyChain.vTotalProductsByVendorLocation AS -- View for total products by each supplier by location SELECT wh.SupplierLocationID AS 'Location' , vpbs.CompanyName AS 'Supplier' , SUM(vpbs.TotalOrderQty) AS 'TotalQuantityPurchased' FROM SupplyChain.vProductsBySupplier AS vpbs INNER JOIN SupplyChain.Warehouse AS wh ON vpbs.ProductID = wh.ProductID GROUP BY wh.SupplierLocationID, vpbs.CompanyName; GO From Views to Visuals: Building the Power BI Report Views are only half the story — someone still has to look at them. Fabric makes the handoff from database to BI tool almost seamless, but there's a small manual step worth calling out: grabbing the connection string. Under the database's Settings, the connection string contains both the server name and the database name buried inside a longer string, and it's these two values — not the full string — that external tools like Power BI Desktop or SQL Server Management Studio actually want. Inside Fabric itself, though, the path to a report is more direct. Selecting New semantic model over the SQL analytics endpoint pulls in the tables and views I'd built — including all three SupplyChain views — and wraps them in a data model describing how everything relates. One default was worth overriding immediately: the Location field on vTotalProductsByVendorLocation is a location identifier, not a quantity, so it needed Summarize by set to None. Otherwise Power BI happily sums location IDs together into a meaningless total, which is a small trap worth knowing about before it shows up in a report. Why This Pattern Matters Fabric collapses a workflow that traditionally spans several tools — a database server, an ETL tool, a BI platform, a monitoring dashboard — into one connected environment. For a supply chain disruption app specifically, that matters because disruptions don't wait for data to be reconciled across systems. The faster raw supplier and inventory data can be joined, queried, and visualized, the faster a team can answer the question that actually matters in a crisis: what's affected, and what do we do next? Building this out end-to-end — workspace, database, ingested and generated data, a layered set of analytical views, and a working Power BI report — took a single sitting. That speed is really the headline feature. The tools get out of the way fast enough that most of the effort goes into modeling the problem, not the plumbing. What's Coming Next in This Series With the data foundation and the first report in place, the next posts in this series will build on top of it: Part 2 — Notebooks, DevOps, and GraphQL: exploring the data interactively in a Fabric Notebook, bringing the database schema under source control with Azure DevOps, and standing up a GraphQL API on top of the vProductsbySuppliers view. Part 3 — The Application: wiring all of it together into a working ASP.NET web app that lets a disruption-response team type in an affected location and instantly see every supplier and product count at risk. Stay tuned — the foundation and the first report are in place; next up is making the data interactive.Microsoft Fabric Copy Job Tutorial for Beginners
In this beginner-friendly tutorial, I explain how to create and configure a Copy Job in Microsoft Fabric step by step — from source connection to destination loading. What is Copy Job in Microsoft Fabric Why Copy Job is important for Data Ingestion How to configure Source & Destination Step-by-step demo Best practices for beginners By the end of this video, you will confidently create your own Copy Job in Fabric. Thank you!! watch?v=mmZSvks32bg?si=s7ulM6OTk2eVg0h1
Turning the Tables - Tabular Data Stores in Fabric
Fabric provides a number of ways to store tabular data: tables on top of structured files in a Lakehouse, tables in a Warehouse, and now tables in a Fabric SQL Database. All support some level of access using the SQL query language. Some support cross-querying from other the other types of tabular data stores. All create their own default semantic model. We’ll compare and contrast these storage types, do some performance comparisons, and try to define how best to use each type. Join us for an opportunity to learn and participate in the dialogue with your peers that follows. This is a great opportunity to present your questions and challenges to a community of people working with similar technology.
