From Fabric Data to Applications: Getting Started with Rayfin
Many organizations have invested in bringing data together in Microsoft Fabric. The next challenge is turning that data into applications people can use—not just reports they can view. Building those applications often means assembling a backend, authentication, APIs, storage, and hosting before a team can test its idea. Rayfin offers a different starting point. It is a backend-as-a-service built on Microsoft Fabric. Developers define an application’s data model using TypeScript decorators, while Rayfin provisions and manages backend capabilities including the database, authentication, data APIs, storage, and hosting. Where Rayfin fits Think of Rayfin as an application-building layer, not a replacement for Fabric analytics. Fabric provides the data platform and governance foundation; Rayfin helps developers build and deploy an application backend on that foundation. This can be useful when a team wants to move from an analytical insight to a workflow or app that people interact with. For example, a team might begin with governed data in Fabric and prototype an application that lets users review records, take action, and track outcomes. Rayfin reduces the amount of backend setup needed to get that prototype running, leaving the team more room to focus on the user experience and business process. A practical way to start Choose one narrow scenario. Identify the users, the decision or action they need to take, and the data the app requires. Define the data model. Start with the entities and relationships the application needs rather than attempting to expose an entire data estate. Build a small working app. Use the Rayfin starter project to explore its generated backend capabilities and deployment workflow. Validate the enterprise requirements. Review access, data handling, integration boundaries, and operational needs before expanding beyond a pilot. The most valuable first outcome is not a large application. It is a working, governed path from data to a useful action—one that developers and business stakeholders can evaluate together. Rayfin, Fabric IQ, and data agents: related, but distinct These capabilities address different parts of an intelligent application. Rayfin helps build the application and its backend. Fabric IQ provides business context for data through concepts such as entities and relationships. Fabric data agents support conversational questions over Fabric data. A solution may use more than one of them, but they should not be described as interchangeable. The takeaway Rayfin is worth exploring when the barrier to a Fabric-based application is the time and complexity of assembling its backend. Start with a focused use case, build a pilot, and assess whether the resulting application meets your organization’s security, governance, and operational requirements.[SimplePBI] Get Tables, Measures, Columns, or Roles info using PBI Rest API and Python
Today, Power BI has become widely adopted. A huge number of companies are already using it, and the number of deployed models and reports is enormous. As a result, the challenges we face today are no longer the same ones we faced in the past. Previously, the community was more focused on learning Power BI tips and figuring out how to build specific user experience features. This was because many companies were creating their first reports. Today, however, we often need to understand reports that have already been implemented by other people or companies. How can we understand what is behind a published semantic model? The latest update/release of the Python library SimplePBI, which can be used to interact with the Power BI REST API, can help us retrieve information about the tables, measures, columns, and even roles that exist in a model. When we need to edit, support, or understand a semantic model in production, it can be a major challenge if proper versioning practices are not in place. Likewise, if we need to quickly understand fine-grained lineage at the table level and we do not have a proper governance catalog, we have a problem. For these situations, and many others, SimplePBI now makes it easy to query information about a semantic model using Python code. What can we retrieve? Tables: information about the tables in the model. Measures: all measures in the model, their expressions, and the tables they belong to. Columns: information about the model's columns and the tables they belong to. Roles: information about RLS roles created in the model. Relationships: how relationships are built. Partitions: table partitions with the power query definition. Let's see how easy it is to query the model. Prerequisites The first thing we need is to understand how to use the Power BI REST API. You can take your first steps with it by following this article. The second requirement is to make sure we have version 1.0.4 or later of the library. If you already have the library installed, you can run the following command in the console: pip install SimplePBI --upgrade Getting Information from the Model From this point on, we can query a semantic model or dataset from a specific workspace. First, we authenticate and generate a token: Then, we create an object for the category we want to work with. In this case, datasets refers to semantic models: sem = semanticmodels.SemanticModels(t.token) Although this functionality was developed specifically for the library, the available methods have user-friendly names that closely resemble those already used in the API documentation: list_roles_from_semantic_model list_tables_from_semantic_model get_tables_schema_from_semantic_model The last method provides information about columns and measures. We can obtain the workspace ID and semantic model ID from the browser URL when we open the model. It looks like this: https://app.powerbi.com/groups/[Workspace ID]/datasets/[Dataset ID]/details?experience=power-bi This is how we would execute a request to retrieve the measures and columns of a table for workspace ws and semantic model ds: As we can see, it only takes a single line to retrieve all the information. Each item in the rows list within the response dictionary represents either a measure or a column. The type field specifies which one it is. In this example, the first five rows show one column from the Cambiador_unidades table, as well as three measures and one column from the Customer table. Among the most useful pieces of information, we can see the object's name, the data type for columns, and the expression for measures created using DAX. NOTE: These requests are part of long-running operations, so the response may not be instantaneous and could take a few seconds. I hope this helps you investigate deployed models more effectively, whether you need to learn about them while providing support, improve your catalog structures, or quickly review measure expressions. Ultimately, the only limit to how you use it is your creativity.
Code review isn't enough for data pipelines
Last year around April, there was a production issue in our data engineering system and we had to roll back a recently merged pull request. This pull request had minor code changes and was reviewed and approved by me and one more person on my team. The change was made in the code of one of the upstream tables of a pipeline, which resulted in a schema violation in a downstream table. The downstream table was part of a separate pipeline, hence it was missed by the data engineer who raised the pull request. The change was quite small - and therefore tempting to approve quickly and be done with. Ideally, I should have looked at the pipeline runs in the development environment and verified all the downstream tables’ schema contracts and data. But practically speaking, this is a cumbersome and time-consuming task. Therefore, on busy days this usually gets skipped. Post that incident, whenever I reviewed PRs, I tried to look at the changes at the data level as well - at least whenever time permitted. I took more time to review (which was of course a bit frustrating both for the PR author and the reviewer). This habit turned into a project over time. I have recently open-sourced it to help others in the data engineering community. This article covers that project. Software teams have a safety net. Data teams mostly don’t. In software engineering, a pull request triggers a build that runs a full suite of tests to ensure every component is working correctly and nothing is breaking due to the current set of changes. The build and the test results give an objective signal to the reviewer about the changes. It also helps newcomers to the team, who may not yet know all the components fully, to author their changes or review others' changes properly. On the other hand, PRs raised in data engineering teams often have gaps in this process. A data engineer makes the code changes, creates/updates/deletes tables in the dev environment, tests the changes by running the pipelines and running other data quality tests. The first gap is here itself - governance. Unlike software engineering PRs, where there are coded tests for almost every related component that are run during the build, there are hardly any such tests set up to run automatically before merging any changes to production. There are usually no written schema contracts that the tests are run against. It depends on the practices followed in the team and the goodwill of the developer as to how thoroughly these tests are done. The second major gap is lack of automation. The pull request itself does not trigger any build as such - one that tests all the code, pipeline runs, impact on downstream tables, etc. As I mentioned at the beginning of the article, the reviewer has to check the runs manually in the dev branch and, at times, run the tests to ensure nothing is breaking downstream. The third gap is lack of visibility for new team members who do not have sufficient experience or knowledge about the team's data estate. They may miss out on testing some of the tables while making the changes or reviewing others' changes. Most data teams have data quality checks in place to identify any issues in production and minimise the risk of corrupting data in any of the downstream tables. These are fine, but they catch the issues in production, which is the fourth and final gap - lack of proactiveness. Although we catch the issues in production through these checks, in most cases the damage is partly already done - be it delayed runtimes, failure to meet SLAs or simply wasted compute in rerunning some of the pipelines. DEBIT - Data Engineering Build Impact Testing In order to close the above gaps, I built DEBIT. On every pull request raised in our team, this framework runs the tests needed automatically at both the code level and the data level. It identifies the tables impacted, checks their contracts mentioned in YAML files, figures out what the downstream and upstream tables for these impacted tables are, and then runs the tests on the actual data in dev environments. Finally, it stores those results and displays them in the PR build log for the reviewer to inspect. It targets Microsoft Fabric and lakehouse workflows, but the core is plain Python with no cloud dependency. The repo includes templates for GitHub Actions and Azure DevOps. DEBIT is open source under the MIT license: github.com/pratap-is-here/debit An Important Note The framework does not replace any testing that the PR author must do. It supports it by bringing governance to the whole process of moving any changes from dev to prod, by ensuring increased thoroughness and efficiency as well as reducing the review workload on the reviewers. So the PR author is still strongly recommended to do the testing on appropriate data and, if needed, copy some of the data from prod to dev before testing. DEBIT assumes the developer has already tested in dev against the latest prod data. If they haven’t, the validations fail, and it becomes visible in the PR. That's deliberate. DEBIT checks that testing happened. It doesn't replace it. 2 Phases Every PR runs through 2 phases. Phase 1 is an offline hard gate. It validates the table declarations in the PR template, reads the git diff to find the affected tables, checks the registry for completeness, cross-checks the lineage for consistency, expands the scope along the lineage graph, and verifies that every table in scope has a complete contract. It needs no cloud access. Phase 2 is optional and live. It builds a contract payload from the table registry, triggers a Fabric notebook through the Jobs API, and runs row count, freshness, schema and custom checks. It then reads the results back from OneLake and prints a structured pass/fail report. By default it warns instead of blocking, and you can switch it to a hard gate. I made Phase 1 the hard gate because it’s deterministic and runs offline. The live checks depend on a Fabric run, so I wanted them to start as warnings until a team trusts them. Lineage decides the Scope Testing everything on every PR is slow, and testing only the changed table misses the damage downstream. So the PR author chooses how far validation reaches. Each table lists its direct downstream consumers in a YAML file, and upstream is worked out from that at runtime. The author picks one of four modes in the PR description. In the diagram, Silver F is the table the PR touched. Affected only validates just F. Upstream adds everything F reads from, six tables in all. Downstream adds what reads from F, three tables. Full lineage covers both directions, eight tables. Silver G and Gold J sit beside F in the graph, not above or below it, so they stay out of scope. Metadata over Code All the rules live in YAML, not in hardcoded logic.\ Five core files are part of the metadata: subset_scope.yml: This file tells which tables, pipelines or notebooks are governed. table_registry.yml: This file covers the per table contracts. table_lineage.yml: This file mentions the downstream lineage. notebook_outputs.yml and pipeline_outputs.yml: These files tell which tables each notebook and pipeline produces. Two more handle execution freshness and custom check notebooks. Here’s what a contract looks like, trimmed from the synthetic example in the repo: silver.fact_orders: min_rows: 1000 freshness_hours: 24 freshness_column: order_date schema: required_columns: - order_id - customer_id - order_amount - order_date not_null_columns: - order_id - customer_id checks: - name: order_amount_non_negative sql: "SELECT COUNT(1) FROM {table} WHERE order_amount IS NOT NULL AND order_amount < 0" expect_eq: 0 There's no Python to write. Changing a threshold is a visible edit in a PR, with a name and a reason attached. A newer engineer can read the file to see what healthy means for a table. Rolling it out You don’t have to cover everything at once. You can start with a set of pilot tables and adjust the checks and framework based on your team’s requirements. Once you are confident with the updates, you can expand to more tables by simply updating the metadata files. Assets that are outside the scope will remain untouched. Did it help? On our team, it cut PR review time by roughly 40% and production incidents by roughly 25 to 30%. Treat those as our experience, not a benchmark. Your tables, your review habits and your starting point will differ. What it doesn’t do It assumes developers test in dev first. If the dev run is sloppy, DEBIT can only tell you that, not fix it. A contract is only as good as what you write into it. Too tight and the checks raise false alarms, too loose and they miss real problems, so the numbers need upkeep. The live checks only support Fabric today. YAML can’t express everything, so there’s a custom SQL check as an escape hatch. I’d rather keep those few, or the files turn into code. I built it for my own team, so expect rough edges. Try it You don't need cloud access to try it. The bundled example and the whole test suite run offline. git clone https://github.com/pratap-is-here/debit cd debit pip install -r requirements.txt python examples/run_validation_example.py python -m pytest -q The example walks one changed table through all four scope modes and prints a pass for each. The tests should report 114 passed. DEBIT is open source under the MIT license: github.com/pratap-is-here/debit If you review data PRs, I’d like to hear what’s missing. And if a field in the metadata files is awkward for your tables, open an issue with a small example of what you’d rather write. A real table that doesn’t fit is more useful to me than a star. Views are my own.Cross-Tenant Delegated OneLake Shortcut in Microsoft Fabric | End-to-End Tutorial
In this video, I walk through how to architect and implement a Cross-Tenant Delegated OneLake Shortcut in Microsoft Fabric. Discover how organisations can securely access and consume data stored in another Microsoft Fabric tenant without physically copying, moving, or replicating the data. Through a practical, step-by-step demonstration, you'll learn how to: ✅ Configure connectivity between two Microsoft Fabric tenants ✅ Implement delegated authentication for secure access ✅ Create a Cross-Tenant OneLake Shortcut ✅ Share data across organisational boundaries while maintaining governance and security ✅ Create Semantic model and Power BI report in the consumer tenant 🔍 What You'll Learn ✅ Understanding Cross-Tenant Delegated OneLake Shortcuts ✅ Microsoft Fabric multi-tenant data architecture ✅ Producer and Consumer tenant architecture design ✅ Configuring cross-tenant access in Microsoft Fabric ✅ Setting up delegated authentication using an Organisational Account or Service Principal ✅ Configuring workspace permissions and OneLake security ✅ Creating a Cross-Tenant OneLake Shortcut ✅ Accessing data across tenants without data duplication ✅ Security considerations and best practices for cross-tenant data sharing 🎯 Who Should Watch? This tutorial is ideal for: 🔹 Microsoft Fabric Data Engineers 🔹 Data Architects 🔹 Cloud Engineers 🔹 Analytics and Platform Teams 🔹 Anyone building secure multi-tenant data-sharing solutions If you're learning Microsoft Fabric and want to understand how OneLake Shortcuts enable governed data sharing across organisational boundaries, this is a tutorial you don't want to miss. 💡 One of the biggest advantages of OneLake Shortcuts is the ability to eliminate unnecessary data duplication while maintaining security, governance, and a single source of truth across tenants. 🎥 Watch the video https://www.youtube.com/watch?v=ihlWLRiia7Q&t=693s and let me know your thoughts in the comments. 👍 If you find the content valuable, please Like, Share, and Subscribe for more practical tutorials on: ✅ Microsoft Fabric ✅ Azure Data Engineering ✅ Data Architecture ✅ Modern Data Platforms #MicrosoftFabric #OneLake #OneLakeShortcut #CrossTenant #DelegatedAuthentication #MicrosoftFabricTutorial #DataEngineering #DataArchitecture #MicrosoftAzure #AzureDataEngineering #Lakehouse #DataSharing #ServicePrincipal #DataSecurity #CloudComputing #FabricLakehouse #DataPlatform #BigData #MicrosoftMVP #DatabricksMVP #AbiolaDavidFabric Data Agents Solve the Last-Mile Problem Between People and Governed Data
Fabric Data Agents can make governed data easier to question, understand, and use. This article explores what they solve, what they do not, and why trusted data, strong semantic models, and real-world testing are essential.Is Your Power BI Report Slow? 5 Data Modeling Mistakes You Should Check First
Power BI performance issues aren’t always caused by large datasets; poor data modeling can also slow down reports, slicers, and DAX measures. In this article, we’ll explore five common data-modeling mistakes that impact performance and practical ways to optimize your Power BI reports. 1. Building One Giant Flat Table The Mistake One of the most common approaches is putting everything into a single table: Customer information Product information Transaction details Location Dates Categories Measures Status fields At first, this seems simpler because there are fewer relationships to manage. But a single, wide table can introduce data duplication and make the model harder to optimize. Why It Can Hurt Performance Power BI's VertiPaq engine uses columnar storage and compression. The efficiency of that compression depends heavily on the characteristics of your columns. A large flat table can contain: Repeated descriptive values High-cardinality columns Unnecessary columns Duplicated customer or product attributes For example, imagine a fact table containing 10 million transactions where Customer Name, Customer City, Customer State, and Product Category are repeated across every transaction. You're storing the same descriptive information over and over again. The Fix: Build a Star Schema Instead of one giant table, separate your model into fact tables and dimension tables. Fact tables Sales Transactions Encounters Claims Inventory Dimension tables Date Customer Product Provider Location The fact table should contain the events or transactions you want to analyze, while dimension tables provide the descriptive attributes used for filtering and grouping. For example: DimDate | | DimCustomer ---- FactSales ---- DimProduct | | DimLocation This structure reduces redundancy, improves model organization, and aligns well with how Power BI's storage engine is designed to work. Pro Tip Don't just remove columns randomly. Instead, ask: Does this column need to exist in this table? If a descriptive attribute belongs in a dimension, move it there. 2. Using Bidirectional Relationships Everywhere The Mistake You create a relationship and immediately set the cross-filter direction to Both because it makes filters appear to "work everywhere." It seems convenient. But as your model grows, bidirectional relationships can introduce unnecessary complexity. Why It Can Hurt Performance With single-direction filtering, filters typically flow from the dimension to the fact table: DimCustomer → FactSales With bidirectional filtering: DimCustomer ↔ FactSales Filter context can now travel in both directions. When multiple tables and relationships are involved, this can create: Ambiguous filter paths More complex query behavior Unexpected filtering Difficult-to-debug totals Additional performance overhead in complex models This becomes especially important when your model contains multiple fact tables or many-to-many relationships. The Fix Use single-direction relationships by default. For a typical star schema: Dimension → Fact Use bidirectional filtering only when you have a specific reason for it and have tested its impact. If you need bidirectional behavior for a particular calculation, you can sometimes handle it within the measure using CROSSFILTER() rather than changing the relationship behavior across the model. Pro Tip If your report suddenly starts showing unexpected totals or filters are affecting tables you didn't expect, inspect your relationships. Relationship direction is often one of the first places worth checking. 3. Putting Multiple Business Processes Into One Fact Table The Mistake You have Sales, Returns, Inventory, Support Tickets, and other business processes, and decide to put everything into one massive fact table. At first, this may seem convenient. But there's a fundamental modeling problem: Different business processes usually have different grains. What Is Grain? The grain defines what one row represents. For example: One row = one sales transaction One row = one product shipment One row = one customer support ticket One row = one patient encounter These represent different business processes with different grains. Trying to combine them into one fact table can result in: Many blank columns Multiple flags indicating row types Complicated DAX Duplicate information Difficult-to-understand logic Larger and harder-to-maintain tables You may eventually find yourself writing logic like: IF( TransactionType = "Sales", ..., IF( TransactionType = "Return", ..., ... ) ) That's often a sign that one table is trying to do too many jobs. The Fix: Use One Fact Table Per Business Process Instead, create separate fact tables: DimDate / \ / \ FactSales FactReturns \ / \ / DimProduct Both fact tables can use shared dimensions such as: Date Customer Product Location These are often called conformed dimensions. This allows you to analyze different business processes consistently without forcing them into a single fact table. Pro Tip Before creating a fact table, write one sentence describing its grain: "One row represents ______." If you can't clearly complete that sentence, your table design probably needs another look. 4. Composite Models: When to Mix Import and DirectQuery The Mistake: Assuming you have to choose one storage mode for your entire model. Why It Matters Import mode loads data into memory using VertiPaq, generally providing fast report interactions. However, the data depends on refreshes. DirectQuery queries the source system when data is requested, allowing fresher data but making performance more dependent on the source database, network, and query complexity. Composite models allow you to combine Import and DirectQuery tables within the same semantic model. When to Consider a Composite Model Use a composite model when different parts of your data have different requirements. For example: Power BI Model | ┌───────────┴───────────┐ ↓ ↓ Import Mode DirectQuery ↓ ↓ Dimensions / Summary Large Fact Table Relatively stable Frequently changing You might: Keep relatively small, frequently used dimension tables in Import mode. Keep very large or frequently changing fact tables in DirectQuery. Use aggregation tables to answer common summary queries from Import-mode data while retaining access to detailed DirectQuery data. Incorporate native queries within DirectQuery sources to offload complex transformations directly to the underlying database engine. You can also extend models using DirectQuery for Power BI semantic models and Analysis Services, allowing live connection sources to be combined with imported data. Technical Note Composite models can introduce additional complexity, particularly when relationships cross storage modes. Test the model with realistic report queries and user interactions rather than assuming that combining storage modes will automatically improve performance. Pro Tip Don't choose DirectQuery simply because your dataset is large. Ask two questions: How fresh does the data need to be? Can Import mode meet that requirement? If your data can be refreshed every few hours or once a day, Import may provide a much better user experience than DirectQuery. 5. Creating Dozens of Duplicate Time-Intelligence Measures The Mistake Your model starts with: Sales Profit Quantity Then you add: Sales YTD Profit YTD Quantity YTD Then, Sales MTD Profit MTD Quantity MTD And eventually, Sales PY Profit PY Quantity PY Before long, your Fields pane contains dozens—or even hundreds—of similar measures. The calculations may be correct, but maintaining them becomes increasingly difficult. A change to your time-intelligence logic may require updating many individual measures. The Fix: Calculation Groups Calculation groups allow you to define reusable calculation logic once and apply it to multiple measures. For example, instead of creating a separate YTD measure for every metric, you can create a YTD calculation item that works with: Sales Profit Quantity Revenue Visits Claims The same approach can be used for: YTD MTD QTD Previous Year Year-over-Year Variance Calculation groups can be created directly in Power BI Desktop from Model view. They can also be managed using external tools such as Tabular Editor. A simplified concept looks like this: Base Measure ↓ Calculation Group ↓ YTD / MTD / PY / YoY For example, a Previous Year calculation item could use: CALCULATE( SELECTEDMEASURE(), SAMEPERIODLASTYEAR('Date'[Date]) ) The SELECTEDMEASURE() function allows the same calculation item to be applied to different base measures. This can significantly reduce duplicated measures and make your semantic model easier to maintain. Important Note Calculation groups are primarily a modeling and maintainability technique. They aren't a magic performance switch. If an underlying measure is inefficient, putting it inside a calculation group won't automatically make it faster. The main benefit is reducing repetitive calculation logic and making your semantic model easier to manage. Pro Tip If you find yourself creating the same calculation pattern repeatedly for different measures, that's a good signal to investigate calculation groups. A Quick Power BI Performance Checklist Before adding more visuals or rewriting your DAX, check these five areas: ✅ Model Structure Am I using a star schema? Have I removed unnecessary columns? Are descriptive attributes stored in dimensions? ✅ Relationships Are relationships mostly single-direction? Do I have unnecessary bidirectional relationships? Are there ambiguous filter paths? ✅ Fact Tables Does each fact table represent a clear business process? Is the grain clearly defined? Am I mixing different grains in the same table? ✅ Storage Mode Do I genuinely need DirectQuery? Could Import mode handle the requirement? Would a composite model or aggregation strategy make sense? ✅ DAX & Semantic Model Am I creating many duplicate measures? Can calculation groups reduce repeated time-intelligence logic? Are my base measures already optimized? Final Thoughts Power BI performance isn't just about how many rows you have. It's about how efficiently those rows are modeled, stored, filtered, and queried. A well-designed model can give you: Faster report interactions Faster refreshes Better scalability Simpler DAX Easier maintenance More predictable results So, the next time your Power BI report starts feeling slow, don't immediately start deleting visuals or rewriting every DAX measure. Start with the model. Check your table structure. Check your relationships. Check the grain of your fact tables. Check your storage mode. And check whether you're duplicating calculation logic unnecessarily. Sometimes the biggest performance improvement comes from fixing the model underneath it.
Build a Fabric IQ Declarative Agent for Microsoft 365 Copilot Chat
Did you know Microsoft 365 Copilot Chat natively supports Fabric IQ? You can query your Power BI reports and semantic models right from the general chat, without any custom setup. Ask a business question in plain English, and M365 Copilot automatically discovers the most relevant Fabric artifact you have access to, inspects its schema, runs queries, and returns a cited answer alongside your broader Microsoft 365 context. When to build a dedicated agent The discovery step works well and can often identify the right items on the first try, but it's not without tradeoffs. If you rely on a curated set of reports and semantic models and want fast, repeatable answers with less guesswork, we can do better. A dedicated agent lets you: Skip the discovery lag and ambiguity. In general chat, M365 Copilot may fan out searches across emails, Teams chats, OneDrive decks, and all your workspaces before settling on a Power BI item, adding several seconds of latency. If you have a dozen reports about "Sales" or "Revenue", it may pick the wrong one or return inconsistent answers from one day to the next. Agents let Copilot bypass tenant-wide search by routing queries to the correct report or model immediately. Create repeatable, controlled experiences without interference from Office content. Because general chat blends all your Microsoft 365 content, an answer might pull numbers from an outdated email thread or Word document instead of your authoritative Power BI report. Scoping the agent to your trusted Power BI reports and semantic models, and only the M365 sources you explicitly enable, ensures every user gets consistent, governed answers. Show native zero-prompt starters and improve follow-ups. Guide users with tailored conversation starters from the moment they open the agent. Provide them with smarter, context-aware follow-up suggestions after each answer. Control presentation and charting logic. You decide how metrics are defined and presented to users, how tables are formatted, and influence how charts are styled and labeled. Package and share easily. A declarative agent is easy to distribute across your organization in a few clicks. In this blog post, I'll walk you through how you can create your own agent for Microsoft 365 Copilot Chat. We'll build it as a declarative agent and use GitHub Copilot to assist with the development. How the Microsoft 365 Copilot declarative agent uses routing instructions and Fabric IQ MCP to select among sales, finance, and inventory reports or semantic models. What we're building Here is our AdventureWorks Insights declarative agent in action, greeting users with tailored zero-prompt conversation starters, and then answering directly from our curated Power BI semantic model: AdventureWorks Insights declarative agent zero-prompt home screen showing tailored conversation starters. AdventureWorks Insights declarative agent ranking five sales territories, with sales, share of all sales and a source citation. How declarative agents work Under the hood, a declarative agent runs on the same orchestrator and foundation models as M365 Copilot Chat. It is configured through a lightweight set of files that define: identity and behavior (name, instructions, constraints, and conversation starters), knowledge (approved grounding sources), capabilities and actions (built-in tools like Code Interpreter or MCP-based plugins), and host metadata (manifest.json and icons for packaging and distribution). A declarative agent project is packaged from a single appPackage/ folder alongside your Agents Toolkit project configuration: my-fabric-iq-agent/ ├── m365agents.yml # Build & validate workflow ├── env/.env.dev # Environment variables └── appPackage/ ├── manifest.json # App identity & icons ├── declarativeAgent.json # Starters, capabilities & actions ├── agent_instructions.md # Routing & chart rules (max 8,000 chars) ├── ai-plugin.json # Fabric IQ MCP plugin manifest └── color.png / outline.png # 192x192 & 32x32 icons Copy-paste prompt to get started You don't need to write these JSON files by hand. Unzip the starter template (.zip) (or use an empty folder), then copy the instructions below and paste them as-is into GitHub Copilot App, CLI, Claude Code, or your favorite harness. The coding agent will interview you for your target reports and preferences, inspect your schemas, and generate the entire project for you. Before you start, think about what you want your agent to do: Capabilities: turn on CodeInterpreter so the agent can write Python to render custom charts. If your users want to blend Power BI numbers with strategy decks or team discussions, enable OneDriveAndSharePoint, GraphConnectors, or TeamsMessages. Instructions, conversation starters & follow-ups: clickable starter cards that launch your most common analyses without typing, paired with instructions that guide smart follow-up suggestions and routing. See section below for how to write effective instructions. Connectors: during authoring, your coding agent inspects your items via the Fabric IQ MCP server (https://fabriciq.svc.cloud.microsoft/v1/mcp/fabriciq), and wires the finished agent's ai-plugin.json to the M365 Fabric IQ integration (known as AskPowerBI). You can also attach additional MCP servers or API plugins if your workflow spans other systems. Branding & packaging: Your agent's short name (max 30 chars), descriptions (short max 80 chars, full max 4,000 chars), two icons (color.png at 192x192 and outline.png at 32x32), and an accent color bundled into an installable M365 app package. Instructions to copy and paste Build a Microsoft 365 Copilot declarative agent backed by the Fabric IQ MCP server (m365agents.yml, env/.env.dev, and appPackage/: declarativeAgent.json, agent_instructions.md, ai-plugin.json, manifest.json, color.png, outline.png). Template: Fetch https://community.fabric.microsoft.com/blog/community_blog/build-a-fabric-iq-declarative-agent-for-microsoft-365-copilot-chat/5377379 (or unzip my-fabric-iq-agent-template.zip) and reuse its JSON templates; skip external doc fetches unless troubleshooting. Workflow: 1. Gather inputs: Ask which Power BI reports/models to include (names, URLs, or discover via Fabric IQ MCP), agent name, and optional capabilities beyond `CodeInterpreter` (`OneDriveAndSharePoint`, `TeamsMessages`, `GraphConnectors`). Wait for my reply. 2. Inspect in parallel: Call `GetReportMetadata` and `GetSemanticModelSchema` in parallel (both accept a Report GUID; use `DiscoverArtifacts` only to resolve names). Run one batched `ExecuteQuery` only if sample values or date ranges are missing. 3. Scaffold strict JSON (no `//` comments): Create `declarativeAgent.json` (v1.8, `"instructions": "$[file('agent_instructions.md')]"`, 4-6 starters), copy `ai-plugin.json` (v2.4, `AskPowerBI`) verbatim, create `manifest.json` (v1.25) + `m365agents.yml`, and generate 192x192 `color.png` & 32x32 `outline.png` (`agent_instructions.md` max 8,000 chars; `name.short` max 30; `description.short` max 80). 4. Write `appPackage/agent_instructions.md` (max 8,000 chars): (1) Artifact Routing Table with GUIDs & rules to skip `DiscoverArtifacts`, (2) 4-6 Example Questions with follow-up context, (3) Producing Answers rules, (4) Charting Guidelines (lollipop/dumbbell/line, light background, business axis titles). 5. Package & validate (min 90s timeout): Run `npx --yes @microsoft/m365agentstoolkit-cli package --env dev && npx --yes @microsoft/m365agentstoolkit-cli validate --env dev`, then help run `provision --env dev` or sideload `appPackage/build/appPackage.dev.zip`. Testing and distributing your agent Once your files are ready, run these commands from your project root using the Microsoft 365 Agents Toolkit CLI, or ask Copilot to run them for you: # 1. Build appPackage/build/appPackage.dev.zip (inlines agent_instructions.md) npx --yes @microsoft/m365agentstoolkit-cli package --env dev # 2. Validate schemas, limits & icons npx --yes @microsoft/m365agentstoolkit-cli validate --env dev # 3. Provision & sideload to M365 npx --yes @microsoft/m365agentstoolkit-cli provision --env dev Test in Copilot Chat. Open m365.cloud.microsoft/chat, select your agent from the right-hand rail, and click your zero-prompt conversation starters to verify routing, citations, and chart rendering. Share with your team or organization. Run npx --yes Microsoft/m365agentstoolkit-cli share --scope shared to grant access to specific security groups, or submit appPackage/build/appPackage.dev.zip to the Microsoft 365 Admin Center (Settings > Integrated apps) for tenant-wide catalog deployment. Every user's queries continue to respect their individual Power BI permissions and row-level security (RLS). Under the hood: configuring a declarative agent This section walks you through the various files that make up a declarative agent. You don't have to understand every detail, but it's helpful to understand how the pieces fit together. declarativeAgent.json This is the core manifest, it defines your agent's identity and capabilities: { "$schema": "https://developer.microsoft.com/json-schemas/copilot/declarative-agent/v1.8/schema.json", "version": "v1.8", "name": "AdventureWorks Insights", "description": "Answers sales, finance, and inventory questions using curated Power BI reports and semantic models.", "instructions": "$[file('agent_instructions.md')]", "conversation_starters": [ {"title": "Top markets by sales", "text": "Which markets bring in the most sales? Show the top five and their share of total sales."}, {"title": "Category breakdown", "text": "What product categories drive our leading market?"}, {"title": "Margin vs. stock check", "text": "Compare gross margin and current stock levels for our top product category."} ], "capabilities": [{"name": "CodeInterpreter"}], "behavior_overrides": {"default_response_mode": "Think deeper"}, "actions": [{"id": "fabric-iq-action", "file": "ai-plugin.json"}] } Note: Keep JSON strict (no // comments); package inlines agent_instructions.md (max 8,000 chars). Adding Fabric IQ connection via ai-plugin.json The "actions" entry above points to ai-plugin.json in the same appPackage/ folder. This file connects your declarative agent to the Fabric IQ M365 MCP integration (AskPowerBI) and maps Power BI source metadata into native M365 Copilot citations and sensitivity labels. You can drop this ready-made version straight into your project: { "$schema": "https://developer.microsoft.com/json-schemas/copilot/plugin/v2.4/schema.json", "schema_version": "v2.4", "namespace": "fabriciq", "name_for_human": "Fabric IQ", "description_for_human": "Answers questions grounded in curated Power BI reports and semantic models.", "description_for_model": "Ask data questions about the Power BI artifacts in instructions. Pass artifactId from the routing table.", "functions": [{ "name": "AskPowerBI", "description": "Answers a Power BI report or model question via DAX. Prefer Report GUIDs; include prior context on follow-ups.", "parameters": {"type": "object", "properties": {"artifactId": {"type": "string"}, "query": {"type": "string"}, "context": {"type": "string"}}, "required": ["artifactId", "query"]}, "capabilities": {"response_semantics": {"data_path": "$.content[*]._meta.artifact_citation", "properties": {"title": "$.Name", "subtitle": "$.Description", "url": "$.Url", "information_protection_label": "$.SensitivityLabelId", "thumbnail_url": "$.IconUrl"}}} }], "runtimes": [{ "type": "RemoteMCPServer", "auth": {"type": "None"}, "spec": { "url": "https://api.fabric.microsoft.com/v1/mcp/fabricaihub/integrations/m365", "mcp_tool_description": {"tools": [{ "name": "AskPowerBI", "description": "Answers a Power BI report or model question via DAX.", "inputSchema": {"type": "object", "properties": {"artifactId": {"type": "string"}, "query": {"type": "string"}, "context": {"type": "string"}}, "required": ["artifactId", "query"]}, "annotations": {"destructiveHint": false, "openWorldHint": false, "readOnlyHint": true}, "_meta": {"variants": "Fabric.Routing.M365.V1", "microsoft/response_semantics": {"data_path": "$.content[*]._meta.artifact_citation", "properties": {"title": "$.Name", "subtitle": "$.Description", "url": "$.Url", "information_protection_label": "$.SensitivityLabelId", "thumbnail_url": "$.IconUrl", "x-ms_unique_id": "$.ArtifactId"}}} }]} }, "run_for_functions": ["AskPowerBI"] }] } How to write efficient instructions Tip: Usually, it is best to put model-specific instructions and examples as close to the data source as possible. If your instructions are model-specific, like covering how to write DAX queries or which parts of your schema are relevant, consider adding them to the semantic model's custom instructions through Power BI's Prep data for AI experience. Keep them in your declarative agent's instructions only when you can't edit the underlying semantic model, or when you want rules that apply exclusively to this agent without changing how the model behaves elsewhere. When you need to define behavior at the agent level, write instructions to agent_instructions.md. We will structure it around four focused blocks. 1. Build an artifact routing table When you include exact artifact GUIDs and contents in a markdown table, the agent skips the discovery search call and routes straight to the right report or semantic model. ## Artifact Routing Table Match the question to the right source and pass Artifact ID directly to AskPowerBI: | Artifact Name | Type | Artifact ID | Key Content | | :--- | :--- | :--- | :--- | | **AdventureWorks Sales** | Report | `60763385-b565-411a-b266-05c22543d6e4` | `[Total Sales]`, `[Orders]`; `Sales Territory`, `Category`, `Fiscal Year` | | **Global Margin Model** | Model | `14c445e9-24b5-4ded-86b4-f2e1db5b0f34` | `[Gross Margin %]`, `[COGS]`, `[Net Revenue]`; `Cost Center`, `Category` | | **Supply Chain Tracker** | Report | `9b2184a1-4c90-4e12-8a15-7f8e9102bc31` | `[Units on Hand]`, `[Backorder Rate %]`; `Warehouse`, `Supplier` | Routing rules: - Sales, volume, and territory rankings to **AdventureWorks Sales**. - Cost, profitability, and gross margin to **Global Margin Model**. - Stock and warehouse availability to **Supply Chain Tracker**. - Cross-domain questions to query each artifact separately and align by `Category`. 2. Include example questions Give the model a few concrete examples so it learns how to disambiguate similar metrics and carry context across turns: ## Example Questions & Routing - "Which markets lead in sales?" to Query **AdventureWorks Sales** (`[Total Sales]` by `Sales Territory[Region]`). - "What drives the leading market?" to Keep prior territory and period; break down `[Total Sales]` by `Product[Category]`. - "How is gross margin trending for Bikes?" to Switch to **Global Margin Model**, keep `Category = "Bikes"`, query `[Gross Margin %]` by `Fiscal Year`. - "What is our revenue?" to Clarify gross `[Total Sales]` (Sales report) vs. `[Net Revenue]` (Finance model). 3. Define how to produce answers Tip: Only add custom answer-formatting rules if the out-of-the-box experience does not suit your needs. Fabric IQ's built-in instructions are already tuned to provide strong defaults for grounding, charts, citations, and formatting. If your audience requires a specific executive style, such as always leading with a one-sentence headline or suggesting cross-model follow-ups, you can layer those preferences in: ## Producing Answers - **Lead with the headline.** Start with a one-sentence finding naming the leader, exact figure, and share of total. - **State denominator and period.** Specify the date range and whether percentages are share of all sales or top N. - **Suggest follow-ups.** End with 2-3 next questions grounded in your routed artifacts. 4. Standardize charting logic Because CodeInterpreter is enabled, you can instruct the agent to render clean, publication-grade visuals automatically, without the user asking for a chart: ## Charting Guidelines - Include a chart for any comparison of 3+ items. - **Visual forms:** Sorted horizontal lollipop for rankings, dumbbell for 2-period comparisons, line for trends. - **Axis labels:** Use business titles (`Market`, `Sales ($M)`); never `X` or `Y`. - **Styling:** Light background, direct data labels, quiet gridlines, and `#b11f4b` accent for the leader. (Tip: In M365 Copilot Chat, users can also open the chart menu and select Display as interactive chart for native inline hover tooltips.)Scaling Machine Learning for Data Science with Dask in Microsoft Fabric
In this edition, we’re exploring why Dask has become such a practical tool for scaling machine learning in Python as your data grows. By the time you’re done reading, you’ll have a clear sense of how Dask fits naturally into a data professional’s way of thinking, without forcing you to give up the tools or habits you already rely on.
