Fabric IQ: Do Ontology entity synonyms work with Data Agent?
Hi, everyone! Short intro Currently I'm using the "super-duper-mega-nano-ultra" product - Microsoft Fabric to build natural language processing flow on-top of Microsoft Fabric Warehouse data. As for now It's rather a POC than production solution. I found a lot of the official Microsoft' documentation related to my task, but I have a little problem... The solution architecture (high-level) I made some investigations and as the result is the following architecture, which I want to implement (picture below): The idea is the next: the Microsoft Fabric Warehouse schema is connected to Microsoft Fabric Lakehouse, using shortcut; the Microsoft Fabric Ontology consumes the Microsoft Fabric Lakehouse as a data source for data binding; the Microsoft Fabric Data Agent uses a Microsoft Fabric Ontology (enriched with business context) to process natural language questions. The Microsoft Fabric Warehouse contains the following (dummy) objects (picture below): Tenant settings Microsoft documentation says, that specific tenant configurations should be applied to use Microsoft Fabric Ontology with Microsoft Fabric Data Agent (https://learn.microsoft.com/en-us/fabric/data-science/data-agent-tenant-settings) - everything is configured properly. Ontology configuration My Microsoft Fabric Ontology is configured as below (the configuration is influenced by Microsoft Fabric Ontology tutorial, which can be found here - https://learn.microsoft.com/en-us/fabric/iq/ontology/overview). Main view - two entities with a single relationship: ETLEntity entity configuration - the entity has the description, one synonym, metadata: ETLEntityRun entity configuration - the entity has the description, one synonym, metadata: As for now the Microsoft documentation says, that Microsoft Fabric Ontology descriptions, synonyms, metadata help Microsoft Fabric Data Agent to better understand the context (https://learn.microsoft.com/en-us/fabric/iq/ontology/how-to-add-semantic-enrichment). The problem My Microsoft Fabric Data Agent is connected to my Microsoft Fabric Ontology, which is described above, but the agent can't answer the simple questions about entities and the questions examples are provided below (the Microsoft Fabric Ontology Graph model was refreshed successfully before questions were asked): ETLEntity successful question without synonym usage: ETLEntityRun successful question without synonym usage: ETLEntity failed question with synonym usage: ETLEntityRun failed question with synonym usage: Looks like Microsoft Fabric Data Agent can't figure out, which entities are unicorn/wizard, even if they have appropriate synonyms. It's not my first iteration - I tried a lot, but result still the same every time. I feel like I missed something obvious in my configuration, but what... What are your thoughts? P.S.: the provided configuration is simple; objects and their metadata has no business context - It's just a sample, which I built to test some scenario; I think It's enough to check such use-case.
How to efficiently maintain complex multi-table aggregations incrementally in Microsoft Fabric?
How to efficiently maintain complex multi-table aggregations incrementally in Microsoft Fabric? Comparison with Snowflake Dynamic Tables Hi everyone, I have a scenario where source data is continuously/incrementally loaded into Microsoft Fabric. We have multiple entities such as Customer, Product, Orders, Sales, Inventory, Invoices and Payments. We need to create business-ready/precomputed tables by joining multiple entities and performing aggregations. For example, a simplified transformation could be Customer + Orders + Product, followed by joins and aggregation by Customer, Product and Month. The challenge is not simply adding new records. A new incremental record can change an existing aggregation group. For example, suppose we already have Customer C001, Product P001, Month Jan and Sales of 1,000. If a new order arrives for the same Customer, Product and Month with Sales of 250, the existing aggregate needs to become 1,250. This becomes more challenging when the transformation involves multiple joins, GROUP BY, aggregations, DISTINCT counts, multiple related entities, and updates or deletes in source data. If we rebuild the complete precomputed table during every refresh, a large amount of historical data may need to be recomputed. As data volume and the number of entities increase, this could result in more data, larger transformations, more compute and longer refresh times. We are therefore trying to determine the best incremental maintenance pattern for these types of tables in Microsoft Fabric. SNOWFLAKE COMPARISON One approach we have been looking at in Snowflake is Dynamic Tables. Dynamic Tables provide a declarative way to define transformed or materialized results and allow Snowflake to manage refreshes based on changes and the configured refresh mode and target lag. The capability we are particularly interested in is maintaining the derived result incrementally instead of treating every refresh as a complete rebuild. We also understand that Snowflake Dynamic Tables do not make every transformation automatically incremental. Query shape, joins, aggregations and unsupported constructs can affect whether incremental refresh is possible. FABRIC OPTIONS WE ARE EVALUATING The first option we are evaluating is Lakehouse Materialized Lake Views (MLVs). MLVs appear to provide a similar architectural pattern where source tables are transformed into a persisted materialized result and the platform manages refresh. We are particularly interested in the optimal refresh and incremental refresh capabilities. We would like to understand how well MLVs handle transformations involving multiple joins, GROUP BY, aggregations, large historical datasets and new records that affect existing aggregation groups. For example, if Customer, Orders and Product are joined and aggregated by Customer, Product and Month, and a new order affects an existing historical Customer + Product + Month group, how does MLV incremental refresh handle this? The second option we are considering is maintaining physical precomputed or serving tables in Fabric Warehouse. The architecture would be roughly Lakehouse, Clean and Validated Data, Transformation and Aggregation, Warehouse Precomputed or Serving Table, Semantic Model and Power BI. One possibility is to use CTAS or staging-based patterns to generate the serving tables. However, if the transformation is rebuilt from the complete historical dataset, we may still end up recomputing a large amount of data during every refresh. We would therefore like to understand whether there is a recommended Fabric pattern for maintaining these physical tables incrementally, particularly when new or changed source records can affect existing aggregate groups. WHAT WE ARE TRYING TO DETERMINE We are not trying to claim that one approach is better than another. We are trying to identify the closest Fabric architecture to the incremental-maintenance capability we are familiar with from Snowflake Dynamic Tables. Can MLVs efficiently maintain complex multi-table joins and aggregations incrementally as data grows? What types of SQL transformations cause MLVs to fall back to full refresh? How does MLV incremental refresh behave when new records modify an existing aggregation group? For complex transformations, is it better to break the logic into multiple MLVs or intermediate layers? If using Fabric Warehouse physical serving tables, what is the recommended pattern for avoiding full historical recomputation? Are there Fabric-native patterns for identifying and recomputing only the affected partitions, keys or aggregation groups? For large enterprise datasets, what approach have others found most scalable and maintainable? OUR CURRENT ARCHITECTURE Our architecture is roughly Source Systems, Bronze, Clean and Validated Data, Precomputed or Serving Tables, Gold or Consumption Layer, Semantic Model and Power BI. The objective is to perform expensive joins and aggregations during data processing rather than repeatedly during interactive BI queries. The open question is how best to maintain the Precomputed or Serving layer incrementally as source data continues to grow and change. I would really appreciate input from anyone who has implemented this at scale in Microsoft Fabric, particularly comparisons between Fabric MLVs, Warehouse-based serving tables and other Fabric-native incremental processing patterns. Thanks in advance!
Having Error when connecting to data source
Hi all, I am getting this error when I try to use the Copy Data with the Dynamic AX connector to connect my D365fo data to the Lakehouse. I have setup the APP registration from Azure with my Microsoft Entra ID. Below is the error detail I get. Error code: 23205 Details: Requesting response from path and query failed with status code Forbidden and message {"Message":"Please verify that the user is valid and set up correctly."}.
Users that subscribe to PowerBi Paginated Report from another tenant
Dear all, One of our project teams developed a custom Fabric notebook that uses the Graph REST API to send a paginated report to multiple users (mailboxes). This custom notebook extracts the report, saves it as an Excel file, and sends it to multiple mailboxes outside the tenant. From what I understand, this approach was taken because, as the users belong to a different tenant, they would not be able to receive the email or subscribe to the report directly. However, I have a feeling there may be a workaround for this. How can users subscribe to a paginated report and receive it via email if they are not part of the tenant? Would it be sufficient to invite these users into our Entra ID as guests — would they then be able to subscribe to the report (themselfs) and receive it in their external mailboxes? Or is the Power BI Service only able to send emails to mailboxes within the same tenant? If the Power BI Service does allow guests to subscribe and receive emails, do we also need to unblock any other settings on the Fabric Admin page, alongside adding them as guests? Many thanks, PedroUsers that subcribe to PowerBi Paginated Report from another tenant
Dear all, One of our project teams developed a custom Fabric notebook that uses the Graph REST API to send a paginated report to multiple users (mailboxes). This custom notebook extracts the report, saves it as an Excel file, and sends it to multiple mailboxes outside the tenant. From what I understand, this approach was taken because, as the users belong to a different tenant, they would not be able to receive the email or subscribe to the report directly. However, I have a feeling there may be a workaround for this. How can users subscribe to a paginated report and receive it via email if they are not part of the tenant? Would it be sufficient to invite these users into our Entra ID as guests — would they then be able to subscribe to the report (themselfs) and receive it in their external mailboxes? Or is the Power BI Service only able to send emails to mailboxes within the same tenant? If the Power BI Service does allow guests to subscribe and receive emails, do we also need to unblock any other settings on the Fabric Admin page, alongside adding them as guests? Many thanks, Pedro
Unable to connect Fabric Web API to Partner Center Referrals API – 401 Incorrect Audience
I am trying to connect Microsoft Fabric to the Microsoft Partner Center Referrals API using the Web API connector. API Endpoint GET https://api.partner.microsoft.com/v1.0/engagements/referrals?$orderby=createdDateTime desc This is related to: Partner Center → Referrals → Insights → Referral Summary Current Setup Azure App Registration An Azure App Registration has been created. The following API permissions are configured: Microsoft Partner - user_impersonation Microsoft Partner Center - user_impersonation Admin consent has been granted. For Microsoft Partner Center only Delegated permissions are available. No Application permissions are exposed. Partner Center The Azure application is registered in Partner Center under: User Management → Microsoft Entra Applications The test user has: Referrals Admin assigned for the entire organization. MFA is enabled and successfully completed. Authentication Test I tested Authorization Code Flow (App + User) and successfully obtained a delegated access token. JWT claims: { "aud": "https://api.partnercenter.microsoft.com", "scp": "user_impersonation", "idtyp": "user", "amr": ["pwd", "mfa"] } This confirms: User authentication works MFA works Authorization Code Flow works Delegated permission is applied Referrals Admin role is assigned Problem When calling: GET https://api.partner.microsoft.com/v1.0/engagements/referrals?$orderby=createdDateTime desc with the delegated token, the response is: { "statusCode": 401, "message": "Unauthorized: Invalid Authorization header, incorrect audience" } Token audience: https://api.partnercenter.microsoft.com Referrals API endpoint: https://api.partner.microsoft.com The API appears to reject the token because of an audience mismatch. Fabric Behavior When attempting to create a connection in Fabric Dataflow Gen2 using: Authentication = Organizational Account Fabric returns: Unable to create connection using an organizational account. Please check to see if the data source path is correct and that the data source supports using an organizational account. Since the same API call fails outside Fabric with the exact audience error, I suspect this is not a Fabric issue but an OAuth configuration issue for the Referrals API. Additional Findings The following scope successfully generates a delegated token: https://api.partnercenter.microsoft.com/user_impersonation However, the resulting token audience remains: https://api.partnercenter.microsoft.com I also verified that the only permission exposed by Microsoft Partner Center in Azure App Registration is: user_impersonation I do not see any Referrals-specific permissions such as: Referrals.Read.All Referrals.ReadWrite.All Question What is the correct OAuth resource URI / scope required to obtain a token that is accepted by: https://api.partner.microsoft.com/v1.0/engagements/referrals Has anyone successfully connected Fabric, Postman, Bruno, Power BI, or a custom application to the Referrals API and can share the required authentication configuration? Thanks!
workspace identity failing to authenticate to sql endpoint
Starting last night 26-Aug we are seeing repeated connection failures to our sql endpoint for pipelines using a connection that has been valid for months. The error indicates the connection may not be being read correctly the user is blank in the error. 'LK_Pre_Silver', User: ''. Check the connection configuration is correct Failures are intermittent, but wreaking havoc with my production runs...
