Architecture Issue : 600 tables treated in loop in Bronze
Dear all, Our architecture consists of four layers: Raw, Bronze, Silver, and Gold. Raw is composed of a single workspace, and Bronze also has its own workspace. Each workspace has its own lakehouse inside it. Tables land in Raw, coming from different sources, and are then shortcutted into Bronze. In Bronze, we have created persisted tables that mirror the Raw tables (with a few additional fields) but include historisation. These are fed incrementally, on a daily basis, based on the fields available in the Raw tables plus the last-run information stored in a table called load_audit, which records successful/failed runs and their characteristics. Both the audit table updates and the movement of data from shortcutted tables into persisted tables are handled by a single notebook. Problem: We are having issues with a specific source. This source comes from SQL Server 2019 and has more than 600 tables. Mirroring from source into Raw is working fine. The issue lies in appending data from the shortcutted tables into the Bronze persisted tables (the incremental load). Currently, this is done sequentially: the notebook loops through each table, applies the increment, and moves on to the next. This is taking a considerable amount of time — imagine, for example, 1 minute per table. There is also the problem of small files accumulating while feeding the audit load table. Simplified, it works like this: For each table: Incremental load into persisted table Insert run result into the audit load table Move to next table I discussed this with the data engineering team, and they are proposing the following: Create two separate notebooks, each with their own common utilities inside. 3.1. One will handle the large tables, and the other the small tables. 3.2. The large tables will be processed using the Spark engine (Spark workload), and the small tables using Python/Pandas. 3.3. A possible rule of thumb: if a table has fewer than 100K records, it goes to the small-table notebook; otherwise, it goes to the big-table notebook. 3.4. Partitioning by target schema and target table. 3.5. How do we address parallel execution across tables within each notebook? Splitting into two notebooks by size doesn't by itself remove the sequential loop — should we also parallelize within each notebook (e.g., concurrent threads for the Pandas notebook, or a Fabric pipeline ForEach with concurrency for the Spark notebook)? 3.6. How does this approach address the small-files problem on the audit load table? Splitting the workload doesn't inherently batch or compact writes — would we need OPTIMIZE/compaction scheduled on the audit table, or should writes be batched instead of one insert per table? 3.7. Should the 100K-record threshold be based on row count, or would data volume (MB/GB) or column count be a more reliable measure? A 100K-row table with 5 narrow columns and a 100K-row table with 200 wide/text columns have very different costs. 3.8. Where will the Pandas workload actually execute — on the driver node? If so, could running many small-table loads concurrently create memory/CPU contention with the Spark jobs running alongside it? What do you think about this approach? Could you please provide some guidance? Thanks a lot, Pedro
Unable to Start Power BI Pro Trial for New Entra ID User
I am trying to set up Microsoft Fabric or Power BI for a new user in my Azure tenant, but I am unable to start the Power BI trial. Here is what I have done: I have an existing Microsoft Azure account/tenant. In the Azure portal, I created a new Microsoft Entra ID user inside the tenant. I then signed in to Microsoft Fabric using this newly created Entra ID user. I attempted to create a new workspace. When creating the workspace, I entered the workspace name and the owner was automatically populated. The license appeared to require Power BI Pro. I selected the option to upgrade Power BI trial, which indicated a 60-day trial. However, the trial could not be started. I receive the following error: Something went wrong Sorry, we are unable to start trial at this time. Please try again later or contact support. Error details: Activity ID: 6a0ca47c-5c5d-45eb-9cc2-c34cff76870d Request ID: a4535129-c542-4fe3-965f-a1fc80a2fbb1 Correlation ID: 1382be3c-9550-873b-f3c5-c8ac4931a47e Status code: 200 Time: Wed Sep 02 2026 12:29:28 GMT-0400 (Eastern Daylight Time) Service version: 13.0.28514.444 Client version: 2608.4.30752-train Could someone please help me understand why the Power BI trial cannot be started for this newly created Entra ID user? I would appreciate any guidance on the correct configuration or steps I should take to resolve this issue. Thank you!Paginated Reports
Dears, My team is using paginated reports Q1) Do they (paginated reports) respect the RLS if its implemented at the semantic layer level? They are also extracting the report , and, in some cases, during the extraction passing a country parameters and converting to excel (which they append to an e-mail and send via Graph API) I don't like the fact that the users receive the information in excel and also, I don't like the fact that currently they are filtering the report per country so that people only see their one things. This seems to me like a custom way of doing RLS.. Q2 ) Do you suggest any other approach? Thanks a lot, PedroUsers that subscribe to PowerBi Paginated Report from another tenant
Dear all, One of our project teams developed a custom Fabric notebook that uses the Graph REST API to send a paginated report to multiple users (mailboxes). This custom notebook extracts the report, saves it as an Excel file, and sends it to multiple mailboxes outside the tenant. From what I understand, this approach was taken because, as the users belong to a different tenant, they would not be able to receive the email or subscribe to the report directly. However, I have a feeling there may be a workaround for this. How can users subscribe to a paginated report and receive it via email if they are not part of the tenant? Would it be sufficient to invite these users into our Entra ID as guests — would they then be able to subscribe to the report (themselfs) and receive it in their external mailboxes? Or is the Power BI Service only able to send emails to mailboxes within the same tenant? If the Power BI Service does allow guests to subscribe and receive emails, do we also need to unblock any other settings on the Fabric Admin page, alongside adding them as guests? Many thanks, Pedro
Setting up customer training accounts
We want to deliver some training on Fabric to various customers. I want to avoid the hassles of user 1 having an account, user 2 not having an account etc, user 3 having an account but with restrictions etc. Is there a way to set up user accounts for training without having to pay for multiple Pro license training accounts, with the associated costs? Ideally, we'd have a demo environment with email addresses like [email protected] [email protected] etc. Is there any way to do that? Thank you
Microsoft Fabric Airflow DAG Queued for 30 Minutes on Starter Pool
Hi everyone, I am trying to run a simple data ingestion DAG using Apache Airflow Jobs in Microsoft Fabric. The purpose of the DAG is to import data into a Microsoft Fabric Lakehouse. The DAG is intentionally simple, and I am currently trying to understand an issue related to DAG execution and queuing. Environment: Orchestration: Apache Airflow Job in Fabric Target: Microsoft Fabric Lakehouse Schedule: Manual (schedule=None) Authentication: OneLake token through an environment variable Compute/Pool: Starter Pool Issue: When I manually trigger the DAG, the DAG run remains in the Queued state for approximately 30 minutes. During this time, the task does not appear to start executing. After approximately 30 minutes, the DAG run eventually changes to Failed.Org Apps - Getting warning on web content
Team - I came across another scenario where there is an org app/Workspace PowerBI app that has Power BI reports & Embedded Tableau reports. These are trusted reports & HPTTS links . When app link is shared with end user they get a warning banner on top of the report with message - "This page contains contents not provided by Microsoft. Web contents could contain code with security or privacy risks" is there a way to suppress this message ? Not sure if anyone else is having similar issue. I was doing some research on this and found similar post in the community dated 10 years ago- there was no fix at that time and it was decided to put in as an idea. https://community.fabric.microsoft.com/discussions/power-bi-designer/getting-warning-on-web-content-in-power-bi-dashboard/35989/replies/36329 -PattLineage in Fabric - Onelake Catalog Vs WS lineage
Dears, Seems to me that, for the time being onelake atalog only allows to see lineage at item level, meaning, seems not doable to see the things at sub-item (tables , notebooks, etc..) How about in Workspaces? there can I choose a table and see its lineage , meaning from where it was created (raw or bronze) till gold and after , reporting? and what about notebboks and gen flows? Also, any idea of if its expected that onelake catalog starts to show lineage at sub-items level? Thanks a lot, PedroFabric Copilot Chat with your data in M365 Copilot: where is business data?
Hi, Since July we can now have access to Fabric Copilot directly in Copilot M365. It's cool but...currently my configuration allows me the retrieve only the Fabric Metadata of my reports, semantic models,... I've set the correct admin settings in the 2 tenants (Copilot and Fabric): In Copilot Admin => enable "Manage access to Microsoft Fabric Data for Microsoft 365 Copilot" In Fabric Admin => + But Copilot can question metadata of Fabric items but not the data. For example: if I ask Copilot M365 to retrieve the net sales of my company XXX in 2026, he says he cannot but it can see the model details such as column name, descriptions,... So currently, this feature is not very useful for end users. Maybe I forget things. Could you help me?Fabric IQ: Do Ontology entity synonyms work with Data Agent?
Hi, everyone! Short intro Currently I'm using the "super-duper-mega-nano-ultra" product - Microsoft Fabric to build natural language processing flow on-top of Microsoft Fabric Warehouse data. As for now It's rather a POC than production solution. I found a lot of the official Microsoft' documentation related to my task, but I have a little problem... The solution architecture (high-level) I made some investigations and as the result is the following architecture, which I want to implement (picture below): The idea is the next: the Microsoft Fabric Warehouse schema is connected to Microsoft Fabric Lakehouse, using shortcut; the Microsoft Fabric Ontology consumes the Microsoft Fabric Lakehouse as a data source for data binding; the Microsoft Fabric Data Agent uses a Microsoft Fabric Ontology (enriched with business context) to process natural language questions. The Microsoft Fabric Warehouse contains the following (dummy) objects (picture below): Tenant settings Microsoft documentation says, that specific tenant configurations should be applied to use Microsoft Fabric Ontology with Microsoft Fabric Data Agent (https://learn.microsoft.com/en-us/fabric/data-science/data-agent-tenant-settings) - everything is configured properly. Ontology configuration My Microsoft Fabric Ontology is configured as below (the configuration is influenced by Microsoft Fabric Ontology tutorial, which can be found here - https://learn.microsoft.com/en-us/fabric/iq/ontology/overview). Main view - two entities with a single relationship: ETLEntity entity configuration - the entity has the description, one synonym, metadata: ETLEntityRun entity configuration - the entity has the description, one synonym, metadata: As for now the Microsoft documentation says, that Microsoft Fabric Ontology descriptions, synonyms, metadata help Microsoft Fabric Data Agent to better understand the context (https://learn.microsoft.com/en-us/fabric/iq/ontology/how-to-add-semantic-enrichment). The problem My Microsoft Fabric Data Agent is connected to my Microsoft Fabric Ontology, which is described above, but the agent can't answer the simple questions about entities and the questions examples are provided below (the Microsoft Fabric Ontology Graph model was refreshed successfully before questions were asked): ETLEntity successful question without synonym usage: ETLEntityRun successful question without synonym usage: ETLEntity failed question with synonym usage: ETLEntityRun failed question with synonym usage: Looks like Microsoft Fabric Data Agent can't figure out, which entities are unicorn/wizard, even if they have appropriate synonyms. It's not my first iteration - I tried a lot, but result still the same every time. I feel like I missed something obvious in my configuration, but what... What are your thoughts? P.S.: the provided configuration is simple; objects and their metadata has no business context - It's just a sample, which I built to test some scenario; I think It's enough to check such use-case.
