Unable to create lakehouse in Trial capacity (trial is active for 25 days)
I am working on a project to load data from ADLS gen2 blob storage into staging tables in lakehouse table, from there is will go to target table of another lakehouse table. I have couple of pipeline and notebook for this task. I am not able to create the target lakehouse because of the below reason: I do not have any pipeline/notebook running in the background. Currently I have only below workspaces and rest are deleted already. This is my trial capacity setting with Sku/size FTL4. Is it possible to increase the capacity of the trial to complete the project? I have other projects to work on as well. Please guide on how can I resolve the issue. Thank you in advance.
Architecture Issue : 600 tables treated in loop in Bronze
Dear all, Our architecture consists of four layers: Raw, Bronze, Silver, and Gold. Raw is composed of a single workspace, and Bronze also has its own workspace. Each workspace has its own lakehouse inside it. Tables land in Raw, coming from different sources, and are then shortcutted into Bronze. In Bronze, we have created persisted tables that mirror the Raw tables (with a few additional fields) but include historisation. These are fed incrementally, on a daily basis, based on the fields available in the Raw tables plus the last-run information stored in a table called load_audit, which records successful/failed runs and their characteristics. Both the audit table updates and the movement of data from shortcutted tables into persisted tables are handled by a single notebook. Problem: We are having issues with a specific source. This source comes from SQL Server 2019 and has more than 600 tables. Mirroring from source into Raw is working fine. The issue lies in appending data from the shortcutted tables into the Bronze persisted tables (the incremental load). Currently, this is done sequentially: the notebook loops through each table, applies the increment, and moves on to the next. This is taking a considerable amount of time — imagine, for example, 1 minute per table. There is also the problem of small files accumulating while feeding the audit load table. Simplified, it works like this: For each table: Incremental load into persisted table Insert run result into the audit load table Move to next table I discussed this with the data engineering team, and they are proposing the following: Create two separate notebooks, each with their own common utilities inside. 3.1. One will handle the large tables, and the other the small tables. 3.2. The large tables will be processed using the Spark engine (Spark workload), and the small tables using Python/Pandas. 3.3. A possible rule of thumb: if a table has fewer than 100K records, it goes to the small-table notebook; otherwise, it goes to the big-table notebook. 3.4. Partitioning by target schema and target table. 3.5. How do we address parallel execution across tables within each notebook? Splitting into two notebooks by size doesn't by itself remove the sequential loop — should we also parallelize within each notebook (e.g., concurrent threads for the Pandas notebook, or a Fabric pipeline ForEach with concurrency for the Spark notebook)? 3.6. How does this approach address the small-files problem on the audit load table? Splitting the workload doesn't inherently batch or compact writes — would we need OPTIMIZE/compaction scheduled on the audit table, or should writes be batched instead of one insert per table? 3.7. Should the 100K-record threshold be based on row count, or would data volume (MB/GB) or column count be a more reliable measure? A 100K-row table with 5 narrow columns and a 100K-row table with 200 wide/text columns have very different costs. 3.8. Where will the Pandas workload actually execute — on the driver node? If so, could running many small-table loads concurrently create memory/CPU contention with the Spark jobs running alongside it? What do you think about this approach? Could you please provide some guidance? Thanks a lot, Pedro
How to efficiently maintain complex multi-table aggregations incrementally in Microsoft Fabric?
How to efficiently maintain complex multi-table aggregations incrementally in Microsoft Fabric? Comparison with Snowflake Dynamic Tables Hi everyone, I have a scenario where source data is continuously/incrementally loaded into Microsoft Fabric. We have multiple entities such as Customer, Product, Orders, Sales, Inventory, Invoices and Payments. We need to create business-ready/precomputed tables by joining multiple entities and performing aggregations. For example, a simplified transformation could be Customer + Orders + Product, followed by joins and aggregation by Customer, Product and Month. The challenge is not simply adding new records. A new incremental record can change an existing aggregation group. For example, suppose we already have Customer C001, Product P001, Month Jan and Sales of 1,000. If a new order arrives for the same Customer, Product and Month with Sales of 250, the existing aggregate needs to become 1,250. This becomes more challenging when the transformation involves multiple joins, GROUP BY, aggregations, DISTINCT counts, multiple related entities, and updates or deletes in source data. If we rebuild the complete precomputed table during every refresh, a large amount of historical data may need to be recomputed. As data volume and the number of entities increase, this could result in more data, larger transformations, more compute and longer refresh times. We are therefore trying to determine the best incremental maintenance pattern for these types of tables in Microsoft Fabric. SNOWFLAKE COMPARISON One approach we have been looking at in Snowflake is Dynamic Tables. Dynamic Tables provide a declarative way to define transformed or materialized results and allow Snowflake to manage refreshes based on changes and the configured refresh mode and target lag. The capability we are particularly interested in is maintaining the derived result incrementally instead of treating every refresh as a complete rebuild. We also understand that Snowflake Dynamic Tables do not make every transformation automatically incremental. Query shape, joins, aggregations and unsupported constructs can affect whether incremental refresh is possible. FABRIC OPTIONS WE ARE EVALUATING The first option we are evaluating is Lakehouse Materialized Lake Views (MLVs). MLVs appear to provide a similar architectural pattern where source tables are transformed into a persisted materialized result and the platform manages refresh. We are particularly interested in the optimal refresh and incremental refresh capabilities. We would like to understand how well MLVs handle transformations involving multiple joins, GROUP BY, aggregations, large historical datasets and new records that affect existing aggregation groups. For example, if Customer, Orders and Product are joined and aggregated by Customer, Product and Month, and a new order affects an existing historical Customer + Product + Month group, how does MLV incremental refresh handle this? The second option we are considering is maintaining physical precomputed or serving tables in Fabric Warehouse. The architecture would be roughly Lakehouse, Clean and Validated Data, Transformation and Aggregation, Warehouse Precomputed or Serving Table, Semantic Model and Power BI. One possibility is to use CTAS or staging-based patterns to generate the serving tables. However, if the transformation is rebuilt from the complete historical dataset, we may still end up recomputing a large amount of data during every refresh. We would therefore like to understand whether there is a recommended Fabric pattern for maintaining these physical tables incrementally, particularly when new or changed source records can affect existing aggregate groups. WHAT WE ARE TRYING TO DETERMINE We are not trying to claim that one approach is better than another. We are trying to identify the closest Fabric architecture to the incremental-maintenance capability we are familiar with from Snowflake Dynamic Tables. Can MLVs efficiently maintain complex multi-table joins and aggregations incrementally as data grows? What types of SQL transformations cause MLVs to fall back to full refresh? How does MLV incremental refresh behave when new records modify an existing aggregation group? For complex transformations, is it better to break the logic into multiple MLVs or intermediate layers? If using Fabric Warehouse physical serving tables, what is the recommended pattern for avoiding full historical recomputation? Are there Fabric-native patterns for identifying and recomputing only the affected partitions, keys or aggregation groups? For large enterprise datasets, what approach have others found most scalable and maintainable? OUR CURRENT ARCHITECTURE Our architecture is roughly Source Systems, Bronze, Clean and Validated Data, Precomputed or Serving Tables, Gold or Consumption Layer, Semantic Model and Power BI. The objective is to perform expensive joins and aggregations during data processing rather than repeatedly during interactive BI queries. The open question is how best to maintain the Precomputed or Serving layer incrementally as source data continues to grow and change. I would really appreciate input from anyone who has implemented this at scale in Microsoft Fabric, particularly comparisons between Fabric MLVs, Warehouse-based serving tables and other Fabric-native incremental processing patterns. Thanks in advance!Users that subscribe to PowerBi Paginated Report from another tenant
Dear all, One of our project teams developed a custom Fabric notebook that uses the Graph REST API to send a paginated report to multiple users (mailboxes). This custom notebook extracts the report, saves it as an Excel file, and sends it to multiple mailboxes outside the tenant. From what I understand, this approach was taken because, as the users belong to a different tenant, they would not be able to receive the email or subscribe to the report directly. However, I have a feeling there may be a workaround for this. How can users subscribe to a paginated report and receive it via email if they are not part of the tenant? Would it be sufficient to invite these users into our Entra ID as guests — would they then be able to subscribe to the report (themselfs) and receive it in their external mailboxes? Or is the Power BI Service only able to send emails to mailboxes within the same tenant? If the Power BI Service does allow guests to subscribe and receive emails, do we also need to unblock any other settings on the Fabric Admin page, alongside adding them as guests? Many thanks, Pedro
Using a standard date table to avoid Auto Date/Time for shared Fabric Semantic Models
How can one use an explicit enterprise Date table and to avoid Auto date/time in shared semantic models within Microsoft Fabric, without requiring developers to manually disable the Auto date/time feature in Power BI Desktop? The goal is to reduce unnecessary date/time tables and semantic model size while ensuring all date-related filtering and time intelligence use the central enterprise Date table. Is there a supported way to enforce or automate this configuration?
Microsoft Fabric Airflow DAG Queued for 30 Minutes on Starter Pool
Hi everyone, I am trying to run a simple data ingestion DAG using Apache Airflow Jobs in Microsoft Fabric. The purpose of the DAG is to import data into a Microsoft Fabric Lakehouse. The DAG is intentionally simple, and I am currently trying to understand an issue related to DAG execution and queuing. Environment: Orchestration: Apache Airflow Job in Fabric Target: Microsoft Fabric Lakehouse Schedule: Manual (schedule=None) Authentication: OneLake token through an environment variable Compute/Pool: Starter Pool Issue: When I manually trigger the DAG, the DAG run remains in the Queued state for approximately 30 minutes. During this time, the task does not appear to start executing. After approximately 30 minutes, the DAG run eventually changes to Failed.Org Apps - Getting warning on web content
Team - I came across another scenario where there is an org app/Workspace PowerBI app that has Power BI reports & Embedded Tableau reports. These are trusted reports & HPTTS links . When app link is shared with end user they get a warning banner on top of the report with message - "This page contains contents not provided by Microsoft. Web contents could contain code with security or privacy risks" is there a way to suppress this message ? Not sure if anyone else is having similar issue. I was doing some research on this and found similar post in the community dated 10 years ago- there was no fix at that time and it was decided to put in as an idea. https://community.fabric.microsoft.com/discussions/power-bi-designer/getting-warning-on-web-content-in-power-bi-dashboard/35989/replies/36329 -Patt
