High CU consumption after enabling Workspace Monitoring
Hi everyone, I’m wondering if anyone else has experienced very high CU consumption after enabling Workspace Monitoring in Microsoft Fabric. In my case, I tested it on an F4 capacity, and the additional CU usage was significant enough that the capacity became difficult to use for the actual workloads running in the workspace. I understand that Workspace Monitoring creates and uses Fabric resources for collecting and storing logs, so some additional capacity consumption is expected. However, the overhead I’m seeing feels quite high and it makes no sense to enable this feature I’m interested if there are any answers on following questions: Have you noticed a significant CU increase after enabling Workspace Monitoring? Is the impact especially noticeable on smaller capacities such as F2/F4/F8? Have you found any practical way to reduce the monitoring overhead? Are you using Workspace Monitoring in production on smaller capacities, or do you avoid it because of the CU cost? At the moment, my concern is that on smaller capacities the monitoring workload itself can consume enough capacity to materially affect the workloads it is supposed to help monitor. Would be interested to hear about other people’s experience and whether this is expected behaviour.Org Apps - Getting warning on web content
Team - I came across another scenario where there is an org app/Workspace PowerBI app that has Power BI reports & Embedded Tableau reports. These are trusted reports & HPTTS links . When app link is shared with end user they get a warning banner on top of the report with message - "This page contains contents not provided by Microsoft. Web contents could contain code with security or privacy risks" is there a way to suppress this message ? Not sure if anyone else is having similar issue. I was doing some research on this and found similar post in the community dated 10 years ago- there was no fix at that time and it was decided to put in as an idea. https://community.fabric.microsoft.com/discussions/power-bi-designer/getting-warning-on-web-content-in-power-bi-dashboard/35989/replies/36329 -Patt
Upsert implementation in Fabric Lakehouse
Hi Fabric Community, We are building a data platform in Microsoft Fabric using Salesforce and Marketo data, following the Medallion Architecture (Bronze / Silver / Gold). We are currently considering whether to use Replace or Upsert when writing data from Bronze to Silver and from Silver to Gold. Our current assumptions are: Bronze → Silver Salesforce Lead / Contact / Account / Opportunity, etc.: Upsert, because existing records can be updated Marketo Person, etc.: Upsert, because existing records can be updated Marketo Activity: Deduplicate and Append, since it is event/log data Silver → Gold Silver data is transformed and joined to create Gold tables optimized for AI use. Dataflow Gen2 with Replace is one option, but we are concerned about processing time and Capacity Unit (CU) consumption as the data volume grows in the future. My understanding from Microsoft Learn is that Dataflow Gen2 supports Replace / Append when writing to a Lakehouse, while Pipeline Copy Activity supports Upsert to a Lakehouse Table and allows key columns to be configured through the UI. Therefore, we would like to avoid using Notebook / Spark Delta MERGE where possible and are considering a low-code/no-code approach where Dataflow Gen2 handles transformations and Pipeline / Copy Activity handles the Upsert into the Lakehouse table. For example: Bronze → Silver Bronze → Dataflow Gen2 (filter/transform changed records) → temporary staging table → Copy Activity (Upsert) → Silver Silver → Gold Silver → Dataflow Gen2 (Gold transformations) → temporary staging table → Copy Activity (Upsert) → Gold However, we do not assume that a staging table is necessarily required. If there is a simpler no-code/low-code pattern for taking the output of Dataflow Gen2 and Upserting it with Copy Activity, we would prefer that approach. I would appreciate your advice on the following questions: Is using Dataflow Gen2 for transformation and Pipeline / Copy Activity for Upsert into a Lakehouse table a recommended pattern in Fabric? We are considering using a temporary staging table between them, but is there a simpler approach? Also, are there any important limitations or considerations compared with using Notebook / Delta MERGE? When the number of changed records per day is small compared with the total table size, can we generally expect Copy Activity Upsert to consume less processing time and fewer CUs than a full Replace? Since Upsert also has costs for key matching and updating Delta files, are there any guidelines or practical thresholds for choosing between Replace and Upsert based on table size or percentage of changed records? Would you also recommend the same Upsert approach for Silver → Gold? If a Gold table is created by joining multiple Silver tables, a change to one Silver record may affect multiple Gold rows. In such cases, is it generally better to Replace the Gold table rather than identify all affected rows and Upsert them? If the goal is to prioritize no-code/low-code implementation while also considering future data growth and CU consumption, which write method and Fabric feature would you recommend for Bronze → Silver and Silver → Gold? Any best practices or real-world experience with similar Fabric architectures would be greatly appreciated. Thanks in advance!
Permissions Issues When Running Spark Notebook Within a Data Pipeline
I am configuring a data pipeline within a fabric workspace. As part of this pipeline I need to run a spark notebook in a for each loop over a given parameter list. When I try to run the pipeline, I'm getting an error saying that "The caller does not have sufficient permissions to access the requested resource". The error message then gives the workspace id in reference to the specific resource for which the caller doesn't have permissions. I'm assuming here that I am the caller in this situation. I am an admin on this workspace. I configured the notebook connection to use the workspace identity. I have ownership of the notebook. The notebook runs fine outside of the pipeline. Just in case it has an impact, technically my account is part of a group and that group is an admin on the workspace. I haven't had any permissions issues with any other action on this workspace. Thank you all for your help with this matter.
Unable to create lakehouse in Trial capacity (trial is active for 25 days)
I am working on a project to load data from ADLS gen2 blob storage into staging tables in lakehouse table, from there is will go to target table of another lakehouse table. I have couple of pipeline and notebook for this task. I am not able to create the target lakehouse because of the below reason: I do not have any pipeline/notebook running in the background. Currently I have only below workspaces and rest are deleted already. This is my trial capacity setting with Sku/size FTL4. Is it possible to increase the capacity of the trial to complete the project? I have other projects to work on as well. Please guide on how can I resolve the issue. Thank you in advance.
Fabric Planning Viewers are redirected to Semantic Model instead of the saved planning sheets
Hello, Users with the Viewer workspace role are unable to open a Microsoft Fabric Plan item correctly. Actual behavior When an affected user opens the Plan item: The saved planning sheet tabs appear briefly. The tabs disappear. Fabric opens Semantic Model Explorer inside the Plan item. Back to Sheet is displayed but disabled. Refreshing the page does not resolve the issue. The semantic model tables and fields are visible in Model Explorer, which confirms that the users can access the semantic model. However, they cannot return to or access the saved planning sheets. Expected behavior Users with the Viewer workspace role should be able to open and consume the existing planning sheets in Reading View. Planning should dynamically assign the appropriate Planning persona when users perform supported actions. Permissions and configuration verified The Plan item is located at the root of the workspace. The App Database Connection is configured. The Semantic Model Connection is configured. Users have the workspace Viewer role. Users have Read + Build permissions on the semantic model. Users have User permission on the App Database Connection. Users can see the semantic model tables and fields. The same users can successfully open another Plan item. The behavior was reproduced with an internal tenant user, so it is not exclusively related to the documented Microsoft Entra B2B limitation. Reducing the amount of data displayed by applying filters did not change the behavior. Browser-state finding The Plan loaded correctly in a clean InPrivate/Incognito browser session. Clearing the normal browser profile’s site data, cache, browser storage and service worker registration also allowed the Plan to load correctly. This suggests that stale client-side state may prevent the Planning catalog from loading. Questions Is this a known issue affecting Fabric Planning Viewers? Why does Planning fall back to Semantic Model Explorer instead of retrying or displaying an error? Thanks in advance, Best regards,
Architecture Issue : 600 tables treated in loop in Bronze
Dear all, Our architecture consists of four layers: Raw, Bronze, Silver, and Gold. Raw is composed of a single workspace, and Bronze also has its own workspace. Each workspace has its own lakehouse inside it. Tables land in Raw, coming from different sources, and are then shortcutted into Bronze. In Bronze, we have created persisted tables that mirror the Raw tables (with a few additional fields) but include historisation. These are fed incrementally, on a daily basis, based on the fields available in the Raw tables plus the last-run information stored in a table called load_audit, which records successful/failed runs and their characteristics. Both the audit table updates and the movement of data from shortcutted tables into persisted tables are handled by a single notebook. Problem: We are having issues with a specific source. This source comes from SQL Server 2019 and has more than 600 tables. Mirroring from source into Raw is working fine. The issue lies in appending data from the shortcutted tables into the Bronze persisted tables (the incremental load). Currently, this is done sequentially: the notebook loops through each table, applies the increment, and moves on to the next. This is taking a considerable amount of time — imagine, for example, 1 minute per table. There is also the problem of small files accumulating while feeding the audit load table. Simplified, it works like this: For each table: Incremental load into persisted table Insert run result into the audit load table Move to next table I discussed this with the data engineering team, and they are proposing the following: Create two separate notebooks, each with their own common utilities inside. 3.1. One will handle the large tables, and the other the small tables. 3.2. The large tables will be processed using the Spark engine (Spark workload), and the small tables using Python/Pandas. 3.3. A possible rule of thumb: if a table has fewer than 100K records, it goes to the small-table notebook; otherwise, it goes to the big-table notebook. 3.4. Partitioning by target schema and target table. 3.5. How do we address parallel execution across tables within each notebook? Splitting into two notebooks by size doesn't by itself remove the sequential loop — should we also parallelize within each notebook (e.g., concurrent threads for the Pandas notebook, or a Fabric pipeline ForEach with concurrency for the Spark notebook)? 3.6. How does this approach address the small-files problem on the audit load table? Splitting the workload doesn't inherently batch or compact writes — would we need OPTIMIZE/compaction scheduled on the audit table, or should writes be batched instead of one insert per table? 3.7. Should the 100K-record threshold be based on row count, or would data volume (MB/GB) or column count be a more reliable measure? A 100K-row table with 5 narrow columns and a 100K-row table with 200 wide/text columns have very different costs. 3.8. Where will the Pandas workload actually execute — on the driver node? If so, could running many small-table loads concurrently create memory/CPU contention with the Spark jobs running alongside it? What do you think about this approach? Could you please provide some guidance? Thanks a lot, PedroHow to efficiently maintain complex multi-table aggregations incrementally in Microsoft Fabric?
How to efficiently maintain complex multi-table aggregations incrementally in Microsoft Fabric? Comparison with Snowflake Dynamic Tables Hi everyone, I have a scenario where source data is continuously/incrementally loaded into Microsoft Fabric. We have multiple entities such as Customer, Product, Orders, Sales, Inventory, Invoices and Payments. We need to create business-ready/precomputed tables by joining multiple entities and performing aggregations. For example, a simplified transformation could be Customer + Orders + Product, followed by joins and aggregation by Customer, Product and Month. The challenge is not simply adding new records. A new incremental record can change an existing aggregation group. For example, suppose we already have Customer C001, Product P001, Month Jan and Sales of 1,000. If a new order arrives for the same Customer, Product and Month with Sales of 250, the existing aggregate needs to become 1,250. This becomes more challenging when the transformation involves multiple joins, GROUP BY, aggregations, DISTINCT counts, multiple related entities, and updates or deletes in source data. If we rebuild the complete precomputed table during every refresh, a large amount of historical data may need to be recomputed. As data volume and the number of entities increase, this could result in more data, larger transformations, more compute and longer refresh times. We are therefore trying to determine the best incremental maintenance pattern for these types of tables in Microsoft Fabric. SNOWFLAKE COMPARISON One approach we have been looking at in Snowflake is Dynamic Tables. Dynamic Tables provide a declarative way to define transformed or materialized results and allow Snowflake to manage refreshes based on changes and the configured refresh mode and target lag. The capability we are particularly interested in is maintaining the derived result incrementally instead of treating every refresh as a complete rebuild. We also understand that Snowflake Dynamic Tables do not make every transformation automatically incremental. Query shape, joins, aggregations and unsupported constructs can affect whether incremental refresh is possible. FABRIC OPTIONS WE ARE EVALUATING The first option we are evaluating is Lakehouse Materialized Lake Views (MLVs). MLVs appear to provide a similar architectural pattern where source tables are transformed into a persisted materialized result and the platform manages refresh. We are particularly interested in the optimal refresh and incremental refresh capabilities. We would like to understand how well MLVs handle transformations involving multiple joins, GROUP BY, aggregations, large historical datasets and new records that affect existing aggregation groups. For example, if Customer, Orders and Product are joined and aggregated by Customer, Product and Month, and a new order affects an existing historical Customer + Product + Month group, how does MLV incremental refresh handle this? The second option we are considering is maintaining physical precomputed or serving tables in Fabric Warehouse. The architecture would be roughly Lakehouse, Clean and Validated Data, Transformation and Aggregation, Warehouse Precomputed or Serving Table, Semantic Model and Power BI. One possibility is to use CTAS or staging-based patterns to generate the serving tables. However, if the transformation is rebuilt from the complete historical dataset, we may still end up recomputing a large amount of data during every refresh. We would therefore like to understand whether there is a recommended Fabric pattern for maintaining these physical tables incrementally, particularly when new or changed source records can affect existing aggregate groups. WHAT WE ARE TRYING TO DETERMINE We are not trying to claim that one approach is better than another. We are trying to identify the closest Fabric architecture to the incremental-maintenance capability we are familiar with from Snowflake Dynamic Tables. Can MLVs efficiently maintain complex multi-table joins and aggregations incrementally as data grows? What types of SQL transformations cause MLVs to fall back to full refresh? How does MLV incremental refresh behave when new records modify an existing aggregation group? For complex transformations, is it better to break the logic into multiple MLVs or intermediate layers? If using Fabric Warehouse physical serving tables, what is the recommended pattern for avoiding full historical recomputation? Are there Fabric-native patterns for identifying and recomputing only the affected partitions, keys or aggregation groups? For large enterprise datasets, what approach have others found most scalable and maintainable? OUR CURRENT ARCHITECTURE Our architecture is roughly Source Systems, Bronze, Clean and Validated Data, Precomputed or Serving Tables, Gold or Consumption Layer, Semantic Model and Power BI. The objective is to perform expensive joins and aggregations during data processing rather than repeatedly during interactive BI queries. The open question is how best to maintain the Precomputed or Serving layer incrementally as source data continues to grow and change. I would really appreciate input from anyone who has implemented this at scale in Microsoft Fabric, particularly comparisons between Fabric MLVs, Warehouse-based serving tables and other Fabric-native incremental processing patterns. Thanks in advance!Writing and deleting files in SharePoint shortcut
I have been trying to write a file to a path in OneLake that is a shortcut pointing to a SharePoint location but received the error message below. I have looked through Microsoft Documentation but cannot find any documentation that states that OneLake SharePoint shortcuts are Read-Only, however, neither could I find any direction on writing or deleting a file from OneLake to a SharePoint shortcut. Error: Py4JJavaError: An error occurred while calling z:notebookutils.fs.put. Operation failed: "Method Not Allowed", "This operation is not supported through shortcuts of account type OneDriveSharePoint." Could anyone direct on how to go about writing or deleting a file write through a SharePoint shortcut?
