data warehouse

642 Topics

Switch workspace from west US to east US
Wondering how to switch WS and all contents from west US to east US. Tried backup to Git repo and restore from there - Does not work synch to a new empty workspace. Original workspace was west US, (contains pipelines, notebook, warehouse) synched WS to git repo, disconnected WS Created new WS east US tried to synch git repo to new empty WS created on east US errored out on every level and nothing got synched. It just sucks with no import export option. Any other suggestions.
AJAJ
15 hours ago Place Data Warehouse
55Views
2likes
7Comments
Onelake Storage Report
Hi All, I checked the OneLake storage Report for the first time, and it blew my mind... I have one warehouse that is > 1+ TB, while it only contains two (!!) tables. - 700K rows - 1.2M rows I dived deeper into it by connecting the warehouse to blob storage, and I noticed in the subfolder onelake/xxxxx/xxxxxx/Files/ ; there is OVER 800 GB OF FILES ! The stored procedures of those two are quite complex with different update statements, but this should not generate so much files ; as we are paying them as well. I already changed the time_travel_retention_cutoff_date, from 30 --> 5 days, but this has no impact on the /files, only the /tables from what I've read (after 36h still no impact there as well though) The files are kept into that folder going two months back; - Is there a way to change this setting? - Is there a way to reduce all those files that are written? - Anyone else noticing this?
JVDA98
5 days ago Place Data Warehouse
115Views
2likes
9Comments
Capacity for warehouse increased abnormal
Dear community, I am managing the capacity of Microsoft Fabric, before 11/08/2026, our system was operating normally, with item kind Warehouse wh_silver is take about 500 000 CU(s) with duration ~ 96 000 (s) per day, after 11/08/2026, the CU(s) for Warehouse increased abnormal, approximately 2.5 times higher than a previous days, despite having the same range duration(s), 1 242 000 CU(s) and duration is 89 000(s) per day I'm just using wh_silver as an example, currently, all items kind of the warehouse are affected. And the things is make our Capacity is overload. Pls let we know Is Microsoft Fabric making any update about item kind Warehouse ? And will the CU(s) for the warehouse decrease? If they keep increasing like this, our F64 capacity won't be sufficient to maintain system stability.
QuangThanh
14 days ago Place Data Warehouse
313Views
3likes
9Comments
OneLake Security support for Fabric Warehouse - planned roadmap or intentional limitation?
I'm evaluating Direct Lake on OneLake for enterprise semantic models that span multiple Fabric items and workspaces. From the current documentation, OneLake Security appears to support: Lakehouses Mirrored Databases Mirrored Catalogs However, Fabric Warehouses are not listed as supported OneLake Security items. This raises a challenge for organizations that have adopted a warehouse-centric architecture. Another challenge is security for Direct Lake semantic models. My goal is to keep semantic models in Direct Lake mode while granting semantic model developers restricted access only to specific tables within specific Warehouse. Today, this appears difficult without granting broader permissions (such as ReadAll access), and OneLake Security is not currently listed as supporting Warehouse items. Ideally, I would like a Warehouse-based security model, Direct Lake on OneLake semantic models to function without exposing the entire Warehouse. For example: Wh_Fico Wh_Sales Wh_Scm Wh_Master Today, Warehouses support SQL-based security features such as: Object Level Security (OLS) Column Level Security (CLS) Row Level Security (RLS) GRANT / DENY permissions However, Direct Lake on OneLake is positioned as the preferred option for semantic models spanning multiple Fabric items, while SQL-based security is documented as a reason to remain on Direct Lake on SQL. My questions are: Is the absence of Warehouse support in OneLake Security an intentional product design decision? Is there any roadmap to enable OneLake Security roles directly on Fabric Warehouses? If Warehouse support is not planned, what is the recommended long-term architecture for warehouse-centric customers who want: Direct Lake on OneLake Cross-workspace semantic models Centralized table/column/row security Should customers expect to introduce a Lakehouse layer purely for OneLake Security, or will OneLake Security eventually become available directly on Warehouse items? Any guidance from the product team would be appreciated.
Solved
ShubhamFabric_1
18 days ago Place Data Warehouse
247Views
1like
8Comments
Warehouse Scaling
Hi everyone, As data volumes continue to grow, I'd like to understand how organizations scale Microsoft Fabric Data Warehouse solutions. Which techniques have had the biggest impact on: Query performance Cost optimization Data modeling Concurrent workloads Long-term maintenance Any lessons learned from production deployments would be greatly appreciated.
Solved
binitafulpagare
20 days ago Place Data Warehouse
377Views
2likes
5Comments
Insert into Warehouse Table from Notebook
Trying to insert data into a warehouse table from pyspark notebook and table in warehouse has a key column with identity datatype , it gives me error regarding mismatch the schemas (apparently it expects to have identity column part of data which would be inserted). Is it possible to do it or it is part of limitation Fabric to insert directly from Notebook to Warehouse . Thanks
Solved
Hamidr
20 days ago Place Data Warehouse
79Views
1like
5Comments
The last mile of Fabric: getting business edits back into your warehouse
Every Fabric implementation I have worked on hits the same wall at roughly the same moment. The pipelines are running, the semantic model is clean, the reports look sharp. Then someone in finance says: "the cost centre mapping is wrong for three rows, can you fix it?" And the elegant architecture answers with a spreadsheet attached to an email. This post is about that last mile, why it is harder in Fabric than people expect, and what the realistic patterns are for solving it. Why the last mile is genuinely hard Fabric is built around analytical read paths. That is the right design for the workloads it targets, but it means the write path for small, human-scale corrections is not obvious. The options most teams land on: A staging table plus a pipeline. Someone uploads a CSV to a Lakehouse folder, a pipeline picks it up, a notebook merges it. Works, but you have built a bespoke ingestion system for what is functionally a typo fix, and now you own it forever. A Power App over the table. Good fit for structured, form-shaped entry. Poor fit for the case where a controller wants to see two hundred rows at once, sort them, and fix the eight that are wrong. Direct SQL access. Fast, but you are handing UPDATE rights to people who do not write SQL, and there is no validation layer between intent and damage. Nobody fixes it. More common than anyone admits. The mapping stays wrong and a filter gets added to the report. The spreadsheet keeps winning because the shape of the work is a spreadsheet: a grid, a lot of rows, a few edits, sorted and filtered by someone who knows the domain. The bit people get wrong: Warehouse and SQL Database are not the same target If you are building or buying anything that writes back into Fabric, this distinction matters more than any other, and it is the thing I see glossed over most often. Fabric SQL Database behaves the way a transactional developer expects. Foreign keys are enforced. You get rowversion for optimistic concurrency. A multi-statement transaction commits or rolls back as a unit. If two people edit the same row, you can detect it and refuse the second write. Fabric Warehouse does not give you those guarantees. Constraints exist for query optimisation but are not enforced. There is no rowversion equivalent for conflict detection. Isolation-level hints are accepted and ignored. A multi-step write sequence can leave you partially applied if something fails midway. Neither of these is a defect. Warehouse is an analytical engine and those trade-offs are why it scales. But it means a write-back tool that promises "all your changes commit together, or none of them do" is telling the truth against SQL Database and stretching it against Warehouse. Practical consequences if you are building this yourself: Do not rely on the database to catch bad references. Against Warehouse you must validate foreign keys in your own layer, before you write, or you will silently create orphans. Build your own conflict detection. With no rowversion, the honest fallback is comparing the primary key plus the values you read, and refusing the write if the row moved underneath you. Decide what a partial failure means, and say it out loud. If step four of six fails, steps one to three are already committed. Your user needs to know that, in the moment, in plain language. Validate before you touch the database, not after. A whole-batch gate (nothing writes unless every row passes) is far kinder than discovering row 147 is bad after 146 rows have landed. What good looks like Whatever route you take, the same handful of properties separate a write-back path that survives contact with real users from one that gets quietly abandoned: It uses the caller's identity, not a service account. If the tool connects as a shared principal, you have lost your audit trail and your permission model in one move. Entra ID passthrough means the database's own security is still doing its job, and SUSER_NAME() in an audit trigger still means something. Validation is authored, not hardcoded. Required fields, ranges, allowed values, regex patterns, uniqueness. These change constantly and should not require a deployment. Type enforcement happens before the write. Text in a numeric column, four decimals in a decimal(9,2), a date that Excel helpfully reinterpreted. Catch these in the client, where the user can still see what they typed. Errors name the row and the reason. "Constraint violation" sends the user to IT. "Row 42: Region must be one of North, South, East, West" gets fixed in ten seconds. Nothing is installed server-side. The moment your write-back solution needs stored procedures or schema changes deployed into the warehouse, it becomes a change-management conversation and the timeline triples. Where we landed We ended up building this as an Excel add-in, because that removed the training problem entirely. The user opens a workbook, picks a table from the catalogue their credentials can see, edits the grid, and clicks publish. Validation runs as a whole-batch gate before anything is written, so a failed row blocks the commit instead of half-applying it, and the failures come back as a plain-language list. Against Fabric SQL Database that commit is a single transaction. Against Fabric Warehouse we run a separate execution path and tell customers plainly what it can and cannot promise, for exactly the reasons above. Making that distinction visible turned out to be a feature, not a caveat, because the alternative is a data engineer discovering it at 2am. It is called Workbook Connect (workbookconnect.com) and there is a free tier if you want to try the pattern rather than build it. But the honest summary of this post is not "use our thing." It is: the last mile is a real architectural problem, it deserves a deliberate answer, and Warehouse and SQL Database need different answers. Curious how others are handling this. Staging tables and pipelines? Power Apps? Something cleverer? Would genuinely like to hear it. Sander Allert works at Plainsight (plainsight.pro), a Belgian Data and AI consultancy.
Solved
sanderallert
25 days ago Place Data Warehouse
67Views
1like
2Comments
Data Warehouse Alter Table add default value constraint
I have been trying to add a DEFAULT constraint to a COLUMN. I saw at least one 2023 post stating to check SQL surface area that, as it read says that the ALTER TABLE is not supported, I added a PRIMARY KEY CONSTRAINT using an ALTER TABLE... it just works and therefore the documentation may need some updates. I have not found any sample or discussion on the use of ALTER TABLE to specify a DEFAULT value. Since the documentation is not current there may be a way... the samples of ALTER TABLE to add a DEFAULT CONSTRAINT works well in on-prem SQL-SERVER but not in Microsoft Fabric Data Warehouse. Hope that someone have some news about this...
Solved
esobrino
27 days ago Place Data Warehouse
7.2KViews
1like
3Comments
Best approach(es) to refresh Dev Warehouse from Prod Warehouse (Read-Write Copy) in Microsoft Fabric
We are currently evaluating the best practices for refreshing a Development Data Warehouse using data from a Production Data Warehouse within the same Fabric tenant. We have evaluated a few approaches based on the use case: Read-Only Access (Zero-Copy): If Dev only needs read access to Prod data, using OneLake Shortcuts is the clear choice. It provides instantaneous access with zero data duplication, no storage overhead, and minimal setup effort. Read-Write Access (Full Schema & Data Copy): If Dev requires a Read-Write copy (allowing developers to run DML operations like UPDATE, INSERT, or DELETE without impacting Production), our current options seem to be: Fabric Data Pipelines (Copy Activity): Scheduled or trigger-based pipelines to pull data from Prod to Dev. Cross-Database Querying / CTAS: Running T-SQL (CREATE TABLE AS SELECT ...) across workspaces. The Challenge: Both Read-Write options (Pipelines and CTAS) can become cumbersome to set up and maintain, especially as datasets grow. Additionally, duplicating large volumes of data incurs extra storage and CU (Capacity Unit) processing costs. Was wondering if there are better, more efficient options available in Microsoft Fabric to maintain or periodically refresh a Read-Write copy of a Warehouse in a Dev workspace? Is Table Cloning / Zero-Copy Clone supported or planned for cross-workspace scenarios? How are others managing periodic Dev environment refreshes from Prod without creating heavy data copy pipelines?
Solved
ChiruGalla
28 days ago Place Data Warehouse
294Views
3likes
11Comments
Insert/delete/Update data using Fabric Warehouse to Lakehouse
Hi Team, I need some guidance regarding insert, update, and delete operations from a Microsoft Fabric Warehouse on tables stored in a Fabric Lakehouse. I am using an external tool through which I can successfully connect to the Fabric Warehouse and query the Lakehouse tables exposed through the Warehouse. However, when I attempt to perform an INSERT operation, I receive the following error: Data Manipulation Language (DML) statements are not supported for this table type in this version of SQL Server. Could you please help me understand whether DML operations (INSERT/UPDATE/DELETE) are supported on Lakehouse tables through the Warehouse SQL endpoint? If not, what would be the recommended approach to write data to the Lakehouse while leveraging the Warehouse? Thanks in advance for your support.
Solved
chetansolanki
1 month ago Place Data Warehouse
153Views
1like
2Comments