data warehouse
1509 TopicsDataflowsStagingLakehouse Size
Hello, My DataflowsStagingLakehouse system storage is 103GB! I have many DF Gen2 with enabled staging, but I really don't understand why this lakehouse is so big! How can I explain me that this storage is so big? What are the good pratices for this kind of lakehouse ?33Views0likes6CommentsUnable to Connect Azure Synapse Database to Power BI Desktop
Hi Team, While I am trying to connect Azure Synapse DB to Power BI Desktop getting error. Steps performed 1.Obtained the Azure Synapse SQL endpoint and database credentials. 2.Opened Power BI Desktop and selected Azure Synapse Analytics as the data source. 3.Entered the Synapse SQL endpoint and database connection details. 4.Attempted to establish the connection. 5.Received an error indicating that Power BI Desktop is unable to establish a connection to the Azure Synapse database. Can some one help me how to resolve it ?17Views0likes3CommentsHow are teams structuring Microsoft Fabric Warehouse for scalable ELT and reporting workloads?
I’m looking at architecture patterns for Microsoft Fabric Warehouse in environments where multiple data sources, transformation jobs, and Power BI workloads all depend on the same platform. A typical flow might look like: Source systems → ingestion → staging → transformations → Fabric Warehouse → semantic models → Power BI I’m particularly interested in how teams are handling: staging vs curated schemas ELT transformations inside the Warehouse incremental loading patterns data quality checks before publishing CI/CD across dev, test, and production workload separation for engineering vs reporting monitoring failed loads and long-running queries deciding when to use Warehouse vs Lakehouse for the same solution For teams running Fabric Warehouse at scale, what architecture patterns have worked best for keeping the platform maintainable and avoiding duplicated transformations?7Views0likes0CommentsAuto-generate INSERT VALUES scripts for configuration data migration across environments
Hi Fabric Team, I would like to suggest a feature that could significantly simplify deployment and environment promotion activities. Today, when configuration tables contain reference or application settings data (for example mappings, parameters, business rules, or integration configurations), moving these records from a Development environment to Test or Production often requires manually creating INSERT INTO ... VALUES scripts. It would be extremely useful to have a built-in capability in Fabric that allows users to select records from a table and automatically generate the corresponding INSERT statements. I believe this capability would be particularly valuable for teams using Fabric to manage configuration-driven solutions where tables contain metadata and application settings that must remain synchronized across environments. Thank you for considering this idea.12Views1like0CommentsAllow Non-Breaking Schema Changes in Fabric CI/CD Deployments for Data Warehouse Objects
Hi, currently, Microsoft Fabric deployment pipelines can fail when a Data Warehouse deployment contains schema changes such as adding, removing, renaming, or modifying columns in existing tables. In environments where Data Warehouse tables already exist and CI/CD pipelines are used to automatically promote changes across Development, Test, and Production, this limitation becomes very important. For example, deployments can fail with errors similar to: DataLoss: The column is being dropped, data loss could occur. or ALTER COLUMN operation is not supported and deployment is blocked. As a result, even legitimate schema evolution scenarios require manual intervention, preventing fully automated deployments and increasing operational risk. Requested Improvement Provide configurable deployment options for Data Warehouse schema changes, such as: Allow schema evolution during CI/CD deployments. Support automatic handling of column additions, removals, and data type changes when explicitly approved. Introduce deployment settings to ignore or accept "data loss" warnings. Provide deployment modes similar to DACPAC publishing options available in SQL Server tooling. Offer pre-deployment validation and impact analysis without completely blocking the release. Business Impact This limitation significantly reduces the effectiveness of Fabric CI/CD pipelines for enterprise environments, where schema changes are a normal part of application and data platform lifecycle management. Without support for controlled schema evolution: Automated releases become unreliable. Manual production interventions are required. DevOps adoption is slowed down. Development, Test, and Production environments can become inconsistent. Supporting controlled schema changes would greatly improve Fabric's enterprise readiness and align deployment capabilities with modern DataOps and DevOps practices. Schema evolution should not require manual intervention in enterprise CI/CD deployment pipelines. Thank you for your support. I sincerely hope this limitation can be addressed, as it represents a significant challenge for organizations implementing automated CI/CD deployment processes with Microsoft Fabric.10Views0likes0CommentsAllow Non-Breaking Schema Changes in Fabric CI/CD Deployments for Data Warehouse Objects
Hi, currently, Microsoft Fabric deployment pipelines can fail when a Data Warehouse deployment contains schema changes such as adding, removing, renaming, or modifying columns in existing tables. In environments where Data Warehouse tables already exist and CI/CD pipelines are used to automatically promote changes across Development, Test, and Production, this limitation becomes very important. For example, deployments can fail with errors similar to: DataLoss: The column is being dropped, data loss could occur. or ALTER COLUMN operation is not supported and deployment is blocked. As a result, even legitimate schema evolution scenarios require manual intervention, preventing fully automated deployments and increasing operational risk. Requested Improvement Provide configurable deployment options for Data Warehouse schema changes, such as: Allow schema evolution during CI/CD deployments. Support automatic handling of column additions, removals, and data type changes when explicitly approved. Introduce deployment settings to ignore or accept "data loss" warnings. Provide deployment modes similar to DACPAC publishing options available in SQL Server tooling. Offer pre-deployment validation and impact analysis without completely blocking the release. Business Impact This limitation significantly reduces the effectiveness of Fabric CI/CD pipelines for enterprise environments, where schema changes are a normal part of application and data platform lifecycle management. Without support for controlled schema evolution: Automated releases become unreliable. Manual production interventions are required. DevOps adoption is slowed down. Development, Test, and Production environments can become inconsistent. Supporting controlled schema changes would greatly improve Fabric's enterprise readiness and align deployment capabilities with modern DataOps and DevOps practices. Schema evolution should not require manual intervention in enterprise CI/CD deployment pipelines!! Thank you for your support. I sincerely hope this limitation can be addressed, as it represents a significant challenge for organizations implementing automated CI/CD deployment processes with Microsoft Fabric.21Views1like0CommentsConcurrency in Fabric Warehouse: how can multiple pipelines write to the same table?
Hi all, I'm designing ingestion for a Fabric Warehouse where several pipelines could write to the same tables. Examples include an hourly incremental load running at the same time as a backfill, and a few source systems landing in a shared fact table. My understanding is that Fabric Warehouse uses snapshot isolation, and that two transactions modifying the same table at the same time can conflict, with one failing. I'd like to design around that from the start instead of finding out in production. Questions: Is conflict detection at the table level, or finer-grained? Does it behave differently for INSERT-only loads vs UPDATE, DELETE or MERGE? What patterns are people using to avoid conflicts? Options I'm considering: Serializing writes per table through pipeline dependencies or a control table Landing each source in its own staging table, then running one consolidating MERGE Retry logic in pipelines for conflict errors Do long-running read queries (for example, Power BI refreshes) block or affect writes? Any monitoring tips? For example, which DMVs or Query Insights views show conflicts or long transactions? Thanks!Solved36Views1like3CommentsAdd Automatic Query Plan Regression Detection to Fabric Data Warehouse
Fabric Data Warehouse is adding richer graphical query plans and AI-assisted performance diagnostics. A common production problem is identifying when a previously healthy query becomes expensive after a deployment, schema change, statistics change, or significant data growth. Please add automatic query plan regression detection to Fabric Data Warehouse. Fabric should maintain a performance baseline for frequently executed queries and compare new executions against previous healthy plans. The comparison should identify major changes in joins, scans, data movement, cardinality estimates, memory requirements, execution duration, and resource consumption. When a regression is detected, Warehouse Monitor should show the previous and current plans side by side and highlight the operators responsible for the change. Administrators should also be able to configure regression thresholds and receive alerts when a critical production query exceeds its normal performance range. This would help teams detect query regressions before they become user-facing performance incidents.7Views0likes0CommentsVarchar(MAX) in SQL Endpoint
I'm testing VARCHAR(MAX) support in a Fabric Lakehouse SQL Analytics Endpoint and discovered that it only worked after: 1. Enabling New Metadata Sync (Preview) 2. Creating a new Lakehouse after enabling the setting Existing Lakehouses didn't seem to pick up the capability. Is this officially expected, or is there a migration/upgrade path for existing SQL Analytics Endpoints?53Views1like6Commentsupdating .sqlproj file (version issue with git)
We recently had a couple issues syncing from main with INFORMATION SCHEMA errors. When we reviewed the documentation Upgrade Fabric Data Warehouse System File Version in a Git Integrated Fabric workspace - Microsoft Fabric | Microsoft Learn, we noticed that we had two errors. The first is that we were self referencing in a couple PROCS, eg WH.dbo.Table from current WH. We fixed that problem, and th INFORMATION SCHEMA sync still persisted, but the self ref error went away. Then we noticed that the sqlproj file is listed as a possible cause. We have the old preview version listed in git. However, if you export the warehouse project, the actual wh sqlproj file shows up to date. So there is a difference b/w git and what is actually in the workspace. We have tried to PR the change to this file, update it directly in git, make a small change in the wh and commit directly to main to see if we could trigger the change. In the one case, we went into ADO, we updated this file directly, committed to it. It actually SHOWED as an update for a hot second, BEFORE converting to a change and the change was showing the old definition. In that referenced article, we have tried a and b, and we cannot for whatever reason get this file to update in git, which then causes sync issues. Does anyone have experience with this, a recommendation? Thanks J.109Views1like7Comments