warehouse

282 Topics

Switch workspace from west US to east US
Wondering how to switch WS and all contents from west US to east US. Tried backup to Git repo and restore from there - Does not work synch to a new empty workspace. Original workspace was west US, (contains pipelines, notebook, warehouse) synched WS to git repo, disconnected WS Created new WS east US tried to synch git repo to new empty WS created on east US errored out on every level and nothing got synched. It just sucks with no import export option. Any other suggestions.
AJAJ
10 hours ago Place Data Warehouse
61Views
2likes
8Comments
Schema Compare in VS Code: Simplifying Fabric Warehouse Deployments
Deploying databases shouldn't feel uncertain. Synchronizing development and production schemas in Microsoft Fabric Warehouse can be challenging as database objects evolve. A frequent issue arises when the .sqlproj file isn't configured with the Fabric-specific schema provider (SqlDbFabricDatabaseSchemaProvider). Misconfiguration may lead Schema Compare to flag supported Fabric objects as unsupported. To avoid this issue, explicitly specify the schema provider in the .sqlproj XML if the GUI does not offer that option. In this blog, let's take a quick look at how to get started. Before using Schema Compare with Fabric Warehouse, make sure you have the following prerequisites. You need access to an existing Warehouse item within a Microsoft Fabric workspace, with Contributor or higher permissions. You also need Visual Studio Code installed on your workstation. Next, install the .NET SDK, which is required to build and publish database projects. Finally, install these two Visual Studio Code extensions: SQL Database Projects SQL Server (mssql) Both extensions are available directly from the Visual Studio Code Marketplace. After installing the extensions, open Visual Studio Code and select "Add Connection" to connect to Fabric Warehouse. Enter the required server and authentication details, and complete the connection. After connecting, access the Warehouse directly in Visual Studio Code. Open Schema Compare Now navigate to Database Projects in Visual Studio Code. You should see your available database projects and connections. Right-click the database project or connection and select "Schema Compare." Schema Compare gives you an object-level view of the differences between the source and target. You can compare schemas between: .dacpac files Databases SQL database projects Comparison results outline actions to align target with source. Instead of treating the database as a single deployment unit, you can review individual changes and decide what should happen next. You can also selectively exclude actions from the comparison results when a particular change should not be deployed. Schema Compare with Fabric Warehouse Schema Compare's integration with Fabric Warehouse allows developers to identify differences in database objects before implementing changes. Database Projects and Git The database project becomes the artifact under review in the PR, not a raw DDL diff — schema changes get the same scrutiny as application code, and the deployment pipeline consumes a validated project state rather than an ad-hoc script. Schema Compare exposes Fabric Warehouse DDL limitations before deployment, helping you proactively address issues and maintain greater control over the deployment process.
techies
16 hours ago Place Data Warehouse Community Blog
8Views
0likes
0Comments
Onelake Storage Report
Hi All, I checked the OneLake storage Report for the first time, and it blew my mind... I have one warehouse that is > 1+ TB, while it only contains two (!!) tables. - 700K rows - 1.2M rows I dived deeper into it by connecting the warehouse to blob storage, and I noticed in the subfolder onelake/xxxxx/xxxxxx/Files/ ; there is OVER 800 GB OF FILES ! The stored procedures of those two are quite complex with different update statements, but this should not generate so much files ; as we are paying them as well. I already changed the time_travel_retention_cutoff_date, from 30 --> 5 days, but this has no impact on the /files, only the /tables from what I've read (after 36h still no impact there as well though) The files are kept into that folder going two months back; - Is there a way to change this setting? - Is there a way to reduce all those files that are written? - Anyone else noticing this?
JVDA98
5 days ago Place Data Warehouse
115Views
2likes
9Comments
The last mile of Fabric: getting business edits back into your warehouse
Every Fabric implementation I have worked on hits the same wall at roughly the same moment. The pipelines are running, the semantic model is clean, the reports look sharp. Then someone in finance says: "the cost centre mapping is wrong for three rows, can you fix it?" And the elegant architecture answers with a spreadsheet attached to an email. This post is about that last mile, why it is harder in Fabric than people expect, and what the realistic patterns are for solving it. Why the last mile is genuinely hard Fabric is built around analytical read paths. That is the right design for the workloads it targets, but it means the write path for small, human-scale corrections is not obvious. The options most teams land on: A staging table plus a pipeline. Someone uploads a CSV to a Lakehouse folder, a pipeline picks it up, a notebook merges it. Works, but you have built a bespoke ingestion system for what is functionally a typo fix, and now you own it forever. A Power App over the table. Good fit for structured, form-shaped entry. Poor fit for the case where a controller wants to see two hundred rows at once, sort them, and fix the eight that are wrong. Direct SQL access. Fast, but you are handing UPDATE rights to people who do not write SQL, and there is no validation layer between intent and damage. Nobody fixes it. More common than anyone admits. The mapping stays wrong and a filter gets added to the report. The spreadsheet keeps winning because the shape of the work is a spreadsheet: a grid, a lot of rows, a few edits, sorted and filtered by someone who knows the domain. The bit people get wrong: Warehouse and SQL Database are not the same target If you are building or buying anything that writes back into Fabric, this distinction matters more than any other, and it is the thing I see glossed over most often. Fabric SQL Database behaves the way a transactional developer expects. Foreign keys are enforced. You get rowversion for optimistic concurrency. A multi-statement transaction commits or rolls back as a unit. If two people edit the same row, you can detect it and refuse the second write. Fabric Warehouse does not give you those guarantees. Constraints exist for query optimisation but are not enforced. There is no rowversion equivalent for conflict detection. Isolation-level hints are accepted and ignored. A multi-step write sequence can leave you partially applied if something fails midway. Neither of these is a defect. Warehouse is an analytical engine and those trade-offs are why it scales. But it means a write-back tool that promises "all your changes commit together, or none of them do" is telling the truth against SQL Database and stretching it against Warehouse. Practical consequences if you are building this yourself: Do not rely on the database to catch bad references. Against Warehouse you must validate foreign keys in your own layer, before you write, or you will silently create orphans. Build your own conflict detection. With no rowversion, the honest fallback is comparing the primary key plus the values you read, and refusing the write if the row moved underneath you. Decide what a partial failure means, and say it out loud. If step four of six fails, steps one to three are already committed. Your user needs to know that, in the moment, in plain language. Validate before you touch the database, not after. A whole-batch gate (nothing writes unless every row passes) is far kinder than discovering row 147 is bad after 146 rows have landed. What good looks like Whatever route you take, the same handful of properties separate a write-back path that survives contact with real users from one that gets quietly abandoned: It uses the caller's identity, not a service account. If the tool connects as a shared principal, you have lost your audit trail and your permission model in one move. Entra ID passthrough means the database's own security is still doing its job, and SUSER_NAME() in an audit trigger still means something. Validation is authored, not hardcoded. Required fields, ranges, allowed values, regex patterns, uniqueness. These change constantly and should not require a deployment. Type enforcement happens before the write. Text in a numeric column, four decimals in a decimal(9,2), a date that Excel helpfully reinterpreted. Catch these in the client, where the user can still see what they typed. Errors name the row and the reason. "Constraint violation" sends the user to IT. "Row 42: Region must be one of North, South, East, West" gets fixed in ten seconds. Nothing is installed server-side. The moment your write-back solution needs stored procedures or schema changes deployed into the warehouse, it becomes a change-management conversation and the timeline triples. Where we landed We ended up building this as an Excel add-in, because that removed the training problem entirely. The user opens a workbook, picks a table from the catalogue their credentials can see, edits the grid, and clicks publish. Validation runs as a whole-batch gate before anything is written, so a failed row blocks the commit instead of half-applying it, and the failures come back as a plain-language list. Against Fabric SQL Database that commit is a single transaction. Against Fabric Warehouse we run a separate execution path and tell customers plainly what it can and cannot promise, for exactly the reasons above. Making that distinction visible turned out to be a feature, not a caveat, because the alternative is a data engineer discovering it at 2am. It is called Workbook Connect (workbookconnect.com) and there is a free tier if you want to try the pattern rather than build it. But the honest summary of this post is not "use our thing." It is: the last mile is a real architectural problem, it deserves a deliberate answer, and Warehouse and SQL Database need different answers. Curious how others are handling this. Staging tables and pipelines? Power Apps? Something cleverer? Would genuinely like to hear it. Sander Allert works at Plainsight (plainsight.pro), a Belgian Data and AI consultancy.
Solved
sanderallert
25 days ago Place Data Warehouse
67Views
1like
2Comments
Use Data Factory pipelines in Microsoft Fabric
Learn how to create a new Lakehouse and warehouse and how to insert data into those using Data Flow Gen 2 and Data pipelines. This session will help you to Ingest data into Microsoft Fabric Learning objectives Ingest data with Dataflows Gen2 in Microsoft Fabric in LakeHouse and Warehouse Ingest data with Data Pipeline in Microsoft Fabric in LakeHouse Ingest data with Data Pipeline in Microsoft Fabric in Warehouse 𝙋𝙚𝙧𝙨𝙞𝙖𝙣 𝙋𝘽𝙄𝙐𝙂- 𝙈𝙞𝙘𝙧𝙤𝙨𝙤𝙛𝙩 𝘾𝙤𝙢𝙢𝙪𝙣𝙞𝙩𝙮: https://bit.ly/3IAg7xT 𝙇𝙞𝙣𝙠𝙚𝙙𝙄𝙣: https://bit.ly/32tGkif 𝙏𝙚𝙡𝙚𝙜𝙧𝙖𝙢 𝘾𝙝𝙖𝙣𝙣𝙚𝙡: https://t.me/PersianPBIUG 𝙔𝙤𝙪𝙏𝙪𝙗𝙚: https://bit.ly/3hk20RL Language: Persian - English
VahidDM
1 month ago Place Persian Power BI User Group
154Views
0likes
0Comments
Optimizing the CSV files load in Warehouse from Lakehouse through Dataflows
Join the Indore Data & BI Community (IDBC) for another insightful online session of DATA CONNECT Series where we explore how to Optimizing the CSV files load in Warehouse from Lakehouse through Dataflows. Date: 22nd March 2025 Venue: Online Speaker: Inturi Suparna Babu (Microsoft Super User) Host: Anmol Malviya (Microsoft Super User) Don’t miss this opportunity to Learn, Collaborate & Innovate with the Microsoft Fabric Community! Join WhatsApp group for more updates: https://chat.whatsapp.com/HL6aOLHMqAGJm1JqpzhkJw ________________________________________________________________________________ Microsoft Teams meeting Join on your computer, mobile app or room device Click here to join the meeting Meeting ID: 934 137 410 019 7 Passcode: YB2N59 Download Teams | Join on the web Learn More | Meeting options ________________________________________________________________________________
anmolmalviya05
1 month ago Place Indore Data & BI Community
392Views
0likes
0Comments
Fabric Data Connect : Microsoft Fabric Deep Dive - From Basics to Brilliance
NextGen Data Aspirants Community (NDAC) Microsoft Fabric Data Connect - An Interactive Knowledge Sharing Event NDAC is excited to announce its first-ever Knowledge Sharing Event! Join us for an insightful session where Speaker Santosh J, CTO KSR Datavizon will dive deep into Microsoft Fabric, exploring its origins, evolution, and significance in today's data landscape. Date: 26th April 2025 Time: 06:00PM IST Durations: 01Hour 30Mins Key Topics to Cover in the Session: - Introduction to Microsoft Fabric – What it is and its transformative role in data analytics - Evolution of Data Platforms – Technologies before Fabric and why change was necessary - Why Microsoft Fabric? – The need for a unified and scalable data solution - Exploring Core Components – Understanding Lakehouse, Warehouse, and Notebooks - Fabric in Action – A live demonstration showcasing its capabilities - Interactive Q&A – Get your questions answered by the expert Microsoft Fabric is revolutionizing data management by integrating data engineering, real-time analytics, business intelligence, and AI-powered insights into a single unified ecosystem. This session will explore how Fabric simplifies processes, enhances collaboration, and unlocks new possibilities in data-driven decision-making. Don't miss this knowledge-packed event—ideal for data professionals, analysts, and tech enthusiasts eager to explore the future of intelligent data solutions! Explore, Learn, Innovate! Join us at WhatsApp: https://chat.whatsapp.com/GHBzZ7rKqylLyXTGJqyTxz Follow us at LinkedIn: NextGen Data Aspirants Community: Overview | LinkedIn Join us at teams: https://teams.live.com/l/community/FEAVIPJvr_9CQiYFgI Thank you! #MicrosoftFabric #DataAnalytics #TechEvent #KnowledgeSharing #DataInnovation #PowerBI #FabricConnect #CloudComputing #DataWarehouse #LakehouseArchitecture #BIAnalytics #DigitalTransformation #TechLeaders #SmartDataSolutions
suparnababu8
1 month ago Place NextGen Data Aspirants Community
1.3KViews
0likes
0Comments
Mastering the Modern Data Stack with Microsoft Fabric Workloads
NextGen Data Aspirants Community (NDAC) Microsoft Fabric Data Connect - An Interactive Knowledge Sharing Event NDAC is thrilled to invite you to its next Knowledge Sharing Event, where we’ll unravel the future of data platforms with Microsoft Fabric—Microsoft’s end-to-end analytics solution that unifies data engineering, data science, real-time intelligence, and business intelligence. Date: 10th May 2025 Time: 06:00 PM IST Duration: 1 Hour 30 Minutes Key Topics Covered in This Session: -Mastering Microsoft Fabric – Understand how Fabric redefines the modern data stack -From Silos to Unity – Evolution of data platforms and why unified analytics is essential today -Inside Microsoft Fabric – Overview of core workloads like Lakehouse, Warehouse, Data Engineering, Real-Time Intelligence, Power BI, and more -The Role of OneLake & Copilot – Explore how Fabric enables collaboration, governance, and AI-powered productivity -Live Walkthrough – A hands-on demo showcasing Microsoft Fabric in action -Audience Q&A – Ask your questions directly to our expert speaker Microsoft Fabric is transforming how organizations think about data—by simplifying architecture, enhancing scalability, and enabling AI-powered decision-making through a single platform. Whether you're a data engineer, analyst, business user, or student eager to learn about the future of analytics, this session is for you! Come learn how to modernize your data stack with confidence. Let’s Explore, Learn, and Innovate—Together! Join us at WhatsApp: https://chat.whatsapp.com/GHBzZ7rKqylLyXTGJqyTxz Follow us at LinkedIn: NextGen Data Aspirants Community: Overview | LinkedIn Join us at teams: https://teams.live.com/l/community/FEAVIPJvr_9CQiYFgI Thank you! #MicrosoftFabric #DataAnalytics #TechEvent #KnowledgeSharing #DataInnovation #PowerBI #FabricConnect #CloudComputing #DataWarehouse #LakehouseArchitecture #BIAnalytics #DigitalTransformation #TechLeaders #SmartDataSolutions
suparnababu8
1 month ago Place NextGen Data Aspirants Community
818Views
0likes
0Comments
Zürich - 72nd Fabric User Group [ONLINE]
Dear Data Wizards, We are looking forward to inviting all of you to our next meetup. Topics What's New - Kristian The Reality of Fabric Warehouses and Deploying Models using Dbt - Andy The session will be recorded and made available on YouTube --> https://aka.ms/FabricUGYouTube The Reality of Fabric Warehouses and Deploying Models using Dbt Microsoft Fabric is Microsoft's flagship Software-as-a-Service Analytics service combining Power BI, Data Engineering, Data Warehousing, Realtime Analytics, and Machine Learning. Underpinning Fabric is a variety of workloads, engines, and languages to support in building a robust Analytics solution. Dbt (data build tools) is a data transformation tool that enables data analysts and engineers to transform data in a cloud analytics warehouse. Is is feasible to build a solution on Fabric Warehouses using Dbt? Why would we want to if so? In this session we'll be diving into the integration of Fabric Warehouses and Dbt including Dbt Core and Dbt Cloud. In this session attendees will understand the use cases of Dbt with Fabric Warehouses and understand the relevant workloads, and crucially the limitations and caveats. Good to know We want this group to be a safe environment that encourages open discussion, exchange of ideas and problems you may face. Therefore, we kindly ask that no members will leverage the information for unsolicited acquisitions of new customers or projects. This group builds on trust, and without it we cannot learn from each other and excel on this topic. Want to be a presenter? We are always looking for new speaker. If you are interested and would like to show something to the Power BI Meetup Group please feel free to contact us!
FabricGuy
1 month ago Place Fabric User Group Switzerland
20KViews
0likes
3Comments
𝗛𝗮𝗻𝗱𝘀-𝗢𝗻 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗙𝗮𝗯𝗿𝗶𝗰 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽
🎉 𝗖𝗲𝗹𝗲𝗯𝗿𝗮𝘁𝗲 𝟭𝟬 𝗬𝗲𝗮𝗿𝘀 𝗼𝗳 𝗣𝗼𝘄𝗲𝗿 𝗕𝗜 𝘄𝗶𝘁𝗵 𝗮 𝗛𝗮𝗻𝗱𝘀-𝗢𝗻 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗙𝗮𝗯𝗿𝗶𝗰 𝗕𝗼𝗼𝘁𝗰𝗮𝗺𝗽! 🚀 Please register using the Google Form Only. We have to share attendees' emails in advance 👉 Registration Link: https://docs.google.com/forms/d/e/1FAIpQLSf2E3JZ1BOX3CU3SrhmvZbYP1P59w5Jq2kUJOuu2Fmdf04N6w/viewform We’re thrilled to invite you to an exclusive, in-person learning experience in Hyderabad—designed to help you re-skill, up-skill and kick-start your career with Microsoft’s leading data-analytics platform. 📅 𝗪𝗵𝗲𝗻: 𝟮𝟲 & 𝟮𝟳 𝗝𝘂𝗹𝘆 𝟮𝟬𝟮𝟱 (two full days of immersive training) 📍 Where: Hyderabad (venue details shared upon registration) 🎟️ Seats: Limited! Why Attend? Dive Deep into Microsoft Fabric: Explore OneLake, Data Factory Gen2, SQL Analytics and seamless Power BI integration. Hands-On Workshops: Build end-to-end pipelines, design semantic models, optimize Power BI reporting. Expert Insights: Learn best practices, real-world use cases and performance tips from industry veterans. Networking: Connect with fellow data professionals, MVPs and community leaders. Who Should Join? Data Analysts & BI Developers Data Engineers Power BI Enthusiasts Anyone eager to re-skill or pivot into a thriving data-analytics career 👉 Secure Your Seat: Fill out our quick registration form to help us tailor the sessions to your background: 🙏 𝗪𝗲’𝗿𝗲 𝗮𝗹𝘀𝗼 𝗹𝗼𝗼𝗸𝗶𝗻𝗴 𝗳𝗼𝗿 𝘀𝗽𝗼𝗻𝘀𝗼𝗿𝘀 who are passionate about giving back to the community. If you'd like to contribute or collaborate, please connect with Adithya Ram Parisa, Amit Chandak, or Thodupunuri Bharath for more details. This initiative is proudly organized by the India Microsoft Fabric User Group, in collaboration with the Hyderabad Data & AI Community. Let’s make this a meaningful and memorable learning experience together!
amitchandak
1 month ago Place India Microsoft Fabric User Group
218Views
0likes
0Comments