NetSuite data in Fabric: where do you put KPI logic, Spark notebooks, Warehouse or DAX?
Hi all, I’d like to ask how you’d structure KPI logic in a Fabric setup built on NetSuite data, because I keep going back and forth on where it should live. The main source is NetSuite ERP, covering finance, sales, CRM, inventory and procurement, along with a few other enterprise systems. Fabric Data Pipelines handle the scheduled transfers into OneLake, and the Lakehouse is organized in Bronze, Silver and Gold layers to separate raw, refined and curated data. Fabric Notebooks with Spark do the cleansing, transformations, business rules and KPI calculations before anything reaches reporting. A Fabric Data Warehouse then serves structured datasets for cross-functional reporting, and Power BI semantic models with DAX measures and RLS sit on top, feeding 50+ dashboards across finance, sales, operations, inventory and executive reporting. Security runs through Entra ID, access controls, lineage and encryption. The part I’m unsure about is KPI logic. Right now some of it is calculated in Spark before the data reaches the Warehouse, and the rest is defined as DAX measures in the semantic models. How do you decide what goes where? Do you push everything you can into the Gold layer and keep DAX thin, or keep the business definitions in the semantic model so they’re easier to change? I’d also like to hear how you handle NetSuite ingestion. With pipelines running on a schedule, how do you deal with incremental loads, late-arriving changes and deletes? Do you ever do full reloads for certain tables? On the Lakehouse vs. Warehouse split, what made you choose one over the other for the serving layer? Did you run into duplicated logic or extra copies of the data between them? And with 50+ reports, do you keep one semantic model per domain or a few shared ones? How do you manage RLS in that setup, especially as self-service use grows and more people start building their own reports on top? Has anyone gone through this and changed their approach later?
How are teams structuring Microsoft Fabric Warehouse for scalable ELT and reporting workloads?
I’m looking at architecture patterns for Microsoft Fabric Warehouse in environments where multiple data sources, transformation jobs, and Power BI workloads all depend on the same platform. A typical flow might look like: Source systems → ingestion → staging → transformations → Fabric Warehouse → semantic models → Power BI I’m particularly interested in how teams are handling: staging vs curated schemas ELT transformations inside the Warehouse incremental loading patterns data quality checks before publishing CI/CD across dev, test, and production workload separation for engineering vs reporting monitoring failed loads and long-running queries deciding when to use Warehouse vs Lakehouse for the same solution For teams running Fabric Warehouse at scale, what architecture patterns have worked best for keeping the platform maintainable and avoiding duplicated transformations?How are teams using AI agents to automate modern data warehouse workflows?
I am exploring how AI agents can help improve data warehouse operations by automating repetitive tasks and assisting data teams with faster decision-making. Some areas I am interested in: Automating data pipeline monitoring and issue detection AI-assisted SQL generation and optimization Automatically identifying data quality issues Triggering workflows based on business events or data changes Generating documentation and insights from warehouse metadata With platforms like Microsoft Fabric Data Warehouse, how are teams approaching AI integration? Are you using: Fabric pipelines with AI-powered automation? Copilot or LLM-based assistants for warehouse development? Custom AI agents connected with data warehouse APIs? Would love to hear real-world architecture patterns and best practices from data engineers working with Fabric.
Near-real-time CDC into Fabric Warehouse: MERGE vs Mirroring vs Lakehouse?
Hi all, I need operational data from an Azure SQL Database available in Fabric for reporting with roughly 5 to 15 minutes of latency. There are about 30 tables, mostly updates, with some deletes. Business users will query it in T-SQL and Power BI. I see three main approaches and would like to understand the trade-offs: Mirroring the source database into Fabric, then building curated tables in a Warehouse on top of it Incremental MERGE into Warehouse tables from a pipeline or copy job that reads changed rows (CDC or a watermark column) Eventstream CDC → Lakehouse Delta tables, queried through the SQL analytics endpoint Questions: Which approach is recommended at this latency and scale, and where does each one break down? How are deletes handled in each? Mirroring is the most direct, but what about MERGE-based loads? How much does running MERGE frequently cost on a Warehouse (capacity usage, small-file or fragmentation effects)? If you mirror, is it better to build curated tables in a Warehouse or use views over the mirrored database? Thanks!
Varchar(MAX) in SQL Endpoint
I'm testing VARCHAR(MAX) support in a Fabric Lakehouse SQL Analytics Endpoint and discovered that it only worked after: 1. Enabling New Metadata Sync (Preview) 2. Creating a new Lakehouse after enabling the setting Existing Lakehouses didn't seem to pick up the capability. Is this officially expected, or is there a migration/upgrade path for existing SQL Analytics Endpoints?Onelake Storage Report
Hi All, I checked the OneLake storage Report for the first time, and it blew my mind... I have one warehouse that is > 1+ TB, while it only contains two (!!) tables. - 700K rows - 1.2M rows I dived deeper into it by connecting the warehouse to blob storage, and I noticed in the subfolder onelake/xxxxx/xxxxxx/Files/ ; there is OVER 800 GB OF FILES ! The stored procedures of those two are quite complex with different update statements, but this should not generate so much files ; as we are paying them as well. I already changed the time_travel_retention_cutoff_date, from 30 --> 5 days, but this has no impact on the /files, only the /tables from what I've read (after 36h still no impact there as well though) The files are kept into that folder going two months back; - Is there a way to change this setting? - Is there a way to reduce all those files that are written? - Anyone else noticing this?
updating .sqlproj file (version issue with git)
We recently had a couple issues syncing from main with INFORMATION SCHEMA errors. When we reviewed the documentation Upgrade Fabric Data Warehouse System File Version in a Git Integrated Fabric workspace - Microsoft Fabric | Microsoft Learn, we noticed that we had two errors. The first is that we were self referencing in a couple PROCS, eg WH.dbo.Table from current WH. We fixed that problem, and th INFORMATION SCHEMA sync still persisted, but the self ref error went away. Then we noticed that the sqlproj file is listed as a possible cause. We have the old preview version listed in git. However, if you export the warehouse project, the actual wh sqlproj file shows up to date. So there is a difference b/w git and what is actually in the workspace. We have tried to PR the change to this file, update it directly in git, make a small change in the wh and commit directly to main to see if we could trigger the change. In the one case, we went into ADO, we updated this file directly, committed to it. It actually SHOWED as an update for a hot second, BEFORE converting to a change and the change was showing the old definition. In that referenced article, we have tried a and b, and we cannot for whatever reason get this file to update in git, which then causes sync issues. Does anyone have experience with this, a recommendation? Thanks J.
