From Chaos to Clarity: How the Manufacturing Dashboard Helps Everyone Understand the Business
Let's walk through what each page actually shows, and why it matters. Page 1: Customer Insights — "Who are we selling to, and are they happy?" Every business lives or dies by its customers, but it's surprisingly hard to keep track of hundreds of relationships in your head. This page acts like a customer relationship "scoreboard." It shows things like: * How many active customers the business currently has * Which customers are considered healthy (buying regularly, paying on time) versus at risk (going quiet, showing warning signs) * How revenue is spread across different customers and regions — are we relying too heavily on just a few big accounts? Think of it like a doctor's checkup, but for relationships instead of health. If ten customers suddenly go from "healthy" to "at risk," that's a signal to act before they walk away for good — not after the revenue has already dropped and everyone's asking why. Page 2: Production & Supply — "Are we making enough, and can we deliver it?" This is the page for anyone who cares about what's actually happening on the factory floor and getting product out the door. It's less about money and more about operations. Key questions it answers: * How much did we produce this month, and is that on target? * Are shipments going out on time, or are we falling behind on delivery promises? * Is one particular plant underperforming compared to the others? If you're a plant manager, this is likely the first thing you check every morning — before coffee, even. It tells you at a glance whether today is a "business as usual" day or a "we need to fix something now" day. Page 3: Overview — "How's the whole business doing, in 30 seconds?" This is the page built for someone with almost no time to spare — a CEO, an investor, or anyone who just needs the headline numbers without digging through details. At the top sit eight simple cards, each showing one important number and whether it went up or down compared to last month: * Total Revenue — how much money came in * Total Production — how much product was made * Capacity Utilization — how much of the factories' potential is actually being used * Gross Profit — how much money is left after production costs * Supply Fulfillment — how reliably orders are being delivered * Inventory Value — how much stock is sitting in the warehouse * Active Customers — how many customers are currently buying * Overall Operational Efficiency — a single score summarizing how smoothly everything is running Below that, charts break revenue and production down by month, by plant, and by region — so if a number drops, you don't just see that it dropped, you can immediately see where. Was it one plant having a bad month, or a slowdown across an entire region? There's also a simple "Alerts & Insights" section that puts the numbers into plain words — things like "Supply on track: fulfillment is 94% and improving" — so nobody has to guess what a chart is trying to tell them. Page 4: Inventory & Working Capital — "Do we have too much stock, or too little?" This page tackles a balancing act every manufacturing business faces. Keep too much inventory sitting around, and you're tying up cash and warehouse space that could be used elsewhere. Keep too little, and you risk running out of product right when a customer needs it — losing sales and trust. This page shows: * Total Inventory Value — how much money is currently tied up in stock * Inventory Turnover — how quickly that stock is being sold and replaced (a higher number generally means things are moving efficiently) * Days Inventory Outstanding — roughly how many days' worth of stock is sitting around unused * Raw Material vs. Finished Goods Stock — how much is still waiting to be turned into product versus ready to ship * Slow-Moving Inventory — stock that isn't selling and may need attention * Stockout Risk — a warning flag for items at risk of running out * Inventory Accuracy — how well the recorded stock counts match what's physically in the warehouse There's even a detailed table listing specific materials by name, plant, and status (like "Slow-Moving" or "Healthy"), so instead of a vague warning, someone gets a precise to-do list of exactly what needs a closer look. Why This Approach Works for Everyone The real magic of this dashboard isn't any single chart — it's the consistency. Every page follows the same basic structure: 1. Filters at the top (date range, region, plant, product category) so anyone can narrow the view to exactly what matters to them 2. A handful of key numbers, shown as simple cards with an up or down arrow — no complicated formulas to interpret 3. A plain-language "Alerts & Insights" section that explains, in a sentence or two, what changed and why it matters That means a machine operator, a plant manager, a customer success rep, and a CEO can all open the same dashboard and immediately find what's relevant to them — no translation needed, no waiting for someone else to "run the numbers." In a business with as many moving parts as manufacturing, that kind of shared clarity isn't a luxury. It's what keeps everyone — from the shop floor to the boardroom — pointed in the same direction.Lakehouse vs. Warehouse in Microsoft Fabric: Which One Should You Actually Use?
If you're new to Microsoft Fabric, you've probably hit this wall already: you go to create an item, and Fabric hands you two very similar-sounding options - Lakehouse and Warehouse. Both store tabular data in Delta format in OneLake and provide SQL access, but they offer different development and transactional experiences. Both let you query with SQL.Fabric presents both as analytical data-store options, and their capabilities overlap enough that the choice is not always obvious. So which one do you pick? Short answer: it depends on who's writing the queries and what shape your data is in. Long answer: keep reading — and try the quick self-check below before you scroll to the recommendation. 60-Second Self-Check Answer these three questions honestly before you create your next Fabric item: 1. Who will write the transformation logic? - A) Python/Spark notebooks, data engineers comfortable with PySpark - B) SQL analysts and BI developers who live in T-SQL 2. What does your source data look like? - A) Mixed — JSON, Parquet, CSV, streaming events, semi-structured - B) Mostly clean, structured, relational-shaped data 3. What's the end consumption pattern? - A) A mix of ML, notebooks, ad-hoc exploration, and reporting - B) Primarily Power BI reports and governed semantic models Mostly A's → Lean Lakehouse. Mostly B's → Lean Warehouse. Mixed bag? → You're not alone — see the "Can I use both?" section below. Lakehouse: The Flexible One A Lakehouse stores data as files (Delta Parquet) in OneLake, and gives you two ways to work with it: - Notebooks (PySpark, Spark SQL) for engineers who want full control - A SQL analytics endpoint that auto-generates so SQL folks can still query the same tables Pick Lakehouse when: - Your data arrives messy, semi-structured, or in large volumes that benefit from Spark's distributed processing - Your team already thinks in notebooks and data science workflows - You want schema flexibility - Delta tables support schema enforcement and controlled schema evolution, offering flexibility while preserving data reliability. - You're building a medallion architecture (bronze → silver → gold) and need engineering muscle at the bronze/silver layers Watch out for: - The SQL endpoint is read-only - The SQL analytics endpoint is read-only for Lakehouse table data, so it does not support INSERT, UPDATE, or DELETE against those Delta tables. Modify or load the data through Spark or another supported ingestion and transformation experience. You can still create supported SQL objects such as views, functions, and stored procedures in the endpoint. - Fabric provides automatic Delta table optimizations, but advanced workloads may still benefit from deliberate file sizing, data layout, partitioning, or optimization strategies. (OPTIMIZE, V-Order, partitioning) python #Typical Lakehouse bronze-to-silver pattern in a notebook df = spark.read.format("json").load("Files/raw/events/") df_clean = df.dropDuplicates().withColumn("load_date", current_date()) df_clean.write.format("delta").mode("overwrite").saveAsTable("silver_events") Python from pyspark.sql.functions import current_date() df = spark.read.format("json").load("Files/raw/events/") df_clean = df.dropDuplicates().withColumn("load_date", current_date()) df_clean.write.format("delta").mode("overwrite") saveAsTable("silver_events") Warehouse: The Familiar One Fabric Warehouse provides a rich T-SQL-first data warehousing experience. — think of it as a cloud data warehouse that happens to store data in OneLake under the hood. Pick Warehouse when: - Your team's primary skill is T-SQL, not Spark/Python - You need full DML (`INSERT`, `UPDATE`, `DELETE`, `MERGE`) with transactional guarantees - You're modeling a governed, relational structure — star schemas, stored procedures, views - You want a more traditional data-warehouse development experience (cross-database queries, You want a familiar relational warehouse development experience with T-SQL, views, stored procedures, and support for compatible SQL tools.) Watch out for: - Less flexible with wildly semi-structured or streaming-first data — you'll typically land that in a Lakehouse first, then move it in Warehouse is designed primarily for T-SQL rather than Spark-native development. If Spark is central to your transformation logic, Lakehouse is usually the more natural starting point. sql -- Typical Warehouse transformation pattern MERGE INTO dbo.DimCustomer AS target USING staging.CustomerUpdates AS source ON target.CustomerID = source.CustomerID WHEN MATCHED THEN UPDATE SET target.Email = source.Email WHEN NOT MATCHED THEN INSERT (CustomerID, Email) VALUES (source.CustomerID, source.Email); Can I Use Both? Yes - and honestly, A supported and commonly discussed architecture is to use a Lakehouse for engineering-oriented layers and a Warehouse for a curated relational serving layer. do exactly this: - Lakehouse for bronze/silver ingestion and heavy transformation (Spark does the messy work) - Warehouse for the polished gold layer that analysts and Power BI consume with familiar T-SQL Both use OneLake, and Fabric supports patterns such as shortcuts and cross-database queries that can reduce or avoid unnecessary data duplication. If you physically load curated data into separate Warehouse tables, however, that creates another stored representation. Try It Yourself Before your next project kickoff, run this checklist with your team: [ ] Who owns the transformation code - engineers or SQL analysts? [ ] Does the source data need Spark-level flexibility, or is it already relational? [ ] Do you need multi-table transactions and T-SQL DML, or can table changes be implemented through Spark and Delta operations? [ ] Could a hybrid (Lakehouse → Warehouse) actually be the real answer? No data to test with yet? A quick way to practice both patterns above is to grab a free sample dataset (or a ready-made dashboard layout to reverse-engineer) from a site like Docynx. Over to you I'd love to hear how your team decided: Did you go Lakehouse, Warehouse, or both? What tipped the decision — team skill set, data shape, or something else entirely? Drop your setup in the comments — especially if you've got a "we picked wrong and had to migrate" story, those are always the most useful ones.Dataflow Gen2 vs Copy Job vs Pipeline: Choosing by Workload, Not by Habit
Every Fabric team has a default tool. People who came from Azure Data Factory build a pipeline for everything. Power BI people open Dataflow Gen2 for everything. Newer teams might put every table into a Copy Job because the wizard is quick. Each tool does its own job very well. Problems start when it gets used for another tool's job: pipelines full of hand-built watermark logic, dataflows used only to copy tables, and Copy Jobs expected to handle transformations they were never built for. This post gives you a simple way to pick the right tool based on what the workload needs. One sentence each Copy Job moves data from A to B, including incremental loads, with as little setup as possible. Dataflow Gen2 shapes data. It cleans, merges, reshapes and applies business rules using Power Query. Pipeline coordinates work. It runs steps in order, handles dependencies and failures, and ties everything together. If you remember only one thing, remember this: Copy Job moves, Dataflow shapes, Pipeline orchestrates. Start with three questions about the workload Ask these before you open any editor. Am I moving data or changing it? If the data arrives at the destination looking mostly like the source, with only column mapping or type changes, you are moving data. If you are joining, deduplicating, deriving columns or applying business logic, you are transforming it. How does the source change? A one-time or full reload is a different problem from an ongoing incremental sync. Incremental loads based on change data capture (CDC) are different again. How many steps depend on each other? A single load on a schedule is one thing. "Load these, then validate, then transform, then run a stored procedure, then alert someone if it fails" is a workflow. Your answers usually point clearly to one tool. Copy Job: when the job is data movement Copy Job is the newest of the three and the one most often overlooked by teams stuck in old habits. Microsoft's decision guide lists its main scenarios as incremental copy and replication (both watermark-based and native CDC), data lake and storage migration, medallion ingestion, and out-of-the-box multi-table copy. Microsoft Learn The incremental support is the main reason to use it. In a pipeline, the Copy activity handles incremental copy through pipeline expressions and control tables, and only with watermarks. That means you build and maintain the control table, the lookup, the parameterized query and the watermark update yourself. Copy Job does this for you. Microsoft Learn Choose Copy Job when: You need to ingest many tables from a database into a lakehouse or warehouse. You want an initial full load followed by incremental updates, and you don't want to build watermark logic. The source supports CDC and you want inserts and updates merged into the destination automatically. The destination is outside Fabric. Copy Job supports 40+ destination connectors. Microsoft Learn Microsoft's own example fits this well: an analyst who needs multi-table selection across regional SQL Server instances, a bulk initial load, and then CDC-based incremental merges picks Copy Job because it supports both watermark-based and native CDC incremental copying through a wizard, and automatically detects CDC-enabled tables. Microsoft Learn Don't choose Copy Job when: You need real transformation. Its transformation support is rated low, and that is intentional. Microsoft Learn The load is one step in a larger workflow with conditions, retries and downstream dependencies. That belongs in a pipeline. Dataflow Gen2: when the job is shaping data Dataflow Gen2 is Power Query running at Fabric scale. It is the right tool when the value lies in the transformation logic. It offers 170+ built-in connectors, 300+ transformation functions in a visual interface, and data profiling tools for checking data quality. Microsoft Learn A common complaint is that dataflows are slow for large volumes. That used to be a fair criticism, but Fabric has added several performance features aimed at specific workloads: Fast Copy is for direct, high-throughput copies from a supported source with no transformations. It uses the same backend as the pipeline Copy activity. microsoftmicrosoft Modern Evaluator helps when you are shaping data from connectors that don't fold, or only partly fold. microsoft Partitioned Compute is for large, partitioned or multi-file datasets that can be processed in parallel. microsoft Staging lets you land raw data first and transform it afterwards (ELT), so ingestion and transformation don't compete in one pass. microsoft Choose Dataflow Gen2 when: Business logic is the main work: cleansing, standardizing codes, merging sources, calculated columns. The people who own the logic know Power Query and would struggle to maintain Spark or SQL. You are combining files, APIs, SharePoint lists and databases into one clean dataset. You want visual data profiling while you build. Don't choose Dataflow Gen2 when: You are only copying tables. Fast Copy makes this workable, but Copy Job is simpler, gives you incremental loads without extra work, and has less to maintain. Your destination isn't supported. Dataflow Gen2 lists around 7+ destination connectors, compared with 40+ for the copy tools. Microsoft Learn The transformations are very complex or code-heavy. At that point a notebook is usually the better choice. Pipeline: when the job is coordination A pipeline is not mainly a data movement tool. It is the orchestrator. Microsoft describes it as low-code orchestration that groups several activities together to complete a task. The Copy activity inside it is powerful, and it remains a strong option for very large migrations. The guide describes it as the best low-code choice for moving petabytes of data into lakehouses and warehouses, either ad hoc or on a schedule. Microsoft LearnMicrosoft Learn The real reason to use a pipeline is the control flow: dependencies, branching, retries, failure handling, parameters, and calling other items. Choose a pipeline when: Several steps must run in a set order, where step B only runs if step A succeeds. You need logic such as If/Else, ForEach over a metadata list, or waiting on an external event. You are combining different item types, such as a Copy Job, a dataflow, a notebook, a stored procedure and a web call. You need error handling and notifications around the whole process. You are doing a large, custom migration where you want detailed control over the Copy activity. Microsoft's scenario for pipelines describes a workflow that runs stored procedures, calls web APIs, moves files and executes other pipelines. That is orchestration, not simple ingestion. Microsoft Learn Don't choose a pipeline when: It would contain a single activity that just runs one dataflow on a schedule. Dataflows can be scheduled on their own. You would be rebuilding incremental loading by hand when Copy Job already supports your source. Side-by-side Copy Job Dataflow Gen2 Pipeline Main job Move and replicate Transform and shape Orchestrate Incremental loads Built in (watermark and CDC) Possible, but not its strength Manual (watermark and control tables) Transformation Low High None itself (calls other items) Destinations 40+ ~7+ Depends on activities Authoring Wizard Power Query Visual canvas and expressions Best owner Data integrator, analyst Analyst, data engineer Data engineer Warning sign you picked wrong Adding transformation workarounds Dataflow has no transformation steps Pipeline has one activity Common mistakes that come from habit The "everything is a pipeline" team. Every source table gets a Lookup, a ForEach, a parameterized Copy activity and a stored procedure to update the watermark. It works, but you now maintain a small custom framework that Copy Job gives you ready-made. Keep the pipeline and let it call the simpler pieces. The "everything is a dataflow" team. Dataflows with forty queries that just select a table and load it. Move the raw ingestion to Copy Job and keep dataflows for the layer where logic actually happens. The "Copy Job does it all" team. Trying to handle business rules with column mappings, then adding SQL views downstream to fix what should have been transformed properly. Once logic appears, add a Dataflow Gen2 or a notebook. The single-activity pipeline. A pipeline whose only purpose is to run one item on a schedule adds a layer to monitor without adding control. Use the item's own schedule until you really need dependencies. The pattern that usually wins: use all three, each for its own job For a typical medallion architecture, the tools fit together naturally: Bronze with Copy Job. Ingest source tables with built-in incremental or CDC loads. No custom watermark logic. Silver and Gold with Dataflow Gen2 (or notebooks for heavy, code-first logic). Apply cleansing, conformance and business rules where they are visible and easy to maintain. Pipeline around everything. Run ingestion, then transformation only if ingestion succeeded, then refresh or post-processing steps, and send an alert if anything fails. Each tool does what it is best at, and each is simpler because it isn't doing another tool's job. A 30-second decision checklist No transformation, and data must stay in sync over time? → Copy Job One-off or very large custom migration needing fine control? → Copy activity in a pipeline Transformation logic is the main work, and owners know Power Query? → Dataflow Gen2 Complex, code-first transformation at scale? → Notebook Multiple steps, dependencies, branching or error handling? → Pipeline, calling the tools above Closing thought The right question isn't "which tool do we use?" It is "what does this workload need?" Movement, shaping and coordination are three different problems, and Fabric gives you a dedicated tool for each. Teams that choose by workload build less custom plumbing, find problems faster, and hand solutions over more easily, because each piece does one clear job. Next time you start a new load, answer the three questions first, then pick the tool.Designing a Reusable Power BI Semantic Model for Multi-Fact Analysis
We are going to design a Power BI semantic model that serves multiple reports built on top of a Snowflake / SQL warehouse using a typical Gold layer (fact and dimension tables) for a hybrid DeFi & TradFi analytics platform. The original question came from a real analytical query that joins two fact tables (FACT_TRADE_EXECUTION and FACT_WALLET_ACCOUNT) with several conformed dimensions (DIM_PROTOCOL, DIM_INVESTOR_TIER, DIM_ASSET_CLASS, DIM_INSTRUMENT_TYPE) and later mixes ledger-statement logic with window functions (ROW_NUMBER, SUM() OVER (...)). The design decision was: one semantic model per fact table, one model per business subject area, or a single wide custom query that pre-computes everything? This article summarizes the conclusions and aligns them with current Microsoft and community guidance.Developing an Azerbaijan Shape Map
A shape map is a powerful visualization tool that allows users to represent geographical data with customized regions. However, not every country’s shape is readily available in Power BI. By developing a custom Azerbaijan shape map, we can unlock better regional insights and enhance data-driven decision-making. In this guide, I will walk you through the process of creating an Azerbaijan Shape Map, ensuring that you can effectively map and analyze location-based data. 📥 Downloadable Materials: https://onedrive.live.com/?authkey=%21AOTKe2P4w9oBvmo&id=357FB5C8090FE1B2%2168360&cid=357FB5C8090FE1B2 I am working with a small sales dataset from multiple retail stores across various cities in Azerbaijan, including Baku, Ganja, Shaki, Sumgait, and Nakhchivan. Let's add another page to the Power BI file and name it "Map." We need to ensure that the Shape Map icon is visible in the Visualization Pane. If it is not available, we can enable it by navigating to File > Options and Settings > Preview Features and ensuring that the Shape Map Visual option is selected. Click on the Shape Map icon in the Visualization Pane, then add the TotalSales(F) measure. You'll notice that the map defaults to the USA Shape Map. Select the map, then go to the Format Pane and choose Map Settings. Click on Map Type, then select Custom Map to upload our map in TopoJSON format. First, let's create our custom map. To do this, search for an Azerbaijan Shapefile using Google Chrome. Click on the Humanitarian Data Exchange (https://data.humdata.org/dataset/cod-ab-aze) link. From there, download the following files: 📥 AZE_AdminBoundaries_TabularData.xlsx 📥 aze_adm_gadm_osm_20231002_SHP.zip Open the downloaded Excel file. You'll see that it contains three worksheets, listing the names of all Azerbaijani cities, districts, and ecoregions in both Azerbaijani and English. Now, we can upload the Excel file into Power BI, perform the necessary transformations, and apply the changes to our Power BI file. Now, open a new tab in Google Chrome and go to mapshaper.org. Click on Select, then upload the downloaded shapefile ZIP to Mapshaper.org. Click on Export to proceed with saving the transformed shapefile. Leave the three selected options as they are, then choose TopoJSON as the export format. Now, return to your Power BI file to continue with the next steps. Upload the exported TopoJSON file to the Custom Map in Power BI by selecting Browse under Map Settings. Now, add ADM1_EN from the recently uploaded Excel file to the Location field in Power BI. Now, we can add a Slicer to display the cities of Azerbaijan in English. Additionally, let's add a Card visual to show the TotalSales(F) value. The final step is to create a relationship between the Cities column in the dStores table and the corresponding column in the newly added Excel file in Power BI. Conclusion: Building a custom Azerbaijan Shape Map in Power BI allows for more precise geographical visualizations, enabling better insights into regional sales and trends. By leveraging tools like Mapshaper.org, custom TopoJSON files, and Power BI's Shape Map visual, we can create interactive and dynamic maps tailored to specific datasets. Integrating Excel data, establishing relationships between tables, and using slicers for city-level analysis further enhances the usability of the report. Mastering the creation and customization of Shape Maps is a valuable skill for any data professionallooking to improve spatial analysis and drive actionable insights. Let us know if this guide was helpful—we’d love to hear your feedback!Dataflows Gen1 and Gen2: Where is my data stored?
tl;dr Gen1 stores your data as CSVs in a CDM folder in your ADLS Gen2 account. To get to it, you need to link it to your data lake storage, otherwise you won't be able access it. If Enhanced Compute Engine is on, refresh also loads a SQL cache you can use. Gen2 with no destination saves output in the semi hidden DataflowsStagingLakehouse and exposes it via DataflowsStagingWarehouse. The data is stored as Delta tables backed by Parquet files. For more details, read the whole blog.Building a Student Dropout Prediction System Using Microsoft Fabric’s Medallion Architecture
This post demonstrates how to build a synthetic, large-scale EdTech dropout prediction pipeline using Microsoft Fabric’s Data Engineering and Machine Learning workloads. It implements the Medallion Architecture (Bronze → Silver → Gold) to produce ML-ready features that predict student dropout risk.Build a Dynamic Loan Calculator in Power BI with What-If Parameters
A Loan Calculator in Power BI allows users to explore different loan scenarios by adjusting key inputs such as loan amount, interest rate, and loan term. By using What-If Parameters, users can instantly see how changes affect monthly payments and total repayment amounts, enabling dynamic decision-making and clear financial insights—all in an interactive report environment. You can find the data source file, along with Power BI files containing both solutions and those without, at the following link: Study Materials
