Managing data does not have to be painful. Learn how Materialized Lake Views in Microsoft Fabric combine simple SQL with automated updates to keep your data ready and your costs at lower levels.

Microsoft Fabric brings Data Engineers, Data Analysts, and Business Users onto a single platform. Data Engineers build the ingestion, Lakehouse, Warehouse, and dbt transformation layers that move raw factory data through the Bronze → Silver → Gold Medallion layers. Data Analysts design the DirectLake Semantic Model, author the DAX measure library, and build the Power BI reports that surface production readiness intelligence.

Business Users (manufacturing operations, supply chain managers, and executives) consume those insights through Power BI, the Inventory Insights data agent, and M365 Copilot, asking questions in natural language without ever opening Fabric. Inspired by the Data Factory & Data Integration Community Challenge. I built and end-to-end analytical solution on Microsoft Fabric, integrating batch-exported operational data from four U.S. factories, transforming it through the Medallion pattern, and surfacing the results through Power BI and an AI data agent. 

A configuration-driven PySpark wrapper for Microsoft Fabric Materialized Lake Views (MLVs). Enables idempotent deployments, automated state tracking via Delta properties, and event-driven pipeline orchestration.

Sensitive data is everywhere, employee records, customer files, operational exports, analytics datasets. The hard part isn't finding PII or PHI. It's de-identifying it in a way that still keeps the data useful for development, testing, analytics, and collaboration.

Everyone asks "Dataflow Gen2 or Fabric Notebook?" and gets vague answers. This article gives you a concrete decision tree, real CU cost numbers, and 4 scenario deep-dives so you can make the right call every time.

You configure row-level security on your gold Lakehouse. You test it rows filter correctly. You ship it. Two weeks later, another team creates a shortcut from their workspace and discovers they see every row. This isn't a Fabric bug. It's the consequence of conflating Power BI RLS, SQL endpoint security policies, and OneLake Security and assuming they propagate through shortcuts the same way. They don't. This article is the reference I wish I'd had.

The Gold layer is built using dbt and is version-controlled through GitHub. It is validated through automated testing and serves as a contract between the data team and the business.

A Big Boost in Productivity

Data engineering is changing fast. Earlier, setting up a Fabric Data Agent meant spending 30 to 60 minutes clicking through portals and doing repetitive manual work. With AgentForge, this has changed completely.

AgentForge brings Fabric Data Agent setup into VS Code, using natural language powered by the Model Context Protocol (MCP). What once took nearly an hour can now be done in just 2 minutes for most agents. Even complex agents that work with large repositories are ready in about 4 minutes.

 

Prompt2Data is an intelligent agent that transforms plain-English descriptions into rich, fully structured synthetic datasets. With a single command, it automatically creates a dedicated project workspace complete with an executed Jupyter notebook that includes data generation logic, visualizations, statistical insights and clean CSV outputs.

Beyond generating a single dataset, the agent can intelligently identify underlying data structures and produce multiple CSV files accordingly, enabling more realistic and scalable data modeling. It also offers a range of configurable parameters, allowing users to fine-tune dataset characteristics, control generation behavior and adapt outputs to diverse use cases.

After months of building production-grade data workflows on Microsoft Fabric, I share what genuinely works, what requires workarounds, and where the platform is heading from ingestion to transformation to serving.

Organizations often rely on SharePoint as a lightweight data exchange layer—teams upload CSVs or Excel files that are later consumed for reporting and analytics. While convenient, this pattern frequently leads to manual ingestion, inconsistent schemas, delayed refreshes, and downstream data quality issues.

To address this gap, we’re excited to opensource FabricSharePointCopy utility, a framework that provides a standardized, metadatadriven way to ingest files from SharePoint into Microsoft Fabric Lakehouse tables, with builtin validation and automation.

The project is now available on GitHub:

https://github.com/microsoft/FabricSharePointCopy 

What is FabricSharePointCopy?

FabricSharePointCopy is a utility framework for seamlessly transferring files from SharePoint into Microsoft Fabric managed tables, enabling structured, Lakehouseready data for downstream analytics and reporting.

The framework focuses on:

• Standardizing ingestion from SharePoint

• Enforcing data quality before publish

• Reducing manual intervention

• Making curated data quickly available to Fabric consumers

It is designed to be generic, reusable, and extensible, rather than tied to any single business domain.

Why We Built This

While Microsoft Fabric provides powerful analytics capabilities, filebased ingestion from SharePoint often requires custom, oneoff solutions:

• Pipelines that only run on schedules

• Manual schema fixes after ingestion 

• Silent failures when files change unexpectedly

• Inconsistent naming and table structures

FabricSharePointCopy addresses these challenges by introducing a metadatadriven ingestion layer that reacts to file changes and enforces validation before data reaches curated tables.

How the Framework Works 

At a high level, FabricSharePointCopy continuously watches configured SharePoint folders and triggers ingestion whenever a file is created or updated.

Endtoend flow:

• Detect change – A new file upload or modification is detected for a configured SharePoint folder.

• Register the file – File metadata (name, path, modified time, size) is captured to drive processing.

• Validate (DQ gate) – Metadatadriven data quality checks run before publish (schema, required columns, thresholds, sheet rules).

• Ingest & transform – CSV or Excel files are read and processed based on configured load type.

• Publish to Fabric – Curated tables are updated in the Fabric Lakehouse and made available for consumption.

• Notify on failure – If validation fails, the framework sends a notification with the failure reason.

This ensures only validated, structured data reaches downstream analytics.

Supported File Formats and Load Types 

FabricSharePointCopy supports common business file formats and ingestion patterns out of the box:

File formats

• CSV

• Excel (including multisheet files, skip rows, and skip columns)

Load types

• Full load

• Delta load

• Custom load logic

All behavior is driven through metadata rather than hardcoded logic.

BuiltIn Data Quality (DQ)

A key design principle of FabricSharePointCopy is fail fast on bad data.

Before any data is published:

• Schema checks ensure expected columns exist

• Required fields can be enforced as nonnull

• Rowcount thresholds can be applied

• Sheet selection rules are validated

When validation fails, the framework stops processing and notifies the relevant owner, preventing corrupted or incomplete data from flowing downstream.

Standardized Naming with Flexibility

To keep curated data easy to discover, the framework applies a consistent naming convention for Silverlayer tables:

Silver_{Folder}_{FileName}

This default can be overridden using metadata when needed, allowing teams to balance clarity, consistency, and customization.

Designed for Microsoft Fabric 

FabricSharePointCopy is built specifically for Microsoft Fabric Lakehouse architectures:

• Works with OneLake paths

• Produces managed tables ready for Direct Lake and downstream analytics

• Aligns with Fabric notebooks and pipelinebased orchestration

Prerequisites and setup details are documented in the GitHub README, including Fabric workspace requirements, SharePoint access, and Lakehouse shortcuts.

Open Source and Extensible

We’ve released FabricSharePointCopy as an opensource project under the MIT license, making it easy for teams to:

• Adopt the framework asis

• Extend validation logic

• Add custom transformations

• Integrate with their own notification or monitoring systems

Who Is This For? 

FabricSharePointCopy is useful for:

• Data teams ingesting operational files from SharePoint

• Analytics engineers standardizing filebased ingestion

• Fabric users looking for near realtime availability of curated data

• Teams aiming to reduce manual data fixes and rework

Contributors
@ghanchiasif, @kranthimeda, @swapnil09 

Org Apps introduce the ability to create multiple apps from one workspace, helping organizations deliver customized analytics experiences without duplicating content or creating unnecessary workspaces.

Stop copying data—start copying insights: Zero-copy access in Azure Databricks is here.

#Databricks #fabric #OneLake #Azure #DataPlatform

One of the most frequent questions I receive from DP-600 learners and data professionals is:

“How do we properly ingest and transform data in Microsoft Fabric?”

Discover how Microsoft Fabric and Dataverse can work together without copying data, giving you real-time insights and faster app development.

A production-ready pattern for ingesting data from multiple Azure Data Explorer (Kusto) databases into Microsoft Fabric Lakehouse using workspace identity, smart refresh logic, and parallel execution.

Discover Direct Lake in Microsoft Fabric, query Delta tables straight from OneLake with no data duplication while achieving low latency, high performance analytics.

Understand how snapshot isolation, incremental framing, and optimized Delta table design enable consistent, up-to-date, and scalable reporting for enterprise-grade Power BI solutions.

Fabric notebooks help teams work faster, collaborate better, and build reliable data solutions using the Lakehouse. They are simple to use but powerful enough for real-world data workloads.

The Confusion Ends Here

Working with Microsoft Fabric? Then it's only a matter of time before encountering the acronym "UDF"—and wondering what it really means. Is it a Power BI thing? Data Engineering? The answer is: it's both.

The good news: once the distinction is clear, choosing the right UDF becomes intuitive. And more importantly, understanding both reveals how Fabric's workloads are designed to work together seamlessly.

What Makes UDFs Worth Understanding

Both User Defined Functions (in Power BI) and User Data Functions (in Data Engineering) embody the same software engineering principle: modularity and the DRY principle—Don't Repeat Yourself. Yet they solve completely different problems.

Power BI's UDFs let analysts encode business logic once and reuse it across every dashboard and report. Data Engineering's UDFs enable data engineers to write transformations once and apply them wherever data needs to be processed. In both cases, the benefit is the same: one source of truth, no duplicated code, and centralized maintenance.

It's the difference between building consistent analytical metrics and processing data at scale—and why organizations need both.

Dive Deeper

Curious about how to leverage both? Ready to architect Fabric solutions that follow software engineering best practices?

What if your Power BI report could teach your AI Data Agent how to answer questions correctly?
In this article, I show how .pbip files become a knowledge base, Power BI DAX becomes ground truth, and Fabric Data Agents turn into self‑learning, production‑ready analytics assistants—with automated accuracy testing and continuous improvement.

Why maintain separate batch pipelines in Fabric? Spark Structured Streaming combined with foreachBatch lets you handle backfills and daily loads without breaking your flow. Batch meets streaming inside OneLake.

Traditional Gold tables can struggle as business logic evolves over time. Analytics lineage becomes harder to trace, governance more complex, and maintaining consistent metrics across reports increasingly challenging. Materialized Lake Views in Microsoft Fabric provide a SQL-based, reusable consumption layer that delivers Gold-level performance while remaining closely aligned with the Silver layer.

Over the last few years, I’ve had the opportunity to build data platforms from scratch using both Microsoft Fabric and Databricks—sometimes as competing options, and increasingly as complementary pieces of the same architecture.

Fabric and Databricks are not chasing the same outcomes and using one to “replace” the other is usually the wrong starting question. This post is not about feature checklists. It’s about how these platforms behave in real-world architectures, why Fabric often wins on speed and coherence, and why Databricks continues to lead when Spark depth and governance precision really matter.

The blog is to explore the Materialized Lake View Available in Microsoft Fabric ,its implementation and real time implementation challenges .Please note the feature is in preview and may not be recommended for production workloads as of the date of writing this content.

This article is based on official Microsoft Fabric documentation and practical learning resources provided by Microsoft. To move beyond theory and demonstrate real implementation, I also followed a hands-on Lakehouse lab published by Microsoft Learning. The lab walks through core concepts such as creating a lakehouse, ingesting data, and exploring it using different Fabric experiences.

If you would like to explore the same step-by-step exercise used in this article and in my demonstration, you can access the lab here:

Lab 

This article explains how Microsoft Fabric integrates with Moodle LMS REST API to create a scalable and reliable learning analytics ecosystem. We will walk through API integration, ingestion, lakehouse storage, Spark optimization, and automated pipelines: the foundation required to operationalize LMS analytics at an enterprise level.