Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

60 Days of Data Days! Live and on-demand sessions, challenges, study groups and more! And it's all FREE!. Join now. Learn more

Find articles, guides, information and community news

Most Recent
abiola_david
Most Valuable Professional
Most Valuable Professional

As data engineers, we spend a significant amount of time writing SQL queries to ingest, transform, and analyze data. However, producing the correct result is only half the story. Equally important is understanding how the SQL Server Query Optimizer executes our queries.

One of the most effective ways to inspect the optimizer's decisions is by using SHOWPLAN_ALL.

In this article, I'll demonstrate how to use SHOWPLAN_ALL in SQL Server and Microsoft Fabric SQL Database, explain what it does, discuss a common pitfall, and show you how to resolve it.

What is SHOWPLAN_ALL?

SHOWPLAN_ALL is a session-level SQL Server setting that instructs the query optimizer to return the estimated execution plan instead of executing the query.

Rather than returning data, SQL Server provides detailed information about the physical operators it intends to use, allowing us to understand how the query will be processed before it runs.

This is particularly useful when:

  • Investigating slow-running queries.

  • Understanding optimizer decisions.

  • Identifying expensive operations such as sorts and scans.

  • Comparing different query implementations.

  • Tuning SQL for better performance.

Unlike PostgreSQL, MySQL, Oracle, or Databricks SQL, which support variations of the EXPLAIN command, SQL Server relies on SHOWPLAN_ALL and graphical execution plans.

Orders Table

For this walkthrough, I'll use anorders table in Fabric SQL

 

CREATE TABLE orders
(
    order_id INT PRIMARY KEY NOT NULL,
    order_date DATE NOT NULL,
    customer VARCHAR(20) NOT NULL,
    amount INT NOT NULL
);

After populating the table with data, I'll calculate a running total using a window function.

ord.PNG

 

Sample Query

SELECT
    order_id,
    order_date,
    customer,
    amount,
    SUM(amount) OVER
    (
        ORDER BY order_date, order_id
    ) AS running_total
FROM orders
ORDER BY customer, order_date, order_id;

Without any execution plan settings enabled, SQL Server executes the query normally and returns the dataset.

ord2.PNG

 

Viewing the Estimated Execution Plan

To inspect how SQL Server intends to execute the query, enable SHOWPLAN_ALL.

SET SHOWPLAN_ALL ON;
GO

SELECT
    order_id,
    order_date,
    customer,
    amount,
    SUM(amount) OVER
    (
        ORDER BY order_date, order_id
    ) AS running_total
FROM orders
ORDER BY customer, order_date, order_id;
GO

Instead of returning rows from the orders table, SQL Server returns an estimated execution plan describing the physical operations that would be performed.

ord3.PNG

 Although the exact operators depend on the optimizer and available indexes, the execution plan typically resembles the following sequence:

  1. Read the data from the orders table.

  2. Perform any required sorting for the window function.

  3. Compute the running total using the Window Aggregate operator.

  4. Apply the final ORDER BY.

  5. Return the results.

This visibility into the optimizer's decision-making process is invaluable when diagnosing performance issues.

A Common Pitfall

One of the most common mistakes developers make is assuming that SHOWPLAN_ALL only affects the next query.

It doesn't.

SHOWPLAN_ALL is a session-level setting.

Once enabled, every subsequent query in the same session returns an execution plan instead of executing.

For example, after running:

SET SHOWPLAN_ALL ON;
GO

Even a simple query such as:

SELECT *
FROM orders;

returns the execution plan rather than the table data as seen below

ord0.PNG

 If you're unaware that SHOWPLAN_ALL is still enabled, it can be quite confusing because every query appears to "stop working."

The Solution

The fix is straightforward.

Disable the session setting.

SET SHOWPLAN_ALL OFF;
GO

After turning it off, SQL Server immediately resumes normal execution.

ord4.PNG

 

Running the same query again returns the expected dataset.

Why This Happens

Many SQL Server settings persist for the duration of the current session.

SHOWPLAN_ALL is one of them.

Other commonly used session-level settings include:

  • SHOWPLAN_XML

  • STATISTICS IO

  • STATISTICS TIME

  • NOCOUNT

Understanding session scope is important when troubleshooting unexpected SQL Server behavior, particularly during performance tuning.

 

As data volumes continue to grow, query performance becomes increasingly important.

Execution plans provide insights that cannot be obtained simply by reading the SQL statement.

They help answer questions such as:

  • Is SQL Server performing a Table Scan or an Index Seek?

  • Is an unnecessary Sort operation occurring?

  • Which operator consumes the highest estimated cost?

  • Is the optimizer using a Window Aggregate efficiently?

  • Can the query be rewritten to reduce resource consumption?

These are exactly the questions that distinguish writing SQL from engineering performant SQL solutions.

Key Takeaways

If you regularly work with SQL Server or Microsoft Fabric SQL Database, keep the following in mind:

  • SHOWPLAN_ALL returns the estimated execution plan without executing the query.

  • It is a session-level setting, not a one-time command.

  • Every query continues returning execution plans until the setting is explicitly disabled.

  • Use SET SHOWPLAN_ALL OFF to restore normal query execution.

  • Learning to interpret execution plans is an essential performance tuning skill for data engineers and database professionals.

Final Thoughts

Window functions, Common Table Expressions (CTEs), and complex analytical queries are becoming increasingly common in modern data platforms. While writing these queries correctly is important, understanding how the SQL Server Query Optimizer executes them is what enables us to build scalable and efficient data solutions.

SHOWPLAN_ALL offers a simple yet powerful way to inspect the optimizer's strategy before a query is executed. Combined with graphical execution plans and tools such as STATISTICS IO and STATISTICS TIME, it forms an essential part of every data engineer's SQL performance tuning toolkit.

The next time you're optimizing a query, don't just verify that it returns the correct result—take a few minutes to examine how SQL Server plans to execute it. The insights you gain can often reveal opportunities for significant performance improvements.

 

Murtaza_Ghafoor
Super User

Tired of creating temporary notebooks just to troubleshoot a pipeline or check a table? Meet the Lakehouse Query Explorer—Microsoft Fabric’s new lightweight Spark SQL editor built for rapid exploratory work.

Read more...

apturlov
Super User
Super User

Fabric gives us several ways to run Python, and at first they can look overlapping.
In this post, I share the practical decision model I use to choose the right option based on execution mode, compute engine, and data access path. If you are code-first and want fewer wrong turns when moving from exploration to production, this guide is for you.

Read more...

Srisakthi
Super User
Super User

Seamlessly Read & Write data to OneLake from Azure Databricks with Unity Catalog governance in place!

Read more...

svelde
Super User
Super User

Whether you are a pro-coder or a software maker, the new Fabric Apps feature offers an easy and powerful way to vibe code custom software applications and host them within Fabric.

 

Although Fabric is a very serious SaaS platform, I now have the tools to play the Zork game within Fabric! 

 

I just deployed the Fabric App Hello World template and asked GitHub Copilot to merge the Visual Zorker into the Fabric App. Then, I only had to deploy it again. All in five minutes tops.

 

Let's check out how this is done.

Read more...

Tamanchu
Super User
Super User

Most Fabric discussions still focus on Lakehouse versus Warehouse.

I believe that's increasingly the wrong question.

Thanks to Direct Lake, many organizations can now go directly from Lakehouse to Power BI without introducing a Warehouse layer. But there are important trade-offs and hidden performance considerations that every Fabric architect should understand before making that choice.

Let's dive into what really drives the decision.

 

Read more...

Pragati11
Super User
Super User

We’ve all been there. It’s Friday afternoon, and you’re looking at your Microsoft Fabric tenant. It’s cluttered with dozens of abandoned test workspaces, half-finished projects, and “oops, I forgot to delete this” environments. You open the portal. You click. You wait for the page to refresh. You click again. You feel the rage slowly building. As admins, we are supposed to be power users, but we often spend more time navigating UI menus than actually managing our data. I decided enough was enough and turned to the Microsoft Fabric CLI (fab) to take back control. But the path to automation wasn’t a straight line. 

Read more...

dimkalamaras
Microsoft Employee
Microsoft Employee

Calling private APIs from Microsoft Fabric — a reference architecture with Managed Private Endpoints and Azure Functions

Read more...

hasrikak
Microsoft Employee
Microsoft Employee

You’ve built a Fabric Data Agent - now where should users interact with it? This post compares the major orchestration paths across the native Fabric experience, Microsoft 365 Copilot, and Teams bot orchestration, with a practical lens on fit, identity, and user experience

Read more...

Murtaza_Ghafoor
Super User
Super User

Managing data does not have to be painful. Learn how Materialized Lake Views in Microsoft Fabric combine simple SQL with automated updates to keep your data ready and your costs at lower levels.

Read more...

sharvu
Microsoft Employee
Microsoft Employee

Microsoft Fabric brings Data Engineers, Data Analysts, and Business Users onto a single platform. Data Engineers build the ingestion, Lakehouse, Warehouse, and dbt transformation layers that move raw factory data through the Bronze → Silver → Gold Medallion layers. Data Analysts design the DirectLake Semantic Model, author the DAX measure library, and build the Power BI reports that surface production readiness intelligence.

 

Business Users (manufacturing operations, supply chain managers, and executives) consume those insights through Power BI, the Inventory Insights data agent, and M365 Copilot, asking questions in natural language without ever opening Fabric. Inspired by the Data Factory & Data Integration Community Challenge. I built and end-to-end analytical solution on Microsoft Fabric, integrating batch-exported operational data from four U.S. factories, transforming it through the Medallion pattern, and surfacing the results through Power BI and an AI data agent. 

Read more...

Lozovskyi
Kudo Collector
Kudo Collector

A configuration-driven PySpark wrapper for Microsoft Fabric Materialized Lake Views (MLVs). Enables idempotent deployments, automated state tracking via Delta properties, and event-driven pipeline orchestration.

Read more...

ashishprmodi
Microsoft Employee
Microsoft Employee

Sensitive data is everywhere, employee records, customer files, operational exports, analytics datasets. The hard part isn't finding PII or PHI. It's de-identifying it in a way that still keeps the data useful for development, testing, analytics, and collaboration.

Read more...

Tamanchu
Super User
Super User

Everyone asks "Dataflow Gen2 or Fabric Notebook?" and gets vague answers. This article gives you a concrete decision tree, real CU cost numbers, and 4 scenario deep-dives so you can make the right call every time.

Read more...

Tamanchu
Super User
Super User

You configure row-level security on your gold Lakehouse. You test it rows filter correctly. You ship it. Two weeks later, another team creates a shortcut from their workspace and discovers they see every row. This isn't a Fabric bug. It's the consequence of conflating Power BI RLS, SQL endpoint security policies, and OneLake Security and assuming they propagate through shortcuts the same way. They don't. This article is the reference I wish I'd had.

Read more...

techies
Super User
Super User

The Gold layer is built using dbt and is version-controlled through GitHub. It is validated through automated testing and serves as a contract between the data team and the business.

Read more...

NHariGouthami
Microsoft Employee
Microsoft Employee

A Big Boost in Productivity

Data engineering is changing fast. Earlier, setting up a Fabric Data Agent meant spending 30 to 60 minutes clicking through portals and doing repetitive manual work. With AgentForge, this has changed completely.

AgentForge brings Fabric Data Agent setup into VS Code, using natural language powered by the Model Context Protocol (MCP). What once took nearly an hour can now be done in just 2 minutes for most agents. Even complex agents that work with large repositories are ready in about 4 minutes.

This is not just a small improvement—it’s a major shift from manual clicks to AI-driven workflows that save time and reduce mistakes.
NHariGouthami_0-1775813780810.png

 

Read more...

hasrikak
Microsoft Employee
Microsoft Employee

Learn how platform engineering teams can automate the provisioning of Microsoft Fabric workspaces using Fabric CLI and Python scripts to deploy lakehouses, connections, shortcuts, sparkpools, pipelines, semantic models and many other artifacts, and compute configurations with dependency‑aware ordering. Discover how a configuration‑driven command can simplify workspace setup by reducing manual overhead and improving deployment consistency across environments.
Read more...

ashishprmodi
Microsoft Employee
Microsoft Employee

Prompt2Data is an intelligent agent that transforms plain-English descriptions into rich, fully structured synthetic datasets. With a single command, it automatically creates a dedicated project workspace complete with an executed Jupyter notebook that includes data generation logic, visualizations, statistical insights and clean CSV outputs.

 

Beyond generating a single dataset, the agent can intelligently identify underlying data structures and produce multiple CSV files accordingly, enabling more realistic and scalable data modeling. It also offers a range of configurable parameters, allowing users to fine-tune dataset characteristics, control generation behavior and adapt outputs to diverse use cases.

Read more...

Tamanchu
Super User
Super User

After months of building production-grade data workflows on Microsoft Fabric, I share what genuinely works, what requires workarounds, and where the platform is heading from ingestion to transformation to serving.

Read more...

dimkalamaras
Microsoft Employee
Microsoft Employee

This end‑to‑end playbook walks through a fully private, enterprise‑grade architecture that enables secure Databricks Mirroring into Fabric—covering VNet‑injected Databricks, ADLS Gen2 with strict network isolation, private DNS, jump‑box access, and the exact Fabric configuration required to make it all work. Designed for architects and engineers, it focuses on real‑world constraints, repeatability, and production‑ready security without compromising governance or network boundaries.
Read more...

ghanchiasif
Microsoft Employee
Microsoft Employee

Organizations often rely on SharePoint as a lightweight data exchange layer—teams upload CSVs or Excel files that are later consumed for reporting and analytics. While convenient, this pattern frequently leads to manual ingestion, inconsistent schemas, delayed refreshes, and downstream data quality issues.

To address this gap, we’re excited to opensource FabricSharePointCopy utility, a framework that provides a standardized, metadatadriven way to ingest files from SharePoint into Microsoft Fabric Lakehouse tables, with builtin validation and automation.

The project is now available on GitHub:

https://github.com/microsoft/FabricSharePointCopy 

 

ghanchiasif_0-1773042032483.png

 

What is FabricSharePointCopy?

FabricSharePointCopy is a utility framework for seamlessly transferring files from SharePoint into Microsoft Fabric managed tables, enabling structured, Lakehouseready data for downstream analytics and reporting.

The framework focuses on:

  • Standardizing ingestion from SharePoint
  • Enforcing data quality before publish
  • Reducing manual intervention
  • Making curated data quickly available to Fabric consumers

It is designed to be generic, reusable, and extensible, rather than tied to any single business domain.

 

Why We Built This

While Microsoft Fabric provides powerful analytics capabilities, filebased ingestion from SharePoint often requires custom, oneoff solutions:

  • Pipelines that only run on schedules
  • Manual schema fixes after ingestion 
  • Silent failures when files change unexpectedly
  • Inconsistent naming and table structures

FabricSharePointCopy addresses these challenges by introducing a metadatadriven ingestion layer that reacts to file changes and enforces validation before data reaches curated tables.

 

How the Framework Works 

At a high level, FabricSharePointCopy continuously watches configured SharePoint folders and triggers ingestion whenever a file is created or updated.

Endtoend flow:

  • Detect change – A new file upload or modification is detected for a configured SharePoint folder.
  • Register the file – File metadata (name, path, modified time, size) is captured to drive processing.
  • Validate (DQ gate) – Metadatadriven data quality checks run before publish (schema, required columns, thresholds, sheet rules).
  • Ingest & transform – CSV or Excel files are read and processed based on configured load type.
  • Publish to Fabric – Curated tables are updated in the Fabric Lakehouse and made available for consumption.
  • Notify on failure – If validation fails, the framework sends a notification with the failure reason.

This ensures only validated, structured data reaches downstream analytics.

 

Supported File Formats and Load Types 

FabricSharePointCopy supports common business file formats and ingestion patterns out of the box:

File formats

  • CSV
  • Excel (including multisheet files, skip rows, and skip columns)

Load types

  • Full load
  • Delta load
  • Custom load logic

All behavior is driven through metadata rather than hardcoded logic.

 

BuiltIn Data Quality (DQ)

A key design principle of FabricSharePointCopy is fail fast on bad data.

Before any data is published:

  • Schema checks ensure expected columns exist
  • Required fields can be enforced as nonnull
  • Rowcount thresholds can be applied
  • Sheet selection rules are validated

When validation fails, the framework stops processing and notifies the relevant owner, preventing corrupted or incomplete data from flowing downstream.

 

Standardized Naming with Flexibility

To keep curated data easy to discover, the framework applies a consistent naming convention for Silverlayer tables:

Silver_{Folder}_{FileName}

This default can be overridden using metadata when needed, allowing teams to balance clarity, consistency, and customization.

 

Designed for Microsoft Fabric 

FabricSharePointCopy is built specifically for Microsoft Fabric Lakehouse architectures:

  • Works with OneLake paths
  • Produces managed tables ready for Direct Lake and downstream analytics
  • Aligns with Fabric notebooks and pipelinebased orchestration

Prerequisites and setup details are documented in the GitHub README, including Fabric workspace requirements, SharePoint access, and Lakehouse shortcuts.

 

Open Source and Extensible

We’ve released FabricSharePointCopy as an opensource project under the MIT license, making it easy for teams to:

  • Adopt the framework asis
  • Extend validation logic
  • Add custom transformations
  • Integrate with their own notification or monitoring systems

 

Who Is This For? 

FabricSharePointCopy is useful for:

  • Data teams ingesting operational files from SharePoint
  • Analytics engineers standardizing filebased ingestion
  • Fabric users looking for near realtime availability of curated data
  • Teams aiming to reduce manual data fixes and rework

Contributors
@ghanchiasif@kranthimeda@swapnil09 

Murtaza_Ghafoor
Super User
Super User

Org Apps introduce the ability to create multiple apps from one workspace, helping organizations deliver customized analytics experiences without duplicating content or creating unnecessary workspaces.

Read more...

Srisakthi
Super User
Super User

Stop copying data—start copying insights: Zero-copy access in Azure Databricks is here.

Read more...

AnujPandey
Microsoft Employee
Microsoft Employee

#Databricks #fabric #OneLake #Azure #DataPlatform

Read more...

Ilgar_Zarbali
Super User
Super User

2026 - Meetup Covers.png

 

One of the most frequent questions I receive from DP-600 learners and data professionals is:

“How do we properly ingest and transform data in Microsoft Fabric?”

 

 

 

 

 

Read more...

Murtaza_Ghafoor
Super User
Super User

Discover how Microsoft Fabric and Dataverse can work together without copying data, giving you real-time insights and faster app development.

Read more...

sivak_microsoft
Microsoft Employee
Microsoft Employee

A production-ready pattern for ingesting data from multiple Azure Data Explorer (Kusto) databases into Microsoft Fabric Lakehouse using workspace identity, smart refresh logic, and parallel execution.

Read more...

pallavi_r
Super User
Super User

Discover Direct Lake in Microsoft Fabric, query Delta tables straight from OneLake with no data duplication while achieving low latency, high performance analytics.

Understand how snapshot isolation, incremental framing, and optimized Delta table design enable consistent, up-to-date, and scalable reporting for enterprise-grade Power BI solutions.

Read more...

Helpful resources

Join Blog
Interested in blogging for the community? Let us know.