What’s new this month
This release introduces four optimizations to the Native Execution Engine for data engineering workloads:
- Broadcast join acceleration that reuses join data across queries.
- Window function acceleration for ranking operations such as rank and dense rank.
- Automatic materialization of reused common table expressions (CTEs), enabled by default.
The following sections explain what each optimization does and why it matters for your workloads.
Smarter broadcast joins
Broadcast joins are one of the most frequent operations in analytical queries, especially when a large fact table is joined with smaller dimension tables. Until now, the engine rebuilt the smaller side of the join—the broadcast hash table—each time it was needed, even when the same data was reused across queries in a session.
With this update, the Native Execution Engine caches the broadcast hash table once per executor and reuses it across queries. Repeated join patterns no longer pay the cost of rebuilding the same structure, which reduces CPU work and shortens execution time on join-heavy workloads. The improvement is fully automatic and requires no changes to your queries.
Faster ranking with window functions
Window functions such as rank and dense rank are common in reporting and analytics scenarios, including top-N reporting, customer segmentation, leaderboards, and event sequence analysis. Previously, when the engine encountered these functions, it could revert to a slower execution path, giving up the benefits of native, vectorized processing.
This release keeps ranking operations inside the native engine end to end, eliminating that fallback. Queries that rely heavily on ranking see substantial gains: on ranking-intensive benchmark queries, we measured improvements of up to 16 percent. This optimization is enabled by default, so ranking-heavy workloads accelerate automatically.
Automatic CTE materialization, on by default
Common table expressions make queries easier to read by letting you name a subquery and reference it multiple times. The catch is that, by default, Spark can recompute a referenced CTE every time it appears—rescanning data and repeating the same transformations for each use.
The Native Execution Engine now materializes reused CTEs automatically. When a CTE is referenced more than once, the engine computes it a single time, caches the columnar result in memory, and reuses it for every reference. This removes duplicate scans and repeated computation from your queries. On workloads that reuse CTEs, this optimization delivers some of the largest gains in this release, and it is enabled by default—no query changes and no configuration required.
How the optimizations work together
Real-world data engineering queries rarely use just one of these patterns in isolation. A single pipeline stage might join a fact table with several dimensions, apply a ranking window, scan the same Delta tables repeatedly, and reference a shared CTE more than once. Because each optimization targets a different stage of query execution, their benefits compound. A join-and-rank query that reads from cached scans and reuses a materialized CTE can gain from all four improvements at once, which is why the combined effect on benchmark suites is larger than any single optimization on its own.
Consider a nightly reporting pipeline that ranks the top products per region from a large sales table joined to product and store dimensions. With these updates, the broadcast tables for the dimensions are reused across the run, the ranking stays in the native engine, the repeated Delta scans avoid redundant validation, and any shared CTE is computed once. The same job can now finish sooner, freeing capacity for other workloads and helping teams make better use of shared Fabric resources.
The engine applies these optimizations transparently during query planning and execution. You do not need to rewrite queries, restructure tables, or tune parameters to benefit. As you adopt the latest runtime, the improvements take effect for the workloads that match each pattern, and you can observe the difference in shorter run times and lower capacity consumption.
Better price-performance, no code changes
The value of these optimizations goes beyond raw speed. Faster queries consume fewer compute resources, allowing workloads to complete using fewer Fabric capacity units. Together, these optimizations improve both performance and resource efficiency for data engineering workloads. Because the optimizations are on by default, you capture the benefit the moment your environment picks up the update.
What's changing:
- Faster pipelines: joins, ranking queries, scans, and CTE-heavy logic all complete in less time.
- Lower cost: the same workloads consume fewer capacity units, freeing capacity for other jobs.
- No code changes: existing notebooks, Spark Job Definitions, and pipelines benefit automatically.
- Consistent acceleration: the engine keeps more of your query inside the native, vectorized pipeline instead of falling back.
Get started
These enhancements continue to expand the performance benefits of the Native Execution Engine while requiring little to no effort from customers. Whether your workloads rely on joins, ranking functions, repeated scans, or reused query logic, the latest optimizations help you process more data in less time while making more efficient use of your Fabric capacity.
These optimizations are available when the Native Execution Engine is enabled, and most of them are on by default once your workspace picks up the latest runtime.
To confirm the engine is enabled for a session, add the following to your Spark configuration:
spark.conf.set("spark.native.enabled", "true")
You can also enable the engine for every session at the environment level. In your workspace, open Environment settings, select Spark compute, and turn on the Native Execution Engine. For step-by-step instructions, refr to the following instructions:
Prerequisites
- A Microsoft Fabric workspace with Spark enabled.
- A supported Fabric Spark runtime with the Native Execution Engine available.
- The Native Execution Engine enabled at the environment or session level.
Next steps
To learn how the engine accelerates your workloads and which formats and data types it supports, refer to Native execution engine for Fabric Data Engineering. We would love to hear how these optimizations perform on your workloads. Share your experience in the Fabric Community forums, and submit suggestions for future improvements on Fabric Ideas.
Microsoft Employee