Link to the notebooks to reproduce:

https://github.com/iurii-iurchenko-1/source-materials/tree/main/2026Q3_spark_runtime_2_0

Outline

• 1. What is Runtime 2.0 and why does it matter?
• 2. The five features explained
• 2.1 Vectorized CSV parsing in the Native Execution Engine
• 2.2 ANSI SQL mode inside the native engine
• 2.3 Incremental liquid clustering
• 2.4 Arrow-native Python UDFs
• 2.5 The VARIANT type in Delta tables
• 3. What are the main improvements as shown in stress testing
• 4. How did I test that?
• – Building the framework
• – The three notebooks
• – The tables and what each one measures
• 5. Main conclusions
• 5.1 If you are an architect
• 5.2 If you are an engineer
• 6. Resources used
• 7. Notebook results
• 7.1 Runtime 1.3 metrics
• 7.2 Runtime 2.0 metrics
• 8. How to reproduce
• 9. Links to documentation

What is Runtime 2.0 and why does it matter?

What is Runtime in Spark notebooks? It is an environment that includes the version of Spark, the default version of Delta tables, and other components that provide the engine to run Spark Notebooks in Fabric.

The previous version of the runtime was Runtime 1.3, which included Spark 3.5. The latest Runtime 2.0 includes Spark 4.x and an upgraded version of Delta Tables.

I chose 5 particular features introduced in Runtime 2.0 and built a framework to measure the difference in performance between versions 1.3 and 2.0.

These features are:

1. Vectorized CSV parsing in the Native Execution Engine
2. ANSI SQL mode inside the native engine
3. Incremental liquid clustering
4. Arrow-native Python UDFs
5. The VARIANT type in the delta tables

Let me explain these features in depth.

1 Vectorized CSV parsing in the Native Execution Engine

This feature means the engine-level optimization of reading CSV files. It may help if your organization has heavy jobs to ingest CSV files at scale.

2 ANSI SQL mode inside the native engine

This feature is powerful, and not a lot of people know about it. If you run Spark queries and a formula fails to calculate some values, Spark quietly returns NULL. If ANSI is enabled, the failing will be loud. It is needed in critical pipelines. Now that feature is supported by the native engine.

3 Incremental liquid clustering

This new feature influences the behavior of the OPTIMIZE command on top of a Delta table. In the old version, the behavior was to reprocess all the files. In the new version, only new unclustered files will be reprocessed. Isn't that good?

4 Arrow-native Python UDFs

This feature introduces the new optimized algorithm to process Pandas UDF functions. When are they needed? During a migration of the Python/Pandas notebooks. So, instead of rewriting Pandas functions and introducing new bugs, implementing this mechanism may help. IN the new engine, it is optimized to avoid converting to Pandas internally, which saves time and resources. Also, if you know PyArrow functions to proceed with strings, they may also be utilized instead of plain Python.

5 The VARIANT type in the delta tables

Variant type promises performance optimization on top of JSON-like data. So, it looks like you decoded the data source as JSON, wrote it once (as a VARIANT column), and it may be processed faster later, when you need to extract some information from it.

What are the Main Improvements as Shown in Stress Testing

Final Results

1. CSV Scan: I observed ~50% performance improvement on reading CSV files in Runtime 2.0 (9.2 sec vs 6.5 sec)
2. ANSI SQL mode - significant improvement in Runtime 2.0 (see the code and logs at the end of the notebook). 5.3 sec ANSI off →12 sec ANSI ON on Runtime 1.3 vs 3.6→3.7 on Runtime 2.0.
3. Incremental liquid clustering - the new algorithm introduces incremental clustering. So, now, we may have two strategies: incremental clustering or full rebuilding of Z-cubes. So, the introduces incremental clustering gives a significant boost in performance, but still loses the battle compared with full optimization/rebuilding of Z-cubes.
4. Arrow-native Python UDFs give almost a 3x boost on the new runtime due to avoiding converting it to Pandas and back
5. Variant type. Here, I observed that Variant may not give a significant boost in reading performance. Sometimes, it may have even slower performance. But it uses disk space better.

How did I test that?

To test that, I vibe-coded a framework in Claude to cover all the features mentioned above. It was a back-and-forth process. I carefully selected 5 features recently introduced to Runtime 2.0, built tech requirements for the system to test it, generated the code for three notebooks, ran it, got the first issues and output, improved it, and repeated it a few times till the process was smooth.

I tested it in my personal MS Fabric environment with F2 capacity. The solution includes three notebooks:

• NB_00_GENERATE_DATA
• NB_10_BENCH_RUNTIME_1_3
• NB_20_BENCH_RUNTIME_2_0

The tables are:

• fact_sales — 20M-row Delta fact table. Feeds the ANSI × NEE arithmetic query and all four UDF variants.
• Files/bench/raw_csv — the same 20M rows written as uncompressed CSV, 2.1 GB. The only asset that tests vectorized CSV parsing in the native engine.
• logs_json — 3M nested JSON documents held in a STRING column. The baseline for the VARIANT comparison, and the source for the VARIANT migration.
• events_lc_r13 — liquid-clustered Delta table, left with unoptimized new data on top. Measures the Delta 3.2 full-rewrite OPTIMIZE on Runtime 1.3.
• events_lc_r20_inc — identical twin of the above. Measures Delta 4.2's incremental OPTIMIZE on Runtime 2.0.
• events_lc_r20_full — identical triplet. Measures OPTIMIZE ... FULL on Runtime 2.0, giving a same-session comparison against the incremental arm.
• logs_variant — not from the generator; created by the Runtime 2.0 notebook itself via parse_json. Its creation time isthe VARIANT migration cost, and it is what the colon-operator query reads.

After that, I ran the notebooks NB_10_BENCH_RUNTIME_1_3 and NB_20_BENCH_RUNTIME_2_0 to get the main metrics. You may find these metrics below. They show how new features in Runtime 2.0 differ in performance for different use cases.

Main Conclusions

If you are an architect

The main point with Runtime 2.0 is that it becomes GA. That means the older versions will be deprecated step-by-step, and it is time to plan the modernizational upgrade of the current infrastructure to use Runtime 2.0.

If you are an engineer

1. Invest your time in learning about NEE (Native Execution Engine) features, and test how they impact your real data flow. They are efficient, especially on complex logic query
2. Arrow native Python UDFs may be 3x faster compared to ordinary UDFs if you use only the library's functionality and don't use Python variables there. This library is rich. So, it is possible, especially for text operations.
3. If you didn't know what ANSI SQL mode is, now you know. If you want to Spark fails loudly if some formula didn't take place, that's it. Now, it is much more supported by the engine.

Resources Used

I tested all that on a personal tenant with Fabric Capacity F2.

Notebooks Results

Runtime 1.3 Metrics

Runtime 2.0 Metrics

How to Reproduce?

1. Import the notebooks mentioned at the beginning of the article to your Fabric tenant
2. Create a Lakehouse "LH_RUNTIME2_0_TEST
3. Run NB_00_GENERATE_DATA
4. Sequentially, run NB_10 & NB_20, creating before runtimes 1.3 and 2.0 for them.

Links to Documentation

• https://community.fabric.microsoft.com/blog/fbc_fabricupdatesblogs/fabric-august-2026-feature-summary/5325824
• https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Fabric-July-2026-Feature-Summary/ba-p/5325823

• https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-2-0

• https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview?tabs=sparksql