Testing Fabric Runtime 2.0 Before It Becomes the Default

Why you should test this week, not next month

Runtime 2.0 has been GA since August, but it is still opt-in. According to the Runtime 2.0 docs, Microsoft plans to make it the default for new workspaces and new Environment items in late September 2026. That is right now.

At the same time, Runtime 1.3 (Spark 3.5) is listed with an end-of-support date of 30 September 2026. It moves into Long Term Support from 1 October for six months, through March 2027 (lifecycle page). So you have a runway, but it is not long.

In my trainings, the question I get most is: "Will my notebooks break when this flips?" My honest answer: most will run fine, a few will fail loudly, and a small number will return different results without failing. That last group is the reason to test. This post walks through the exact method I use to test in isolation, without touching production.

What you need: a Fabric capacity (F2 or trial is fine), Contributor or higher on a workspace, and 30–45 minutes.

What actually changes between 1.3 and 2.0

Every core component moves a major or minor version. Versions below are from the official Runtime 1.3 and Runtime 2.0 pages.

The four areas that cause real trouble in my experience:

1. ANSI mode. In Spark 3.5, CAST('abc' AS INT) quietly returns NULL. In Spark 4.x it throws an error. The same applies to integer overflow and invalid dates. Pipelines that relied on silent nulls now fail, which is actually good, but you need to know before a 2 a.m. run does.
2. Scala 2.13 / Java 21. If your Environment has custom JARs, they must be rebuilt. There is no workaround.
3. Python library pins. A requirements.txt or Environment library list pinned for Python 3.11 is the most common failure at publish time.
4. Delta 4.x features. The docs are explicit: Delta Lake 4.x-specific features are experimental and only work in Spark experiences. If the SQL analytics endpoint, Power BI Direct Lake or the Warehouse read the same table, do not enable them. Check Delta Lake interoperability first.

The good news: Runtime 2.0 ships with the Native Execution Engine, which Microsoft benchmarks at up to 6x faster than open-source Spark on TPC-DS, with no code change.

The safe test method: a side-by-side Environment

The trick is simple. Never change the runtime at workspace level first. An Environment item pinned to 2.0 overrides the workspace default, but only for the notebooks you attach to it. Production keeps running on 1.3 while you test the same code on 2.0.

Step 1: Create a test workspace

Create a workspace such as WS-Runtime2-Test on the same capacity. If your production workspace is Git-connected, branch out to it, or simply export the 3–5 notebooks that matter most. Add a shortcut to the production lakehouse tables you read from, so you test against real data without copying it.

Step 2: Create two Environment items

Create env_rt13 and env_rt20 in the test workspace. Two environments, same libraries, only the runtime differs. That is what makes the comparison fair.

1. New item → Environment, name it env_rt20.
2. In the Runtime dropdown, select 2.0 (Spark 4.1, Delta 4.2).
3. Under Spark compute → Acceleration, turn on Native Execution Engine.
4. Copy your production library list into Public libraries (and custom JARs/wheels if you use them).
5. Save, then Publish. Publishing is where library conflicts surface, so watch this step.
6. Repeat for env_rt13 with 1.3 (Spark 3.5, Delta 3.2) and the same libraries.

Step 3: Attach and run

Open a copy of each notebook, and in the environment dropdown on the notebook ribbon, pick env_rt20. Run it. Then run the same notebook attached to env_rt13. A Spark job definition works the same way through its settings.

Optional: the early access release channel

There is a second mechanism worth knowing about for later. Release channels (preview) let you test the next set of updates within a runtime before they become default. You set it in the Environment's Spark properties:

spark.fabric.pools.skipStarterPools=true spark.computeConf.runtime.releaseChannel=earlyAccess

Early access does not use the Starter Pool, so session start is slower, and the setting is fixed for the lifetime of a session. Billing is the same. Use it after your 2.0 migration to catch library updates early.

The test notebook I use in class

Four cells. Run the whole notebook once on env_rt13 and once on env_rt20. It takes about 10 minutes, most of it session start.

Cell 1: Prove which runtime you are on

Don't trust the dropdown; print it. This also gives you the VHD image name that support will ask for if you raise a ticket.

import sys from importlib.metadata import version, PackageNotFoundError jvm = spark.sparkContext._jvm def pkg(name): try: return version(name) except PackageNotFoundError: return "not installed" info = { "spark": spark.version, "python": sys.version.split()[0], "java": jvm.System.getProperty("java.version"), "scala": jvm.scala.util.Properties.versionNumberString(), "pandas": pkg("pandas"), "native engine": spark.conf.get("spark.native.enabled", "not set"), "ansi mode": spark.conf.get("spark.sql.ansi.enabled"), "vhd image": spark.conf.get("spark.synapse.vhd.name", ""), } for k, v in info.items(): print(f"{k:<15} {v}")

Cell 2: The ANSI probe

This is the cell that makes the room go quiet in a workshop. Same SQL, different behaviour.

probes = { "bad cast": "SELECT CAST('abc' AS INT) AS v", "int overflow": "SELECT CAST(2147483647 AS INT) + 1 AS v", "bad date": "SELECT CAST('2026-02-30' AS DATE) AS v", "div by zero": "SELECT 1/0 AS v", } for name, sql in probes.items(): try: print(f"{name:<13} -> {spark.sql(sql).first()['v']}") except Exception as e: print(f"{name:<13} -> ERROR: {type(e).__name__}")

On 1.3 you get None, -2147483648, None, None. Four silent problems. On 2.0 with default settings, all four raise errors. Now search your code for CAST(, arithmetic on IDs and date parsing from strings.

Cell 3: Regression check on your real logic

Paste one production transformation, write the result from each runtime, and compare row by row.

tag = "rt20" if spark.version.startswith("4") else "rt13" result_df = spark.sql(""" -- paste one real production query here SELECT ... """) result_df.write.mode("overwrite").format("delta").saveAsTable(f"regress_{tag}")

After both runs, compare from either environment:

a = spark.table("regress_rt13") b = spark.table("regress_rt20") print("row counts :", a.count(), b.count()) print("only in 1.3 :", a.exceptAll(b).count()) print("only in 2.0 :", b.exceptAll(a).count())

Zero and zero is what you want. If you get differences on float columns, round them before comparing; tiny floating point drift is not a bug.

Cell 4: Make sure other engines can still read the table

A table written from Runtime 2.0 must stay readable by the SQL analytics endpoint and Direct Lake. Check the protocol:

(spark.sql("DESCRIBE DETAIL regress_rt20") .select("minReaderVersion", "minWriterVersion", "tableFeatures") .show(truncate=False))

Compare it with regress_rt13. They should match. Then open the lakehouse's SQL analytics endpoint and run SELECT TOP 10 * FROM regress_rt20. If that works, your downstream reports are safe.

Finally, note the duration of each run from the Monitor hub. With the Native Execution Engine on, 2.0 is often faster, but measure it on your workload rather than quoting benchmarks.

Fixes, rollout plan and checklist

These are the fixes I reach for when the test fails.

On ANSI mode: you can set spark.sql.ansi.enabled=false in the Environment to get the old behaviour back. I use it only as a temporary bridge, never as the fix. Silent nulls are exactly the data quality issue ANSI mode exists to catch.

My rollout plan

1. test workspace, two Environments, run the four cells on your top 5 notebooks.
2. fix libraries and ANSI failures; rerun until Cell 3 shows zero differences.
3. attach env_rt20 to production notebooks one pipeline at a time. Keep env_rt13 so rollback is one dropdown change.
4. once everything runs on 2.0, change the workspace default in Workspace settings → Data Engineering/Science → Spark settings → Environment → Runtime version.
5. Before March 2027: nothing left on 1.3 when LTS ends.

Checklist

• Test workspace created, production untouched
• env_rt13 and env_rt20 published with identical libraries
• Cell 1 confirms Spark 4.1 / Python 3.13 / Java 21
• ANSI probe run and code searched for risky casts
• Cell 3 regression shows zero differences
• SQL analytics endpoint reads tables written by 2.0
• Custom JARs rebuilt for Scala 2.13
• Workspace default switched last

If you do only one thing from this post, run Cell 2 on your busiest notebook today.

Sources

• Runtime 2.0 in Fabric
• Runtime 1.3 in Fabric
• Lifecycle of Apache Spark runtimes in Fabric
• Fabric runtime release channels (preview)
• Delta Lake table format interoperability
• Spark Runtime releases and updates (GitHub)