Bringing Rust-native Spark compatibility to Microsoft OneLake with Sail and LakeSail

Microsoft Employee

15 hours ago

Coauthor: Shehab Amin

Microsoft OneLake gives every Microsoft Fabric tenant a single, unified data lake built on open formats. One copy of data, many engines: that is the promise, and it is what makes OneLake such a strong foundation for analytics and AI alike. This post looks at how Sail, the open-source distributed compute framework, and LakeSail Cloud, the managed platform for running Sail in production, connect natively to OneLake, and what that unlocks for teams running Spark and Python workloads on Fabric.

Sail is a drop-in replacement for Apache Spark, rebuilt from the ground up in Rust, with Rust-native support for Delta Lake and Apache Iceberg out of the box. That makes it a natural fit for OneLake, where lakehouse tables are stored in exactly these open formats. Point Sail at a Fabric lakehouse and it reads and writes the same tables the rest of Fabric uses, with no data copies, no extra connectors to install, and no changes to existing Spark code.

One engine for data and AI

Sail implements the Spark Connect protocol, the gRPC interface that decouples Spark clients from the engine that executes their plans. Existing Spark SQL and DataFrame code targeting Spark 3.5 or 4.x runs as is from any Spark Connect client, whether PySpark, Scala, Go, or Rust. The only thing that moves is the connection string. Before switching the endpoint, however, check the workload’s APIs, UDFs, and dependencies against Sail’s supported features.

from pyspark.sql import SparkSession 
 
spark = SparkSession.builder.remote("sc://sail-host:50051").getOrCreate()

Behind that familiar interface sits a very different engine. Sail is written entirely in Rust with no JVM. Data stays in Apache Arrow columnar format end to end; query execution builds on Apache DataFusion with vectorized SIMD kernels, and an async actor runtime coordinates distributed work without locks.

For teams adding AI steps to data pipelines, Python execution overhead can become a bottleneck. Apache Spark runs Python UDFs in separate worker processes, requiring data transfer across a process boundary. Sail removes that boundary entirely: It embeds the Python interpreter in the engine, allowing Arrow-native UDFs to share Arrow buffers with query execution. This avoids that inter-process transfer, Python becomes a native part of the execution plan rather than a detour out of it, which matters for AI workloads, where heavy lifting so often happens in Python.

The same engine runs in two deployment modes. On Kubernetes, it runs as a distributed cluster for production scale. On a single node, whether a laptop or one large VM, it is a great fit for local development, testing, and smaller datasets, and it works out of the box on Windows, macOS, and Linux: with no JVM to install or configure, a pip install is the whole setup. Either way, a Sail server starts in seconds, and its workers are stateless and lightweight, so a cluster can scale out quickly when load arrives and back down to zero when it passes.

LakeSail’s derived TPC-H benchmark compared Sail 0.7.0 with Apache Spark 4.2.0 using 22 queries over 100 GB of raw data stored as Parquet on the same AWS EC2 r8g.4xlarge host. Sail completed the queries in 52.81 seconds versus 534.78 seconds for Spark, with reported per-query speedups ranging from 176% to 2819%. Peak memory usage was 26 GB for Sail and 72 GB for Spark; shuffle spill to disk was 0 GB and about 115 GB, respectively. LakeSail modeled a nearly 98% compute cost reduction by combining shorter runtime with an assumed move to a smaller instance, with memory treated as the dominant sizing and pricing factor. That estimate is not a measured OneLake workload saving. Evaluate your own pipelines to establish performance and cost in your environment.

The workloads this serves

The integration is aimed at patterns that will look familiar to most data teams on Fabric:

Scheduled ETL. Nightly PySpark jobs that transform raw orders, events, or clickstream data into curated tables in a lakehouse. The code already exists; the goal is to run it faster and cheaper without a rewrite.
Interactive exploration. Analysts and engineers query production-scale tables from a notebook or IDE, where a tight iterate-and-refine loop matters more than anything.
AI enrichment. Generating embeddings over product reviews for semantic search, classifying support tickets with a language model, or running feature engineering that goes beyond built-in SQL functions. These steps live in Python, and they increasingly sit in the middle of ordinary data pipelines rather than off to the side.
Agent-driven analytics. AI agents that explore tables, run queries, and validate results on a person’s behalf. Sail ships an MCP server so agents can query lakehouse data conversationally, plus a one-shot CLI: pipe a PySpark script into sail spark run and a local server starts instantly, executes the script, and shuts down when it finishes. The Sessions capability covered below gives each agent its own scope, revocable access to real compute.

These workloads share a practical goal: use existing Spark and Python skills to transform OneLake data without maintaining a separate stored copy. Sail brings SQL and Python execution into one engine, reducing the need to move between runtimes as AI steps enter the pipeline.

Try Sail open source with OneLake

To try the OneLake example, use a test lakehouse and a Microsoft Entra identity with permission to read the source tables and write the output table. Install Sail and a compatible PySpark Spark Connect client before running the script :

pip install "pysail"

Sail ships a OneLake catalog provider that talks to OneLake’s open table APIs. It supports two modes: iceberg, which uses the Iceberg REST catalog endpoint, and delta, which uses the Unity Catalog-compatible Delta metadata endpoint. For the full list of supported authentication methods, see the OneLake catalog guide in the Sail documentation.

The example below turns order data in OneLake into daily revenue by region and writes the result back for downstream analysis. It authenticates with Microsoft Entra ID, registers the lakehouse as a catalog, and connects PySpark to an in-process Sail server. Replace the sample credentials and workspace/lakehouse.Lakehouse location with your test environment’s values. A lakehouse named sales in the analytics workspace uses analytics/sales.Lakehouse; tables in a schema-enabled lakehouse use dbo by default.

import os 
 
from pysail.spark import SparkConnectServer 
from pyspark.sql import SparkSession 
 
# Authenticate with a Microsoft Entra service principal 
os.environ["AZURE_TENANT_ID"] = "<tenant-id>" 
os.environ["AZURE_CLIENT_ID"] = "<client-id>" 
os.environ["AZURE_CLIENT_SECRET"] = "<client-secret>" 
 
# Register the lakehouse as a catalog named "fabric" 
os.environ["SAIL_CATALOG__LIST"] = ( 
    '[{type="onelake", name="fabric", url="workspace/lakehouse.Lakehouse", api="iceberg"}]' 
) 
 
# Start a Sail server in-process and connect to it 
server = SparkConnectServer() 
server.start() 
_, port = server.listening_address 
 
spark = SparkSession.builder.remote(f"sc://localhost:{port}").getOrCreate() 
 
spark.sql("SHOW TABLES IN fabric.dbo").show() 
 
orders = spark.table("fabric.dbo.orders") 
 
daily_revenue = ( 
    orders 
    .groupBy("order_date", "region") 
    .sum("amount") 
    .withColumnRenamed("sum(amount)", "revenue") 
) 
 
daily_revenue.write.format("iceberg").saveAsTable("fabric.dbo.daily_revenue")

The script writes a native Iceberg table to OneLake. Fabric workloads can consume supported Iceberg tables through OneLake’s metadata virtualization. Validate the output in the Fabric portal and in the downstream experiences you plan to use, including Power BI through Direct Lake, Fabric notebooks, and the SQL analytics endpoint. To use the Delta Lake path instead, set the catalog API to delta and use a Delta Lake-compatible writer; changing the catalog setting alone does not change the output format.

AI enrichment with Arrow Python UDFs

Python user-defined functions (UDFs) let teams add custom transformations, such as generating embeddings, to a Spark pipeline. Sail supports Python, pandas, and Arrow UDFs, including mapInArrow and applyInArrow, with support varying by API and release. Arrow-native UDFs receive Arrow arrays directly from the engine, avoiding copies at that boundary. This provides a bridge between ETL and AI without requiring a separate execution engine.

Here is an Arrow Python UDF that generates sentence embeddings over product review text stored in OneLake, using an open-source model. Each invocation receives a whole Arrow batch of strings, so the model can encode them together instead of row by row:

import pyarrow as pa 
from pyspark.sql.functions import arrow_udf 
from pyspark.sql.types import ArrayType, FloatType 
from sentence_transformers import SentenceTransformer 
 
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") 
 
Arrow_udf(returnType=ArrayType(FloatType())) 
def embed(texts: pa.Array) -> pa.Array: 
    vectors = model.encode(texts.to_pylist(), batch_size=256) 
    return pa.array([v.tolist() for v in vectors]) 
 
reviews = spark.table("fabric.dbo.product_reviews") 
 
embedded = reviews.withColumn("embedding", embed("review_text")) 
embedded.write.format("iceberg").saveAsTable("fabric.dbo.product_reviews_embedded")

The job reads product reviews from OneLake, generates embeddings in Python, and writes the results back to OneLake. Those embeddings can support a downstream semantic search application;

Secure, on-demand access with LakeSail Cloud Sessions

Sail is open source and can be self-hosted, including on Kubernetes. LakeSail Cloud provides managed Sail compute inside your own cloud account. Its Sessions capability lets people and agents connect to that compute through Spark Connect to work with OneLake data.

A Session lets a Spark Connect client reach a private production cluster through a proxy that validates a scoped, time-limited token. The cluster does not need its own public address. For this integration, Sail compute in your cloud account accesses data stored in OneLake, and query results can return to the client. Session compute is released after the configured idle timeout and close delay; token expiry controls authentication separately from the session’s lifetime.

Connecting a Session to OneLake takes three steps:

Create a catalog. In LakeSail Cloud, add a catalog of type Microsoft OneLake, provide the workspace/lakehouse.Lakehouse URL, choose the Iceberg REST or Delta Lake API, and supply credentials.
Create a session. Select a compute profile associated with the target cluster and OneLake catalog, with OneLake configured as the profile’s default catalog. Create the session, then generate a connection token from its detail page and use the full Spark Connect URL provided there.
Connect from anywhere. Point any Spark Connect client at the endpoint:

from pyspark.sql import SparkSession 
 
spark = SparkSession.builder.remote( 
    "sc://<platform-host>:8085/;token=<jwt>;x-session-id=<session-id>" 
).getOrCreate() 
 
spark.sql("SHOW TABLES in fabric.dbo").show()

That lists the Fabric lakehouse tables through the session’s private cluster. Because the OneLake catalog is the session default, unqualified names also resolve against the lakehouse, so plain SHOW TABLES IN dbo works just as well. A local notebook can explore production-scale OneLake tables interactively, with no sample copied down and no local engine that behaves differently from the real one. A scheduled pipeline or CI step can mint a short-lived token, run a UDF-heavy job like the embedding example above, and finish without any long-lived credential baked into the runner. When the token expires, the gate stops honoring it.

Sessions can give an external AI agent a Spark Connect runtime for working with OneLake data. The agent uses the session’s compute profile and attached catalogs, so access must is limited through the profile configuration and the credentials used for OneLake. The session owner issues the connection token; keep it in the agent runtime rather than in prompts or model context. Separate sessions for parallel agents make ownership and access boundaries easier to manage.

Better together

OneLake’s open-format foundation means the same copy of data can be served by many engines, each chosen for what it does best. Sail adds a Rust-native, Python-first option to that ecosystem: drop-in Spark compatibility for the code teams already have, in-process Python for the AI work they are adding, and a performance and cost profile that leaves more budget for building. Everything it writes back to OneLake is instantly discoverable, governed by your OneLake permissions, and consumable across Fabric, from Direct Lake reports to notebooks to the SQL analytics endpoint. One copy of data, many engines, and now one more way to move fast on it.