Improve performance for Python UDFs and complex data types in Microsoft Fabric’s Native Execution Engine

[Santhosh_Ravin1]

Where this matters: A concrete example

For example, you have a data pipeline (extract, transform, load—ETL) that ingests JSON telemetry events, each containing nested arrays of user actions. Your pipeline uses a Python UDF to apply custom business rules—say, classifying events based on logic that doesn’t map cleanly to SQL expressions.

Before NEE, this pipeline hits two performance penalties at once:

1. The Python/Scala UDF forces serialization between the JVM and a Python worker for every batch.
2. The nested arrays cause the engine to fall back from optimized columnar execution to slower row-based processing.

With NEE enabled, both operations run in the native execution path. Internal benchmarks show 2–5x faster execution, depending on UDF complexity and data nesting depth.

 

Why Python or Scala UDFs and complex types have been slow

In standard Spark execution, Python or Scala UDFs require the engine to:

1. Serialize rows from Spark’s internal format.
2. Transfer data to a separate Python worker process.
3. Execute the UDF, then serialize results back to the JVM.

Each round-trip adds CPU overhead, memory pressure, and breaks vectorized execution. Similarly, complex data types (arrays, maps, structs) can force the engine off its optimized columnar path into row-based processing—negating the benefits of native execution for the rest of your query.

As a result, teams either avoid UDFs (rewriting logic in SQL or Scala) or flatten nested schemas at ingestion time, which adds extra engineering effort.

What the Native Execution Engine does differently

NEE processes Python or Scala UDFs and complex types directly in the native columnar engine. Specifically:

• Python or Scala UDFs: The engine reduces serialization roundtrips between the JVM and Python, keeping data in columnar format longer. Vectorized UDFs see the largest gains, but standard UDFs also improve.
• Complex data types: Arrays, maps, and structs are processed natively in the columnar engine. Operations like explode, map access, and struct field extraction no longer force fallbacks to row-based execution.
• No code changes: Your existing Spark notebooks and jobs benefit automatically when NEE is enabled. No new API or syntax is required.

 

Real performance gains

Internal benchmarks on representative workloads show:

• Vectorized Python or Scala UDFs: up to 5.76× faster.
• Complex UDFs: 1.08–2.5× faster depending on UDF complexity.
• TPC-DS end-to-end workloads (including queries with complex types): up to 2.35× faster execution.

These benchmarks were run on production-scale datasets using enterprise-typical cluster configurations.

Figure: Relative execution time with and without the Native Execution Engine. Lower bars indicate faster performance. Gains are most pronounced for vectorized UDFs, with consistent improvements across complex UDF and TPC-DS workloads.

These are meaningful gains for enterprise-scale data engineering:

• Faster ETL pipelines
• Reduced cluster compute time
• Potentially lower operational costs
• Improved interactive analytics performance

Actual performance improvements will vary based on workload characteristics, data shape, and cluster configuration.

 

Native support for complex data types

Fabric’s Native Execution Engine also adds optimized support for Arrays, Maps, and Structs.

This allows pipelines to stay fully optimized without switching execution modes.

Operations involving nested data can now benefit from native optimizations while preserving the flexibility developers expect from Spark.

This is especially useful for advanced lakehouse optimization scenarios such as:

• Z-ORDER optimization
• Liquid clustering
• Semi-structured analytics
• Event-driven architectures

Instead of flattening data or restructuring pipelines for performance reasons, teams can work with complex schemas without restructuring pipelines.

 

Why this matters for the future of data engineering

Analytics workloads are evolving to include AI, real-time decision-making, semi-structured data, and the growing importance of Python and developer-centric workflows. As workloads evolve beyond traditional BI, performance engines can no longer focus only on accelerating simple SQL queries.

Platforms now need to support complex business logic, data science pipelines, deeply nested data formats, and hybrid workflows that combine ETL, machine learning, and interactive analytics in a single environment.

Fabric’s Native Execution Engine reflects this shift. It’s designed not just for classic SQL acceleration, but for the next generation of analytical workloads that demand flexibility, scale, and high-performance execution across diverse programming models. You no longer need to trade off flexibility for performance.

 

Get started

The Native Execution Engine in Microsoft Fabric removes long-standing Spark bottlenecks by bringing Python UDFs and complex data types directly into the optimized native execution path. This helps improve performance and reduce execution overhead.

To get started:

1. Enable the Native Execution Engine — Navigate to Workspace Settings > Data Engineering/Science > Spark Settings > Environment. Change the default environment to one that has Native Execution Engine enabled. You can enable NEE in the Compute section of any Environment item.
2. Run your workload — Execute an existing notebook or Spark job that uses Python or Scala UDFs or complex types. Compare execution time against a run without Native Execution Engine.
3. Verify native execution — Check inline alerts to confirm there are no fallback alerts, indicating operations run on the native engine.

 

Learn more

• Native Execution Engine overview
• Enable Native Execution Engine in your Fabric workspace

We’d love to hear how the Native Execution Engine accelerates your Spark workloads. Try it today and share your feedback in the Fabric Community forum.