User Profile
Santhosh_Ravin1
Microsoft Employee
Joined 1 year ago
User Widgets
Contributions
Job concurrency and queue monitoring for Spark, now with a capacity-level view (Generally Available)
Spark jobs don't always start immediately. When jobs are queued or throttled, it can be difficult to determine if the cause is workspace limits, capacity saturation, or contention from other workspaces sharing the same capacity. Job concurrency and queue monitoring for data engineering are now generally available, and it includes a new capacity-level view that shows exactly how each workspace consumes your capacity over time.Customer-managed key encryption now extends to Spark jobs in Microsoft Fabric
Security and compliance teams increasingly expect full control over the encryption keys that protect their data—not only while it sits in storage, but while it is actively being processed. In Microsoft Fabric, Spark powers some of the most data-intensive workloads, from large-scale data engineering pipelines to data science and machine learning. As these jobs run, they read, write, and temporarily stage data on the compute cluster. Customer-managed key support has been extended to that processing layer for Spark jobs.Native Execution Engine now accelerates CSV workloads in Microsoft Fabric Spark
Accelerate CSV Workloads with Native Execution Engine in Microsoft Fabric Spark Comma-separated values (CSV) remain one of the most widely used file formats in data engineering. Whether ingesting log files, processing exports from legacy systems, or loading flat-file datasets, CSV workloads are a daily reality for Fabric Spark users. The Native Execution Engine in Microsoft Fabric Spark now natively accelerates CSV file reads, delivering up to 2× faster performance on benchmark workloads with no code changes required. New Native CSV Reader in Execution Engine Previously, when the Native Execution Engine processed CSV files, reads fell back to Spark's default CSV reader. As a result, CSV workloads didn't benefit from the vectorized columnar execution that accelerates Parquet and Delta workloads. The latest Native Execution Engine update includes a purpose-built CSV reader powered by the Velox engine with SIMD-optimized parsing. CSV file reads now execute directly within the vectorized pipeline, eliminating row-to-columnar conversion overhead and delivering significant performance improvements. What Performance Can You Expect? Benchmark Without NEE (Baseline) With NEE 20 GB CSV read/write workload 56 seconds 34–38 seconds (~1.5–1.65× faster) TPC-DS benchmark suite (CSV) Baseline Up to 2× faster TPC-H benchmark (SIMD optimized) Baseline ~35% improvement TPC-DS end-to-end Baseline ~20% improvement Note: Performance gains vary depending on workload characteristics, schema complexity, and data distribution. CSV workloads with larger datasets and simpler schemas typically see the largest improvements. Impact for Data Engineering Workflows Faster ETL Pipelines - CSV ingestion stages that previously bottlenecked pipelines can now complete more quickly, accelerating end-to-end data processing. Lower Compute Costs - Faster execution means workloads consume Fabric capacity for less time, potentially reducing compute costs. No Code Changes Required - If Native Execution Engine is enabled, CSV acceleration happens automatically. Existing notebooks, pipelines, and Spark SQL queries benefit immediately. Consistent Acceleration Across Formats - CSV joins involving Parquet, Delta, and other natively accelerated formats benefit from a unified performance model. How It Works Velox CSV Parser with SIMD Optimizations The parser leverages CPU-level vector instructions to process multiple bytes simultaneously, accelerating: Field parsing Delimiter detection Type conversion Mison-Based Structural Indexing For compatible workloads, the engine uses structural indexing to identify field boundaries in a single pass before parsing. This reduces CPU utilization and improves performance on large files. Together, these optimizations keep CSV data flowing through the vectorized columnar pipeline without falling back to Spark's row-based reader, preserving the performance characteristics of Native Execution Engine. Supported CSV Options The native CSV reader supports most commonly used Spark CSV options, including: Custom delimiters Custom quote characters Header inference Explicit schema specification Null value handling Multi-line records Encoding configuration Escape character configuration For a complete list of supported options and known limitations, see the Native Execution Engine documentation. Getting Started CSV acceleration is available automatically whenever Native Execution Engine is enabled. No additional configuration is required. spark.conf.set("spark.native.enabled", "true") You can also enable Native Execution Engine at the environment level for all Spark sessions. For setup instructions, see Enable Native Execution Engine. Prerequisites Microsoft Fabric workspace with Spark enabled Fabric Runtime 3.5 or later Native Execution Engine enabled at the environment or session level Next Steps Learn more about the Native Execution Engine Follow the step-by-step guide to Enable Native Execution Engine Explore supported data types and file formats in the Native Execution Engine documentation. Feedback Share your experience in the Fabric Community Forums Ask questions on Microsoft Q&AIntroducing High Concurrency Mode in Notebooks for Data Engineering and Data Science workloads in Microsoft Fabric
We are excited to announce a new high concurrency mode in Fabric for Data Engineering and Data Science. This allows users to share Spark compute across multiple notebooks within a workspace which means that you can run multiple Spark notebooks simultaneously on the same Spark session without compromising performance or security when paying for a single session. High concurrency mode offers an instant run experience avoiding session start delays and ~30X faster session start experience for the shared notebooks when running on custom pools. What is High Concurrency Mode? High concurrency mode allows sharing of Spark compute across multiple notebooks and allows their queries to execute in parallel. In this shared mode, the resources and configurations of each notebook are isolated from each other. As the session sharing is always within a single user boundary, users cannot access or modify the data or variables of another user's high concurrency session. High concurrency mode also leverages FAIR allocation to optimize the resource utilization and performance of the notebooks and ensures that each notebook gets a fair share of the executors available for the Spark application. Why Use High Concurrency Mode? High concurrency mode offers several benefits for Fabric Spark users, such as: Faster and easier Spark session start: You don’t have to wait for the Spark pool to spin up or configure the node sizes when you start a Spark session. As the session is already warmed up and running, attaching a new notebook to an existing Spark session gives a session start experience within ~5 seconds. You can also use custom Spark pools, which allow you to size the nodes, enable autoscaling, and dynamically allocate executors based on your Spark job requirements and with custom pools you would get a 30X faster session start experience for shared notebooks. Enhanced security and isolation: You can ensure that each user or query has its own isolated Spark session, which prevents data leakage or tampering. Do more by paying less: Achieve better compute cost savings by sharing a single session across multiple notebooks for your Data Engineering or Data Science workloads and only get billed only for the single session. How to Enable High Concurrency Mode? To enable high concurrency mode for your Fabric Spark workspace, you need to follow these steps: Go to the workspace settings in your Fabric workspace. Navigate to the Data Engineer/Science section. Select the Spark Compute menu Enable the High Concurrency Mode Option if its disabled (This option should be enabled by default for all Fabric Workspaces) 4. Save your changes. Once you enable high concurrency mode, you can run your notebooks in High Concurrency mode from the notebook menu ribbon. To learn more about using high concurrency in notebooks read: Sharing Spark Compute across Notebooks with High Concurrency Mode in Fabric. For more information on high concurrency mode, please read Overview of High Concurrency Mode in Microsoft FabricIntroducing High Concurrency Mode for Notebooks in Pipelines for Fabric Spark
We're excited to introduce high concurrency mode for notebooks in pipelines, bringing session sharing to one of the most popular orchestration mechanisms for enterprise data ingestion and transformation. Notebooks will now automatically be packed into an active high concurrency session without compromising performance or security, while paying for a single session. Key Benefits: Faster Session Start: High concurrency mode offers a significantly faster session start experience, reducing time to ~5 seconds for shared notebooks. This is approximately 30 times faster than traditional methods, resulting in substantial performance gains in pipeline execution. Session Tags: We've also introduced support for session tags, allowing users to target notebooks to specific high concurrency sessions for better session management. Why Use High Concurrency Mode? Rapid Spark Session Start: Notebook steps no longer need to wait for on-demand Spark pool spin-up when using custom pool configurations. By leveraging pre-warmed high concurrency sessions, notebook steps can quickly attach to an existing Spark session, significantly boosting overall pipeline performance. Cost Savings: Achieve better compute cost savings by sharing a single session across multiple notebooks for your data engineering or data science workloads. You'll only be billed for the single session, preventing potential queuing issues during peak usage hours. Example: Consider a pipeline with five notebook steps, each taking 5 minutes to execute. With traditional methods, starting a Spark session (3 minutes) for each step would result in a total runtime of approximately 40 minutes. Using high concurrency mode, this time can be reduced to 28 minutes, a 30% performance improvement. How to Enable High Concurrency Mode for Notebooks in Pipelines? To enable high concurrency mode for your Fabric Spark workspace, you need to follow these steps: Go to the workspace settings in your Fabric workspace. 2. Navigate to the Data Engineer/Science section. 3. Select the Spark Compute menu. 4. Navigate to the High concurrency tab. 5. Enable the option "For pipeline running multiple notebooks". 6. Save your changes. Once you enable high concurrency mode for pipelines in a workspace, all Spark sessions triggered by notebook steps in a pipeline will be High Concurrency sessions and the system automatically starts packing notebooks into the shared session. High Concurrency Mode for Notebook Steps in Pipelines By adopting high concurrency mode, you can enjoy faster pipeline execution, reduced costs, and improved overall efficiency for your data-driven workloads. To learn more about using high concurrency for notebooks in pipelines please refer to our documentation, High Concurrency Mode for Notebooks in Pipelines For more information on high concurrency mode, please read Overview of High Concurrency Mode in Microsoft FabricMore resilient Spark jobs with Efficient Scaledown (Preview)
Running large-scale data engineering workloads in Apache Spark often means dealing with expensive shuffle operations. Shuffle is one of the most resource-intensive operations that Spark runs on most, if not all, jobs. When executors holding shuffle data are removed during scale-down, Spark must re-execute entire stages—causing task retries, wasted compute, and unpredictable job runtimes. Efficient Scaledown in Microsoft Fabric Spark helps solve this problem. It decouples shuffle data from executor lifetime by routing shuffle output to Azure Blob Storage (or migrating it on demand), enabling faster cluster scale-down, improved fault tolerance, lower compute costs, and resilient jobs—with no changes to your queries, notebooks, or pipelines. The challenge: Shuffle data pins executors and drives up cost When Spark processes a query, it redistributes data between stages—a shuffle. By default, shuffle data is stored on each executor’s local disk, creating a tight coupling: Executors cannot be released until every downstream consumer has finished reading their shuffle data. If an executor is lost (crash, spot reclamation, or scale-down), Spark throws a FetchFailedException and must re-execute entire stages. Idle executors pile up because autoscale cannot reclaim nodes that still hold shuffle blocks, preventing Spark from scaling down effectively. This tight coupling is the single biggest reason clusters can’t scale down quickly and why shuffle-heavy jobs have unpredictable runtimes and inflated costs. How Efficient Scaledown solves this Efficient Scaledown breaks the coupling between executor lifetime and shuffle data through four cooperating capabilities: Remote Shuffle Manager (RSM): Allows Fabric Spark to read and write shuffle data to remote Azure Blob Storage instead of executor local disks. RSM coordinates where shuffle blocks are stored and retrieved. Shuffle Migration: Moves shuffle blocks off an executor before it is decommissioned instead of dropping them. Decision Layer: Per-stage runtime routing that keeps small shuffles local (fast) and offloads large shuffles to remote storage (resilient). Adaptive Query Execution (AQE) shuffle write optimization: Adjusts shuffle partitions at runtime for optimal throughput and performance. Figure: How Efficient Scaledown delivers best price performance by decoupling shuffle data from executor lifetime. Key benefits for your workloads Reduce compute cost by up to 43% With Efficient Scaledown, executors are released as soon as their work is done and no longer sit idle holding shuffle data. In recent TPC-DS benchmarks on Spark 4.1—where all queries ran in a single session with a 3-minute pause between queries: Total compute dropped from 3,724 to 2,121 VM-minutes, delivering a 43% cost reduction. Autoscale removes nodes immediately after task completion, eliminating “zombie” executors. Large shuffles go directly to Blob Storage instead of requiring large local disks. More resilient Spark jobs When shuffle data lives only on local disk, even small disruptions—like executor crashes or transient network issues— can force Spark to recompute entire stages. Efficient Scaledown ensures shuffle data is persisted or migrated, allowing jobs to continue without interruption. Executor crashes: Shuffle data survives and completed stages are not re-executed. Node preemption: Jobs proceed normally without costly retries. Graceful decommissioning: Shuffle blocks are migrated instead of dropped. Transient network issues: Fetch failures are avoided, preventing cascading job failures. Enable faster, truly elastic scaling The autoscaler can freely remove idle nodes and resize the cluster in response to workload changes. No executor is kept alive solely to hold shuffle or cache data. Improve runtime performance with smart routing The Decision Layer routes small shuffles to local disk and only large shuffles to remote storage. This delivers up to 57% runtime improvement compared to routing all shuffles remotely, while maintaining the same scaledown benefits. Get started Efficient Scaledown requires the Native Execution Engine (NEE) and is available starting with Fabric Spark Runtime 3.5 and later. Enable the full Efficient Scaledown capability with the following notebook configurations: # Remote Shuffle Manager (RSM) spark.conf.set("spark.remote.shuffle.enabled", "true") # Decision Layer - route large shuffles to remote storage and small shuffles locally spark.conf.set("spark.sql.rsm.decisionlayer.enabled.level", "stage") # Adaptive Query Execution (AQE) shuffle write optimization spark.conf.set("spark.sql.adaptive.shuffleWrite.enabled", "true") # Shuffle Migration during executor decommissioning spark.conf.set("spark.storage.decommission.shuffleBlocks.enabled", "true") spark.conf.set("spark.storage.decommission.shuffleBlocks.migrateToFallbackStorage", "true") # Prevent Delta snapshot cache from blocking cluster scale-down spark.conf.set("spark.dynamicAllocation.excludeDeltaSnapshotCache", "true") Alternatively, if you are configuring Spark settings at the workspace, environment, or Spark Job Definition level, use the equivalent Spark configuration properties: spark.remote.shuffle.enabled=true spark.sql.rsm.decisionlayer.enabled.level=stage spark.sql.adaptive.shuffleWrite.enabled=true spark.storage.decommission.shuffleBlocks.enabled=true spark.storage.decommission.shuffleBlocks.migrateToFallbackStorage=true spark.dynamicAllocation.excludeDeltaSnapshotCache=true For full configuration details, refer to the Efficient Scaledown documentation . Next steps Enable the Native Execution Engine Configure Efficient Scaledown Learn about autoscale in Fabric Spark We’d love to hear how Efficient Scaledown improves your workloads. Share feedback in the Fabric Community or on Microsoft Q&A.Improve performance for Python UDFs and complex data types in Microsoft Fabric’s Native Execution Engine
Where this matters: A concrete example For example, you have a data pipeline (extract, transform, load—ETL) that ingests JSON telemetry events, each containing nested arrays of user actions. Your pipeline uses a Python UDF to apply custom business rules—say, classifying events based on logic that doesn’t map cleanly to SQL expressions. Before NEE, this pipeline hits two performance penalties at once: The Python/Scala UDF forces serialization between the JVM and a Python worker for every batch. The nested arrays cause the engine to fall back from optimized columnar execution to slower row-based processing. With NEE enabled, both operations run in the native execution path. Internal benchmarks show 2–5x faster execution, depending on UDF complexity and data nesting depth. Why Python or Scala UDFs and complex types have been slow In standard Spark execution, Python or Scala UDFs require the engine to: Serialize rows from Spark’s internal format. Transfer data to a separate Python worker process. Execute the UDF, then serialize results back to the JVM. Each round-trip adds CPU overhead, memory pressure, and breaks vectorized execution. Similarly, complex data types (arrays, maps, structs) can force the engine off its optimized columnar path into row-based processing—negating the benefits of native execution for the rest of your query. As a result, teams either avoid UDFs (rewriting logic in SQL or Scala) or flatten nested schemas at ingestion time, which adds extra engineering effort. What the Native Execution Engine does differently NEE processes Python or Scala UDFs and complex types directly in the native columnar engine. Specifically: Python or Scala UDFs: The engine reduces serialization roundtrips between the JVM and Python, keeping data in columnar format longer. Vectorized UDFs see the largest gains, but standard UDFs also improve. Complex data types: Arrays, maps, and structs are processed natively in the columnar engine. Operations like explode, map access, and struct field extraction no longer force fallbacks to row-based execution. No code changes: Your existing Spark notebooks and jobs benefit automatically when NEE is enabled. No new API or syntax is required. Real performance gains Internal benchmarks on representative workloads show: Vectorized Python or Scala UDFs: up to 5.76× faster. Complex UDFs: 1.08–2.5× faster depending on UDF complexity. TPC-DS end-to-end workloads (including queries with complex types): up to 2.35× faster execution. These benchmarks were run on production-scale datasets using enterprise-typical cluster configurations. These are meaningful gains for enterprise-scale data engineering: Faster ETL pipelines Reduced cluster compute time Potentially lower operational costs Improved interactive analytics performance Actual performance improvements will vary based on workload characteristics, data shape, and cluster configuration. Native support for complex data types Fabric’s Native Execution Engine also adds optimized support for Arrays, Maps, and Structs. This allows pipelines to stay fully optimized without switching execution modes. Operations involving nested data can now benefit from native optimizations while preserving the flexibility developers expect from Spark. This is especially useful for advanced lakehouse optimization scenarios such as: Z-ORDER optimization Liquid clustering Semi-structured analytics Event-driven architectures Instead of flattening data or restructuring pipelines for performance reasons, teams can work with complex schemas without restructuring pipelines. Why this matters for the future of data engineering Analytics workloads are evolving to include AI, real-time decision-making, semi-structured data, and the growing importance of Python and developer-centric workflows. As workloads evolve beyond traditional BI, performance engines can no longer focus only on accelerating simple SQL queries. Platforms now need to support complex business logic, data science pipelines, deeply nested data formats, and hybrid workflows that combine ETL, machine learning, and interactive analytics in a single environment. Fabric’s Native Execution Engine reflects this shift. It’s designed not just for classic SQL acceleration, but for the next generation of analytical workloads that demand flexibility, scale, and high-performance execution across diverse programming models. You no longer need to trade off flexibility for performance. Get started The Native Execution Engine in Microsoft Fabric removes long-standing Spark bottlenecks by bringing Python UDFs and complex data types directly into the optimized native execution path. This helps improve performance and reduce execution overhead. To get started: Enable the Native Execution Engine — Navigate to Workspace Settings > Data Engineering/Science > Spark Settings > Environment. Change the default environment to one that has Native Execution Engine enabled. You can enable NEE in the Compute section of any Environment item. Run your workload — Execute an existing notebook or Spark job that uses Python or Scala UDFs or complex types. Compare execution time against a run without Native Execution Engine. Verify native execution — Check inline alerts to confirm there are no fallback alerts, indicating operations run on the native engine. Learn more Native Execution Engine overview Enable Native Execution Engine in your Fabric workspace We’d love to hear how the Native Execution Engine accelerates your Spark workloads. Try it today and share your feedback in the Fabric Community forum.Custom Live Pools for Fabric Data Engineering (Preview)
Author: Santhosh Kumar Ravindran, Principal Product Manager - Custom Live Pools for Fabric Data Engineering is a new way to schedule and manage “ready to run” Spark capacity so your teams spend less time waiting for compute and more time delivering outcomes.Notebook export controls in Microsoft Fabric
Author: Santhosh Kumar Ravindran, Principal Product Manager - Prevent data exfiltration by restricting the ability to download notebooks and rich DataFrames. This capability ensures that sensitive data remains within governed environments, reducing the risk of unauthorized access, sharing, or leakage outside approved boundaries.
