Spark Runtime 2.0 - Practical Overview and Stress Testing (with a built framework)
Link to the notebooks to reproduce: https://github.com/iurii-iurchenko-1/source-materials/tree/main/2026Q3_spark_runtime_2_0 Outline 1. What is Runtime 2.0 and why does it matter? 2. The five features explained 2.1 Vectorized CSV parsing in the Native Execution Engine 2.2 ANSI SQL mode inside the native engine 2.3 Incremental liquid clustering 2.4 Arrow-native Python UDFs 2.5 The VARIANT type in Delta tables 3. What are the main improvements as shown in stress testing 4. How did I test that? – Building the framework – The three notebooks – The tables and what each one measures 5. Main conclusions 5.1 If you are an architect 5.2 If you are an engineer 6. Resources used 7. Notebook results 7.1 Runtime 1.3 metrics 7.2 Runtime 2.0 metrics 8. How to reproduce 9. Links to documentation What is Runtime 2.0 and why does it matter? What is Runtime in Spark notebooks? It is an environment that includes the version of Spark, the default version of Delta tables, and other components that provide the engine to run Spark Notebooks in Fabric. The previous version of the runtime was Runtime 1.3, which included Spark 3.5. The latest Runtime 2.0 includes Spark 4.x and an upgraded version of Delta Tables. I chose 5 particular features introduced in Runtime 2.0 and built a framework to measure the difference in performance between versions 1.3 and 2.0. These features are: Vectorized CSV parsing in the Native Execution Engine ANSI SQL mode inside the native engine Incremental liquid clustering Arrow-native Python UDFs The VARIANT type in the delta tables Let me explain these features in depth. 1 Vectorized CSV parsing in the Native Execution Engine This feature means the engine-level optimization of reading CSV files. It may help if your organization has heavy jobs to ingest CSV files at scale. 2 ANSI SQL mode inside the native engine This feature is powerful, and not a lot of people know about it. If you run Spark queries and a formula fails to calculate some values, Spark quietly returns NULL. If ANSI is enabled, the failing will be loud. It is needed in critical pipelines. Now that feature is supported by the native engine. 3 Incremental liquid clustering This new feature influences the behavior of the OPTIMIZE command on top of a Delta table. In the old version, the behavior was to reprocess all the files. In the new version, only new unclustered files will be reprocessed. Isn't that good? 4 Arrow-native Python UDFs This feature introduces the new optimized algorithm to process Pandas UDF functions. When are they needed? During a migration of the Python/Pandas notebooks. So, instead of rewriting Pandas functions and introducing new bugs, implementing this mechanism may help. IN the new engine, it is optimized to avoid converting to Pandas internally, which saves time and resources. Also, if you know PyArrow functions to proceed with strings, they may also be utilized instead of plain Python. 5 The VARIANT type in the delta tables Variant type promises performance optimization on top of JSON-like data. So, it looks like you decoded the data source as JSON, wrote it once (as a VARIANT column), and it may be processed faster later, when you need to extract some information from it. What are the Main Improvements as Shown in Stress Testing Final Results CSV Scan: I observed ~50% performance improvement on reading CSV files in Runtime 2.0 (9.2 sec vs 6.5 sec) ANSI SQL mode - significant improvement in Runtime 2.0 (see the code and logs at the end of the notebook). 5.3 sec ANSI off →12 sec ANSI ON on Runtime 1.3 vs 3.6→3.7 on Runtime 2.0. Incremental liquid clustering - the new algorithm introduces incremental clustering. So, now, we may have two strategies: incremental clustering or full rebuilding of Z-cubes. So, the introduces incremental clustering gives a significant boost in performance, but still loses the battle compared with full optimization/rebuilding of Z-cubes. Arrow-native Python UDFs give almost a 3x boost on the new runtime due to avoiding converting it to Pandas and back Variant type. Here, I observed that Variant may not give a significant boost in reading performance. Sometimes, it may have even slower performance. But it uses disk space better. How did I test that? To test that, I vibe-coded a framework in Claude to cover all the features mentioned above. It was a back-and-forth process. I carefully selected 5 features recently introduced to Runtime 2.0, built tech requirements for the system to test it, generated the code for three notebooks, ran it, got the first issues and output, improved it, and repeated it a few times till the process was smooth. I tested it in my personal MS Fabric environment with F2 capacity. The solution includes three notebooks: NB_00_GENERATE_DATA NB_10_BENCH_RUNTIME_1_3 NB_20_BENCH_RUNTIME_2_0 The tables are: fact_sales — 20M-row Delta fact table. Feeds the ANSI × NEE arithmetic query and all four UDF variants. Files/bench/raw_csv — the same 20M rows written as uncompressed CSV, 2.1 GB. The only asset that tests vectorized CSV parsing in the native engine. logs_json — 3M nested JSON documents held in a STRING column. The baseline for the VARIANT comparison, and the source for the VARIANT migration. events_lc_r13 — liquid-clustered Delta table, left with unoptimized new data on top. Measures the Delta 3.2 full-rewrite OPTIMIZE on Runtime 1.3. events_lc_r20_inc — identical twin of the above. Measures Delta 4.2's incremental OPTIMIZE on Runtime 2.0. events_lc_r20_full — identical triplet. Measures OPTIMIZE ... FULL on Runtime 2.0, giving a same-session comparison against the incremental arm. logs_variant — not from the generator; created by the Runtime 2.0 notebook itself via parse_json. Its creation time isthe VARIANT migration cost, and it is what the colon-operator query reads. After that, I ran the notebooks NB_10_BENCH_RUNTIME_1_3 and NB_20_BENCH_RUNTIME_2_0 to get the main metrics. You may find these metrics below. They show how new features in Runtime 2.0 differ in performance for different use cases. Main Conclusions If you are an architect The main point with Runtime 2.0 is that it becomes GA. That means the older versions will be deprecated step-by-step, and it is time to plan the modernizational upgrade of the current infrastructure to use Runtime 2.0. If you are an engineer Invest your time in learning about NEE (Native Execution Engine) features, and test how they impact your real data flow. They are efficient, especially on complex logic query Arrow native Python UDFs may be 3x faster compared to ordinary UDFs if you use only the library's functionality and don't use Python variables there. This library is rich. So, it is possible, especially for text operations. If you didn't know what ANSI SQL mode is, now you know. If you want to Spark fails loudly if some formula didn't take place, that's it. Now, it is much more supported by the engine. Resources Used I tested all that on a personal tenant with Fabric Capacity F2. Notebooks Results Runtime 1.3 Metrics Runtime 2.0 Metrics How to Reproduce? Import the notebooks mentioned at the beginning of the article to your Fabric tenant Create a Lakehouse "LH_RUNTIME2_0_TEST Run NB_00_GENERATE_DATA Sequentially, run NB_10 & NB_20, creating before runtimes 1.3 and 2.0 for them. Links to Documentation https://community.fabric.microsoft.com/blog/fbc_fabricupdatesblogs/fabric-august-2026-feature-summary/5325824 https://community.fabric.microsoft.com/t5/Fabric-Updates-Blog/Fabric-July-2026-Feature-Summary/ba-p/5325823 https://learn.microsoft.com/en-us/fabric/data-engineering/runtime-2-0 https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview?tabs=sparksqlVibe Coding Fabric Notebooks in VS Code: A Production Checklist
1. Key Insights Level: 300 - practical implementation. You need to know Spark and Fabric notebooks. You do not need to know how the extension works internally. The situation. Fabric notebooks are an excellent environment for developing and operating Spark workloads. But once a notebook passes a few hundred lines, the engineering activities - refactoring, documentation, architecture review, large-scale changes - become hard to do in a browser. The alternative. The Fabric Data Engineering extension for VS Code opens a different workflow: Download the notebook from Fabric Work on it locally with a modern coding agent - Claude Code, the Claude extension, GitHub Copilot agent mode, OpenAI Codex, or a local agent Publish it back You keep Fabric's managed Spark platform, and you add what VS Code already has: advanced refactoring, Source Control, review workflows, and project-wide AI context. What surprised me. The value was not where I expected it, and the line is not where most people draw it. Generating a notebook from scratch was the weakest use. Ask for the whole thing at once and you get something plausible that you then have to verify end to end - which costs more than writing it yourself. Generating code block by block worked well. "Add one more cell that does X" is bounded. I can read it, run it, and reject it in seconds. Improving notebooks that already exist was the strongest of all. The existing code is the specification, so the agent has something to be correct against. So the useful rule is not generation bad, refactoring good. It is about scope: the agent is reliable when the change is bounded and verifiable in one reading, and unreliable when it is unbounded. What it costs. Every capability here has its limitation, and they belong in the same paragraph: Notebook synchronization is not a real-time collaborative experience. Treat it as checkout, not as co-editing. AI-generated changes require human review - and the notebook diff you review is not a code diff. That is Section 3, and it is the part I did not expect. Production workloads still need governance, validation and security controls. The agent does not provide them. So, the conclusion up front. Successful Spark vibe coding is less about replacing engineering discipline and more about augmenting it. If you skip the review step, you have not saved time - you have only moved the risk later. What is in this article. A practical workflow: prerequisites, the development patterns that worked, three review strategies with their trade-offs, production safeguards, the prompt set I reuse, and the lessons from real experimentation. The prompts are at the end. 2. When Spark Notebooks Vibe Coding is Helpful Experimenting, I found a few cases where having Spark in VSCode is more beneficial than having to solve its complexity and logistic: Converting messy 30-cells notebook into a well-structured sanctuary Solving complex technical problems (API ingestion, complex transformation, complex dependencies), to save time for copy-paste, and doing that coding step-by-step, with powerful AI assistance in place Refactoring becomes just easier using VSCode. It may be done using AI, by prompting. Or, if we want to do it manually, F2 (Windows) or Fn-F2 when the desired variable is selected will help. It is a very clean way to refactor code at scale, with minimal risk. 3. When is Vibe Coding not worth the cost? Simple, straightforward notebooks - easier to modify using the web or copy-paste AI-generated code. Highly critical pipelines, where the mistake costs a lot or the privacy model is incompatible with AI provider terms - these are subject to reconsider using AI in autonomous or partly autonomous mode. Now, let's observe some workflows and prerequisites to be able to vibe-code in Spark notebooks. 4. Prerequisites What is needed to start vibe-coding in Fabric/Spark? VSCode - indeed; Fabric Data Engineering VS Code - to connect to Fabric; Coding agent (I use the Claude extension, installed in VSCode) Turned on AutoSave in VSCode will help avoiding overwriting and keep the changes actual. 5. Workflow Extension -> Workspace -> Notebook -> Download -> Open Then - modifying the code, using vibe-coding. Then - publishing back. If the notebook was updated on the web, it is possible but harder to merge all the changes. I would omit that scenario entirely to avoid any issues. Full journey may look like that: During testing, I found that notebook synchronization between Fabric and VS Code should not be treated as real-time collaboration. If substantial changes were made through the Fabric web interface, re-downloading the notebook may be safer than attempting to merge changes manually. Disclaimer: If somebody changed the code in the web interface, getting these changes may be a challenge. Simple Update icon doesn't help, even though it says the update was successful. So, additional awareness is required for that matter. It will be an alert during publishing the code with the option to accept local or remote change: 6. Production-ready best practices 6.1. Security & Privacy If we work in production, it is important to follow the organization's rules, and use only appropriate AI tools. 6.2. Responsibility AI may change code very unpredictably. It may break production silently. Two options for handling that. First is asking AI to show all the changes. The second one is to manually check points at the beginning of vibe-coding and at the end. 6.2.1. Strategy one - risky: Asking AI to show differences Before vibe-coding, give an instruction similar to this: Before making any file modifications, show me the proposed diff and wait for my approval. As a result, the agent may showcase the changes it made. The key thing here is it may miss something or hallucinate. So, for highly critical pipelines, it may not work. This strategy is good for prototype, PoC, and MVP development. But, actually, I would say, it is overall the best time to use vibe-coding. Fully productionized critical system - it is debatable whether the outcome outweighs risks using AI assistants to write code in agentic mode. It is much safer to use manual coding, by copy-paste-debug without writing to the final tables. 6.2.2. Strategy two - conservative: Backup notebook and related data Backing up related data and code - it is a highly conservative strategy. It make sence doint this only if vibe-coding will bring much more benefits than the other ways of changing the logic 6.2.3. Strategy three: Checking the changes before and after Open notebook in VS Code. Initialize Source Control (Git). Let Claude modify the notebook. Open Source Control (Cmd+Shift+G). Click the modified notebook, to see the differences, before publication. 7. Additional block - Prompts for success 7.1. Prompt 1 - Notebook Structure Overview the notebook, and add its title, description, and after that - all the Headlines and sub-headlines, with prefixes like 1., 2., 3., 1.1., 2.2., etc 7.2. Prompt 2 - Find weak points Check code and find what may be improved. Don't do any changes. Only give the comments. 7.3. Prompt 3 - Refactor Notebook Refactor this notebook without changing functionality. Extract duplicated logic into reusable functions, improve variable naming, add comments where necessary, and reorganize code into logical sections. 7.4. Prompt 4 - Documentation This is probably the highest ROI prompt. Generate markdown documentation for this notebook. Explain business purpose, inputs, outputs, dependencies, assumptions, and execution flow. Many Fabric notebooks have zero documentation. 7.5. Prompt 5 - Performance Review This is very Spark-specific. Review this PySpark code for performance issues. Identify unnecessary shuffles, collect operations, driver-side processing, inefficient joins, and repeated scans. 7.6. Prompt Set for Feature Building 7.6.1. API Ingestion Prompt: Review this notebook and create a reusable API ingestion framework with retry logic, pagination support, logging, and configuration-driven endpoints. 7.6.2. Data Quality Framework Prompt: Convert all validation checks into reusable functions and generate a summary report at the end of execution. 7.6.3. Monitoring Notebook Prompt: Extract all monitoring logic into reusable functions and create a centralized alerting section. 7.6.4. Legacy Notebook Cleanup Prompt: Refactor this notebook into logical sections and eliminate duplicated code. 8. Before & After examples Before: After: 9. What agents may we use here? The agents selection is not limited. When we work with the notebooks that way, they are stored locally. It means, Claude Code, OpenAI Code, Loca agent from VSCode, Github Copilot Agend mode and others are fully available for us. 10. Lessons Learned Documentation generation produced the highest ROI. Structural refactoring was more reliable than business logic generation. Source Control review was essential. Small notebooks rarely justified the overhead. Large notebooks benefited the most. This makes the article feel based on real experimentation rather than theory. 11. Conclusion Spark notebooks are a surprisingly good candidate for vibe coding. The extension also supports Python notebooks and Spark SQL, from what I tested. Large notebooks often contain repetitive patterns, technical debt, and insufficient documentation-areas where modern coding agents can provide significant value. However, AI-generated changes should always be reviewed before publication, especially for production workloads. Combined with VS Code Source Control and the Fabric Data Engineering extension, vibe coding can become a practical addition to a data engineer's toolkit rather than a replacement for engineering judgment.Direct Lake on SQL Endpoint + OneLake Security: Delegated vs. User Identity Mode (A Field Guide)
Direct Lake on SQL Endpoint + OneLake Security: the field guide. Model owned by an SPN, report published, refresh green — but every visual fails with QueryUserError? The reason is which identity actually reads the data, and it's not the one you think. Delegated vs. User identity mode, shortcut + native tables, SPN refresh traps, and the new Fabric default — decoded with diagrams and permission matrices. Save yourself the debugging session.Direct Lake on SQL Endpoint + OneLake Security: Delegated vs. User Identity Mode (A Field Guide)
Direct Lake on SQL Endpoint + OneLake Security: the field guide. Model owned by an SPN, report published, refresh green — but every visual fails with QueryUserError? The reason is which identity actually reads the data, and it's not the one you think. Delegated vs. User identity mode, shortcut + native tables, SPN refresh traps, and the new Fabric default — decoded with diagrams and permission matrices. Save yourself the debugging session.
OneLake Security in Microsoft Fabric
Tired of managing security three different ways for SQL, Spark, and Power BI? Microsoft Fabric's OneLake Security changes the game with a single unified layer that enforces table, row, column, and folder-level access control across every engine. Define your security policies once in the portal, and they're automatically enforced whether users query through notebooks, SQL endpoints, or DirectLake reports. This blog walks you through setting up real-world demos with sample data, creating granular roles via UI clicks, and validating enforcement across engines. Ready to simplify your data security? Let's build it step-by-step.Fabric at Scale - Part 1: Automating Table Discovery with SemPy and the Fabric API
How do you keep track of hundreds of tables scattered across 50–200 lakehouses? Manually, you can't — not reliably. This post walks through two practical approaches to automating a Fabric table audit: SemPy for simplicity, and the direct Fabric API when you need more control. Either way, your governance game just got an upgrade.Fabric SDD+AI Series (3/3) | Mastery in Production: Governance, CI/CD and Final Checklist [PT/EN/ES]
🇧🇷 PT: Chegamos à reta final! Descubra como garantir um "Go-Live" sem estresse no Microsoft Fabric usando automação CI/CD, governança de dados e o checklist definitivo de prontidão para produção. Aprenda a usar a IA para revisar a segurança do seu projeto antes do lançamento. 🇺🇸 EN: We've reached the final stretch! Discover how to ensure a stress-free "Go-Live" in Microsoft Fabric using CI/CD automation, data governance, and the ultimate production readiness checklist. Learn how to use AI to review your project's security before launch. 🇪🇸 ES: ¡Llegamos a la recta final! Descubra cómo garantizar un "Go-Live" sin estrés en Microsoft Fabric utilizando automatización CI/CD, gobernanza de datos y el checklist definitivo de preparación para producción. Aprenda a usar la IA para revisar la seguridad de su proyecto antes del lanzamiento.Migrating SSIS, SSAS, SSRS to Fabric | Migrando para o Fabric | Migración a Fabric
PT-BR: Guia estratégico para migrar SSIS, SSAS e SSRS para a arquitetura unificada do Microsoft Fabric. EN: Strategic guide for migrating SSIS, SSAS, and SSRS to the unified Microsoft Fabric architecture. ES: Guía estratégica para migrar SSIS, SSAS y SSRS a la arquitectura unificada de Microsoft Fabric.
