If you haven’t already, check out Arun Ulag’s hero blog “Microsoft Build 2026: Building Agentic Apps with Microsoft Fabric and Microsoft Databases” for a complete look at all of our Microsoft Build announcements across our Fabric and database offerings.
___________________________________
The evolution of data transformation in Microsoft Fabric has reached a new milestone with Dataflow Gen2 (CI/CD). Recent benchmarks reveal not just incremental progress, but a leap forward—delivering order-of-magnitude improvements in both cost and performance compared to previous generations, including Semantic Models. This blog post explores the numbers, the architectural impact, and why Dataflow Gen2 (CI/CD) with Lakehouse and Power BI Direct Lake is the best setup for customers.
Dataflow Gen2 introduces a broader set of capabilities than previous generations—while keeping the “get started” experience straightforward. Customers can begin with defaults and then adopt additional optimizations as needed for their specific data volumes, transformation patterns, and operational requirements. In most cases, moving the same Power Query logic to Dataflow Gen2 with default settings already delivers a meaningful performance improvement. A big reason is the Modern Query Evaluation Service used by Dataflow Gen2 (CI/CD), which is designed to accelerate M evaluation—particularly in patterns that are extremely common in real-world customer queries, like row-by-row transformations and other transformation-heavy steps where folding is limited.
Dataflow Gen2 performance of options
One important shift with Dataflow Gen2 (and especially Dataflow Gen2 (CI/CD) is that performance no longer comes from a single “faster engine.” Instead, Gen2 offers multiple, complementary features you can use depending on your workload—high-throughput ingestion into OneLake, faster M evaluation for transformation-heavy logic, and parallel execution for file-based scale-out patterns.
To keep things simple, teams can follow a progression: start with Dataflow Gen2 defaults (many customers see immediate gains here), then lean on the Modern Evaluator for transformation-heavy workloads, and only then add more specialized optimizations like Fast Copy (when ingestion throughput is the bottleneck) or partitioned compute (when partition-friendly sources can be parallelized). This blog helps customers recognize which scenario they are in and which lever to pull—without needing to pull them all.
- Staging: materializes data once so downstream queries can reuse it without rereading the source—ideal for ELT patterns where you want to separate ingestion from transformation and support multiple downstream outputs from the same foundation.
- Fast Copy: optimized for very large ingestion into OneLake (Lakehouse) using a high-throughput copy backend—ideal for ELT patterns where you want to land data quickly and transform after landing.
- Modern Query Evaluator: a newer M execution engine that can significantly reduce evaluation time, especially for transformation-heavy logic and row-by-row operations.
- Partitioned compute: parallelizes parts of your logic when the engine can partition the source (commonly file-based sources), reducing end-to-end refresh time through concurrent processing.
- …and more: Gen2 continues to add new performance, scale, and operational capabilities over time, so it’s worth revisiting options as your workloads evolve.
Performance and scale features: How they work and when to use them
Staging in Dataflow Gen2 allows a query’s output to be materialized once and then reused by downstream referenced queries without rereading the original source. This creates a cleaner separation between ingestion and transformation, which is especially valuable in ELT patterns and other scenarios where the same intermediate result needs to support multiple downstream outputs.
In practice, staging helps reduce repeated source access, improves consistency across derived queries, and provides a stronger foundation for scaling more complex or reusable data preparation workflows. It also serves as the basis for additional optimizations, since capabilities like Fast Copy and downstream compute options become more effective when built on top of a staged result.
Fast Copy: High-throughput ingestion into OneLake
Fast Copy helps move large amounts of data faster in Dataflow Gen2 by switching ingestion to a copy-optimized backend once data size crosses certain thresholds. Customers keep the familiar dataflow authoring experience but get the performance characteristics of a pipeline-style copy operation. Today, Fast Copy loads directly to a Lakehouse destination, and it works best when ingestion-time transformations remain lightweight (for example, selecting/renaming columns and type changes for file sources). If richer transformations are required, a common pattern is to use Fast Copy to land data to staging quickly, and then reference that staged data in a follow-on query for heavier transforms at scale.
Modern query evaluator: Optimized M performance for per-row transformations
The Modern Query Evaluation Engine (“Modern Evaluator”) is a newer execution engine for Power Query (M) in Dataflow Gen2 (CI/CD). It’s designed to reduce evaluation overhead and improve execution plans, which can translate into substantially faster refreshes for many real-world queries—including transformation-heavy flows and scenarios that do row-by-row operations during ingestion. It’s enabled by default in new Dataflow Gen2 (CI/CD) items, and you can disable it to fall back to the standard engine if you hit compatibility issues with a specific query or connector.
Partitioned Compute: Parallel execution for partition-friendly patterns
Partitioned compute (currently preview and available in Dataflow Gen2 with CI/CD) focuses on doing more work in parallel. When the engine can partition the source (for example, splitting a large set of files into partitions), it can evaluate those partitions concurrently and significantly reduce end-to-end refresh time.
In practice, teams enable the partitioned compute option, author queries in a partition-friendly way (often via the Combine Files experience which can create partition keys), and keep the partition key column in the query so the engine can apply the optimization. This is a great fit for file-based ingestion scenarios where folding/partitioning is available, and evaluation can be scaled out rather than only accelerating the initial copy.
Canonical scenarios: How Dataflow Gen1 and Dataflow Gen2 compare
Different Dataflow Gen2 capabilities shine in different patterns. Some workloads are primarily about moving large amounts of data quickly, others are limited by transformation complexity, others benefit from parallel execution, and others work best when ingestion and transformation are separated into a staged ELT pattern. The sections below summarize the most common scenarios and the feature that is typically the best fit for each.
Scenario 1: Copy Data
- What it does: Bulk-load consolidated Parquet files from ADLS Gen2 into a Lakehouse with no transformations.
- Capability enabled: Fast Copy
- Benchmark results (execution time, hh:mm:ss):
- Dataflow Gen1: 01:42:18
- Dataflow Gen2: 00:07:43
- Takeaway: Use Fast Copy when the workload is primarily about moving large volumes of data into OneLake as efficiently as possible, with little or no transformation during ingestion. It is the best fit for EL-style patterns where throughput and lower computing cost matter more than in-flight shaping.
Scenario 2: Heavy data shaping
- What it does: Apply non-foldable transformations such as filtering, derivations, and cleansing before loading into a Lakehouse.
- Capability enabled: Modern Evaluator
- Benchmark results (execution time, hh:mm:ss):
- Dataflow Gen1: 01:13:44
- Dataflow Gen2: 00:46:15
- Takeaway: Use the Modern Evaluator when the bottleneck is transformation work itself, especially in queries that are non-foldable or only partially foldable. This is the right fit for shaping heavy flows where the goal is to improve execution time without changing the underlying Power Query logic.
Scenario 3: Combine files
- What it does: Combine and transform partitioned Parquet files in parallel, then load the result into a destination.
- Capability enabled: Partitioned Compute
- Benchmark results (execution time, hh:mm:ss):
- Dataflow Gen1: 01:40:57
- Dataflow Gen2: 00:04:48
- Takeaway: Partitioned compute is a strong fit when data is spread across many files or partitions and the work can be parallelized. In these scenarios, performance improves by evaluating partitions concurrently instead of processing everything sequentially through a single path.
Scenario 4: ELT patterns
- What it does: Stage data once, then run downstream referenced transformations and analytical outputs from that staged result.
- Capability enabled: Staging + Fast Copy
- Benchmark results (execution time, hh:mm:ss):
- Dataflow Gen1: 02:42:44
- Dataflow Gen2: 00:05:53
- Takeaway: Use staging when ingestion and transformation in a single query begin to compete for the same resources. By materializing data once and allowing downstream queries to reference that staged result, Dataflow Gen2 creates a cleaner separation between load and transform phases. This pattern becomes even more powerful when combined with Fast Copy to accelerate ingestion and High-Scale Compute to scale downstream transformation work.
The important takeaway is not that there is a single best option for every workload, but that Dataflow Gen2 gives customers multiple levers—Fast Copy, Modern Evaluator, partitioned compute, and staging—so they can choose the right approach for their performance, scale, and transformation needs.
What’s changed in Dataflow Gen2
The landscape of data transformation in Microsoft Fabric has evolved dramatically. Dataflow Gen2 (CI/CD) introduces a modern query evaluation engine and parallelized execution, fundamentally changing what’s possible for enterprise-scale data transformation. These improvements are not just incremental; they redefine the boundaries of speed, scale, and cost efficiency.
Using Dataflow Gen2 with Lakehouse and Direct Lake
Pairing Dataflow Gen2 (CI/CD) with a Lakehouse destination unlocks seamless downstream usage in Power BI Direct Lake. This setup enables real-time analytics, direct data access, and eliminates the need for intermediate ETL steps. Customers can transform data at scale, land it in a Fabric Lakehouse, and immediately leverage Power BI’s Direct Lake mode for blazing-fast insights—delivering both performance and simplicity.
Why lower execution time now also means lower cost
For customers, the value of these benchmark results is now even clearer: faster execution can also mean lower overall cost. Following the Dataflow Gen2 pricing improvements released in September 2025 at FabCon Europe, the same performance gains that reduce refresh time can also significantly reduce CU consumption for the workload. In other words, when Dataflow Gen2 helps you complete the same transformation faster, you can also benefit from better cost efficiency compared to Dataflow Gen1.
Figure: Dataflow Gen2 Pricing improvements.
For a detailed breakdown of how Dataflow Gen2 pricing works across Standard Compute, High Scale Compute, and Fast Copy, see the Dataflow Gen2 pricing documentation.
Key takeaways
- Order-of-magnitude improvements in both cost and performance with Dataflow Gen2 (CI/CD).
- Multiple performance and scale levers in Gen2—Fast Copy for high-throughput ingestion into OneLake, Modern Evaluator for faster M execution (especially transformation-heavy logic), and partitioned compute for parallelism in partition-friendly patterns.
- Lakehouse and Direct Lake unlock real-time analytics and operational simplicity.
For customers still running on Gen1 or relying solely on Semantic Models, this is a good time to upgrade. Dataflow Gen2 (CI/CD) with Lakehouse and Power BI Direct Lake isn’t just an incremental step—it’s a leap into the future of enterprise-scale analytics.
Additional resources
Check out the following resources to learn more:
Community Admin