At Microsoft, we believe that empowering every user to shape and transform data should be simple, powerful, and cost-effective. A major wave of enhancements is now available for Dataflow Gen2, the next-generation data transformation engine in Microsoft Fabric.
These updates are the result of deep collaboration with our customers and community. You spoke, and we listened. Whether you're building quick prototypes or orchestrating enterprise-scale data prep pipelines, Dataflow Gen2 is now faster, smarter, and more affordable than ever.
Smarter Pricing, More Value
We’re making Dataflow Gen2 (CI/CD) dramatically more cost-effective, starting now. Introducing a new 2-Tier Pricing Model Based on Query Evaluation Duration:
- First 10 minutes for each query: now billed at 12 CU — a 25% reduction.
- Beyond 10 minutes for each query: billed at just 1.5 CU — a 90% reduction.
This change directly responds to customer feedback, your feedback, making Dataflow Gen2 more cost-effective for data transformation scenarios involving data of any shape and size.
Effective immediately for Dataflow Gen2 (CI/CD) operations. To benefit, upgrade non-CI/CD items using 'Save as Dataflow Gen2 (CI/CD)'. For more details, refer to the Dataflow Gen2 pricing for Data Factory in Microsoft Fabric documentation.
Making Dataflow Gen2 (CI/CD) Runs Significantly Faster
Performance matters, which is why we’ve rolled out the Modern Query Evaluation Service, which delivers substantially faster dataflow run times (depending on your scenario). Whether you're refreshing data hourly or processing large volumes overnight, you’ll see a noticeable boost in speed and efficiency.
We are also adding the ability to parallelize query runs through partitioning. Together, both these improvements yield dramatic runtime performance that will have an overall positive impact not only on the user experience but will yield further reduction in cost.
Enhanced Design-Time Performance
With our new Preview only steps feature, the design-time experience for Dataflow Gen2 (CI/CD) items is now significantly faster. You can sample data previews and reduce wait times for subsequent previews by orders of magnitude. It’s a game-changer for productivity.
Learn more, refer to the Preview only step in Dataflow Gen2 (Preview) documentation.
Expanded Output Options for Dataflow Gen2
- Fabric Lakehouse Files (CSV): We are introducing the ability to write directly to CSV files into your Lakehouse for further processing. This is a game-changer for teams working with Spark, Python, or other open-source tools. Simply use the existing Lakehouse connector and switch to new file mode right away when you are choosing your destination target.
- Azure Data Lake Storage Gen2: We’re adding a new data destination that enables you to write new CSV files into your Azure Data Lake Storage Gen2. This is particularly valuable for organizations managing large volumes of structured and unstructured data. It supports centralized ingestion pipelines and staging areas for transformation, leveraging Azure’s governance and access control frameworks.
- Snowflake databases integration: Currently in sneak peek, preview will launch soon. Snowflake’s scalable architecture enables seamless data storage and analytic capabilities, providing users with high performance and flexibility for modern data workflows. This preview marks a significant step in our commitment to expanding our Dataflow Gen2 data destination offerings to include even more destinations spanning multiple clouds and data platforms in the future.
- SharePoint Files (CSV) destination (Generally Available), Excel (XLSX) support (Coming Soon). SharePoint Files destination streamlines data integration into collaborative workflows, enhancing document management with secure access controls, versioning, and efficient cloud storage.
The Enhanced Output Data experience now includes:
- OneLake Catalog integration for seamless discovery of Fabric destinations.
- An enhanced Default Destination experience, allowing you to easily configure a default output destination to be used by all your queries in a dataflow.
Check out the following video to see all these Data Destination enhancements in action:
Learn more about these features: Dataflow Gen2 data destinations and managed settings.
Parameterized Dataflows (Generally Available)
With enhanced experiences to discover and specify parameter values in the Dataflow activity within Pipelines, and we’ve added support for parameterizing output destinations too. This makes it easier to reuse and deploy dataflows across environments with minimal changes.
To learn more about how to Use public parameters in Dataflow Gen2 (Preview), refer to the documentation.
Fabric Variable Libraries now natively supported in Dataflows
You can now reference Fabric Variable Libraries directly within your dataflows, making it such that your Dataflow Gen2 (CI/CD) logic can adapt based on workspace-level variables.
To learn more about how to Use Fabric variable libraries in Dataflow Gen2 (Preview), refer to the documentation.
A smarter, AI-powered, Modern Get Data experience
We are enhancing the Modern Get Data experience within Dataflow Gen2 (CI/CD) with Copilot capabilities. Now you can easily ingest and transform data with natural language as part of the data discovery and connection experience.
To learn more, refer to the Unblocking Copilot in Get Data for Modern data ingestion and transformation (Preview) documentation.
AI-powered Data Transformation prompts in Dataflow Gen2 (Coming Soon)
Use natural language prompts to apply transformations like Sentiment Analysis, Summarization, Categorization, and more. It has never been easier to transform & enrich your data with intelligence.
Upgrading from Gen1? We’ve Got You Covered
Still using Dataflow Gen1? Now’s the perfect time to upgrade. The 'Save Dataflow Gen1 as Dataflow Gen2' feature is now Generally Available, with enhanced capabilities to make your transition smooth and seamless.
To learn more, refer to the Migrate to Dataflow Gen2 (CI/CD) using Save As (Preview) documentation.
We’re Just Getting Started
These updates reflect our deep commitment to listening to our customers and acting on your feedback. We’re incredibly excited about the new Dataflow Gen2 (CI/CD) capabilities these enhancements bring to Fabric users—and we can’t wait to see what you build with them.
As always, thank you for being part of this journey with us. Let’s keep transforming data—and possibilities together.