Forum Discussion
What tools are used for Data Engineering in Fabric?
What tools are commonly used for data engineering tasks in Microsoft Fabric?
I would like to understand which tools are recommended for building data pipelines, data transformation, and storage.
Hello raginijaiswar19
Here are the tools Microsoft Fabric offers for data engineering—covering pipelines, transformations, and storage.
Data Ingestion & Orchestration
Pipelines (Data Factory Gen2)
- Enables orchestration workflows with activities like Copy Data, Dataflow Gen2, Lookup, ForEach, If Condition, etc. These are akin to Azure Data Factory and support both batch and streaming ETL.
- Pipelines let you sequence dataflows, notebooks, stored procedures, and copy tasks in a dependable pipeline.
Dataflow Gen2
- A low-code, Power Query-like interface with 300+ transformations for cleaning and shaping data.
- Ideal for transforming data before it’s stored in your lakehouse.
Compute & Transformation
Apache Spark Jobs
- Define Spark job definitions to process structured or unstructured data at scale—supports batch and streaming.
- Includes autoscale, library management, mentors like Spark advisor, and integration with Livy API for programmatic control.
Notebooks
- Use interactive notebooks with Python, Scala, or T-SQL for ingestion, transformation, and analysis.
- Fully integrated with VS Code, Git, and deployment pipelines.
Storage & Data Modeling
Lakehouse on OneLake
- Combines lake (raw/unstructured) and warehouse (curated) data in Delta tables, all stored in OneLake.
- Supports the Medallion architecture (Bronze/Silver/Gold) for progressive transformation.
Materialized Lake Views
- Precompute and store results of complex queries to accelerate analytics and implement medallion patterns.
Fabric Warehouse (SQL Pools)
- A SQL-based analytics store optimized for BI/reporting workloads and tightly integrated with lakehouse data.
Developer Experience & Tools
VS Code Extension & Spark Environments
- Author notebooks and Spark job definitions directly in VS Code, manage environments, deploy via Git.
Livy API
- Programmatic interface to submit Spark jobs for custom automation use cases.
GraphQL API
- Expose lakehouse data via a GraphQL endpoint for application integration.
Copilot for Data Engineering
- AI-powered assistant to generate Spark/T-SQL code, detect transformation errors, and automate pipeline creation via natural language.
Integration & Governance
Synapse Data Engineering integration
- Offers a unified experience for Spark-based engineering, tightly aligned with OneLake and lakehouse architecture.
Seamless Azure Synapse Integration
- Enables hybrid architectures using Synapse for MPP workloads and Fabric for lakehouse, pipelines, notebooks, BI, and governance.
Hope this helps, please appreciate giving a Kudos or accepting as a Solution!
I would focus on Data Pipelines (former Data Factory) and Spark Notebooks as a core.
Data pipelines are great for the following purposes:
1) Scheduling notebooks running
2) Ingesting tables as is (e.g. copy SQL Server table to data lake)
3) If people prefer a graphical interface over code (but disclaimer - it may create a massive tech debt, because it is much harder to see hidden complex relations in the visual code)
Spark notebooks (PySpark + Spark SQL) are a go-to option for me for the following cases:
1) bronze->silver layer
2) API ingestion
3) complex transformation in any place
I would mention Spark Notebooks separately with SQL code. They are great for building a gold layer on top of a clean, enriched layer. They are easy to read and operate. But the precondition is a clean and friendly silver layer.
What else should I mention here?
1) Key vaults - a strong addition to keep secure info (API keys, etc)
2) Power BI reporting to visualize (obviously)
3) Fabric Capacity Metrics to monitor overloading of the cluster
4) Capacity estimator to understand the needed resources (link: https://www.microsoft.com/en-us/microsoft-fabric/capacity-estimator)
5) Some people use VSCode successfully for their convenience to run notebooks
Delta tables for storing data are the default option. But most of the others are supported as well (parquet, json, jsonl, csv, etc).
Feel free to ask more specific questionsHi raginijaiswar19 ,
There are multiple tools you could use and it all depends on the use case. Microsoft itself has documented it nicely on the decision guide. Kindly refer the document
Guide for choosing pipeline/dataflow etc:
https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-pipeline-dataflow-spark
Guide for integration and orchestration:
Guide for data movement/integration:
Guide to choose data store:
https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-data-store
Note: Consider deployment options as well while designing your data platform.
Regards,
Srisakthi
5 Replies
Hi raginijaiswar19 ,
There are multiple tools you could use and it all depends on the use case. Microsoft itself has documented it nicely on the decision guide. Kindly refer the document
Guide for choosing pipeline/dataflow etc:
https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-pipeline-dataflow-spark
Guide for integration and orchestration:
Guide for data movement/integration:
Guide to choose data store:
https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-data-store
Note: Consider deployment options as well while designing your data platform.
Regards,
Srisakthi
Hello raginijaiswar19
Here are the tools Microsoft Fabric offers for data engineering—covering pipelines, transformations, and storage.
Data Ingestion & Orchestration
Pipelines (Data Factory Gen2)
- Enables orchestration workflows with activities like Copy Data, Dataflow Gen2, Lookup, ForEach, If Condition, etc. These are akin to Azure Data Factory and support both batch and streaming ETL.
- Pipelines let you sequence dataflows, notebooks, stored procedures, and copy tasks in a dependable pipeline.
Dataflow Gen2
- A low-code, Power Query-like interface with 300+ transformations for cleaning and shaping data.
- Ideal for transforming data before it’s stored in your lakehouse.
Compute & Transformation
Apache Spark Jobs
- Define Spark job definitions to process structured or unstructured data at scale—supports batch and streaming.
- Includes autoscale, library management, mentors like Spark advisor, and integration with Livy API for programmatic control.
Notebooks
- Use interactive notebooks with Python, Scala, or T-SQL for ingestion, transformation, and analysis.
- Fully integrated with VS Code, Git, and deployment pipelines.
Storage & Data Modeling
Lakehouse on OneLake
- Combines lake (raw/unstructured) and warehouse (curated) data in Delta tables, all stored in OneLake.
- Supports the Medallion architecture (Bronze/Silver/Gold) for progressive transformation.
Materialized Lake Views
- Precompute and store results of complex queries to accelerate analytics and implement medallion patterns.
Fabric Warehouse (SQL Pools)
- A SQL-based analytics store optimized for BI/reporting workloads and tightly integrated with lakehouse data.
Developer Experience & Tools
VS Code Extension & Spark Environments
- Author notebooks and Spark job definitions directly in VS Code, manage environments, deploy via Git.
Livy API
- Programmatic interface to submit Spark jobs for custom automation use cases.
GraphQL API
- Expose lakehouse data via a GraphQL endpoint for application integration.
Copilot for Data Engineering
- AI-powered assistant to generate Spark/T-SQL code, detect transformation errors, and automate pipeline creation via natural language.
Integration & Governance
Synapse Data Engineering integration
- Offers a unified experience for Spark-based engineering, tightly aligned with OneLake and lakehouse architecture.
Seamless Azure Synapse Integration
- Enables hybrid architectures using Synapse for MPP workloads and Fabric for lakehouse, pipelines, notebooks, BI, and governance.
Hope this helps, please appreciate giving a Kudos or accepting as a Solution!
I would focus on Data Pipelines (former Data Factory) and Spark Notebooks as a core.
Data pipelines are great for the following purposes:
1) Scheduling notebooks running
2) Ingesting tables as is (e.g. copy SQL Server table to data lake)
3) If people prefer a graphical interface over code (but disclaimer - it may create a massive tech debt, because it is much harder to see hidden complex relations in the visual code)
Spark notebooks (PySpark + Spark SQL) are a go-to option for me for the following cases:
1) bronze->silver layer
2) API ingestion
3) complex transformation in any place
I would mention Spark Notebooks separately with SQL code. They are great for building a gold layer on top of a clean, enriched layer. They are easy to read and operate. But the precondition is a clean and friendly silver layer.
What else should I mention here?
1) Key vaults - a strong addition to keep secure info (API keys, etc)
2) Power BI reporting to visualize (obviously)
3) Fabric Capacity Metrics to monitor overloading of the cluster
4) Capacity estimator to understand the needed resources (link: https://www.microsoft.com/en-us/microsoft-fabric/capacity-estimator)
5) Some people use VSCode successfully for their convenience to run notebooks
Delta tables for storing data are the default option. But most of the others are supported as well (parquet, json, jsonl, csv, etc).
Feel free to ask more specific questionsThank you for reaching out to the Microsoft Fabric Forum Community.
deborshi_nag 4iurchenko Srisakthi Thanks for the inputs.
The information provided by users was helpful. If you still have questions, please don't hesitate to reach out to the community.
Hope everything’s going smoothly on your end. I wanted to check if the issue got sorted. if you have any other issues please reach community.
