What tools are used for Data Engineering in Fabric?

• Pipelines (Data Factory Gen2)

  • Enables orchestration workflows with activities like Copy Data, Dataflow Gen2, Lookup, ForEach, If Condition, etc. These are akin to Azure Data Factory and support both batch and streaming ETL. 
  • Pipelines let you sequence dataflows, notebooks, stored procedures, and copy tasks in a dependable pipeline. 

• Dataflow Gen2

  • A low-code, Power Query-like interface with 300+ transformations for cleaning and shaping data. 
  • Ideal for transforming data before it’s stored in your lakehouse.

Compute & Transformation

• Apache Spark Jobs

  • Define Spark job definitions to process structured or unstructured data at scale—supports batch and streaming. 
  • Includes autoscale, library management, mentors like Spark advisor, and integration with Livy API for programmatic control. 

• Notebooks

  • Use interactive notebooks with Python, Scala, or T-SQL for ingestion, transformation, and analysis. 
  • Fully integrated with VS Code, Git, and deployment pipelines.

Storage & Data Modeling

• Lakehouse on OneLake

  • Combines lake (raw/unstructured) and warehouse (curated) data in Delta tables, all stored in OneLake. 
  • Supports the Medallion architecture (Bronze/Silver/Gold) for progressive transformation. 

• Materialized Lake Views

  • Precompute and store results of complex queries to accelerate analytics and implement medallion patterns. 

• Fabric Warehouse (SQL Pools)

  • A SQL-based analytics store optimized for BI/reporting workloads and tightly integrated with lakehouse data. 

Developer Experience & Tools

• VS Code Extension & Spark Environments

  • Author notebooks and Spark job definitions directly in VS Code, manage environments, deploy via Git. 

• Livy API

  • Programmatic interface to submit Spark jobs for custom automation use cases. 

• GraphQL API

  • Expose lakehouse data via a GraphQL endpoint for application integration. 

• Copilot for Data Engineering

  • AI-powered assistant to generate Spark/T-SQL code, detect transformation errors, and automate pipeline creation via natural language. 

Integration & Governance

• Synapse Data Engineering integration

  • Offers a unified experience for Spark-based engineering, tightly aligned with OneLake and lakehouse architecture. 

• Seamless Azure Synapse Integration

  • Enables hybrid architectures using Synapse for MPP workloads and Fabric for lakehouse, pipelines, notebooks, BI, and governance.

Hope this helps, please appreciate giving a Kudos or accepting as a Solution!