topic Re: Machine learning pipelines in Microsoft Fabric in Data Science

Machine learning pipelines in Microsoft Fabric

hsn367 — Thu, 04 Apr 2024 11:51:08 GMT

I have a background in building machine learning pipelines in AzureML using AzureML SDK, this really helps us in orchestrating the end to end data science workflows. In workflows in our organization, we have ML pipelines written using AzureML SDK and then we have CI/CD pipelines that are supposed to publish these ML pipelines to AzureML studio.

Now moving into Fabric with this background, I have a couple of questions that I did not get answers to when going through the documentations.

1) How to orchestrate the data science workflow in Fabric. For instance we have multiple scripts for our end-to-end solution, we can easily build pipelines over it using AzureML SDK in AzureML studio but in fabric what is the alternative, how are we suppose to build ML pipelines?

2) Data drift monitoring in an important component of end-to-end data science solution, we can monitor drift of the model's data in AzureML but what is the alternative available in Fabric?

Re: Machine learning pipelines in Microsoft Fabric

Anonymous — Fri, 05 Apr 2024 05:08:45 GMT

Hi hsn367
Thanks for using Fabric Community.
Transitioning from AzureML to Microsoft Fabric involves adapting to the tools and services that Fabric offers for machine learning and data science workflows.

Orchestrating Data Science Workflows in Fabric:
1) In Fabric, you can use Fabric notebooks for data science scenarios, which allow you to ingest data into a Fabric lakehouse using Apache Spark, load existing data from delta tables, and clean and transform data using Apache Spark and Python-based tools.
2) You can create experiments and runs to train different machine learning models within these notebooks.
3) For orchestrating workflows, you can construct data analytics workflows with Fabric Data Factory data pipelines, which provide a low-code solution for data integration and ETL projects.
4) The Data Factory in Fabric allows you to build automated workflows that combine different artifacts in your workspace, such as files, notebooks, and dataflows, to create an end-to-end data analytics workflow.
Please refer to these links:
Data science tutorial - get started - Microsoft Fabric | Microsoft Learn
Construct a data analytics workflow with a Fabric Data Factory data pipeline | Microsoft Fabric Blog | Microsoft Fabric

Data Drift Monitoring in Fabric:
1) Fabric doesn't have a built-in data drift monitoring tool like AzureML. However, you can leverage various options for drift detection
2) Monitoring in Fabric is centralized through the Monitoring hub, which enables users to monitor Fabric activities, including data pipelines, dataflows, lakehouses, notebooks, and semantic models.
3) While specific features for data drift monitoring like those in AzureML may not be directly mentioned, the Monitoring hub provides a comprehensive view of all activities and could be used to track changes and performance over time.
Use the Monitoring hub - Microsoft Fabric | Microsoft Learn

Hope this helps. Please let me know if you have any further questions.

Re: Machine learning pipelines in Microsoft Fabric

hsn367 — Fri, 05 Apr 2024 06:48:19 GMT

Hi Anonymous

Thank you so much for the detailed response. So here is what I got from your response.

1) AzureML pipelines alternative available in Fabric is Fabric Data Factory pipelines where we can orchestrate multiple python scripts of our data science solutions.

2) Data drift monitoring is not available in Fabric yet. You mentioned monitoring hub but that does not fulfil the needs of drift monitoring.

I have a couple of more questions regarding migrating to Fabric coming from AzureML background.

1) In AzureML, we were heavily relying on AzureML SDK's data asset management for data versioning of our data science solution or to version the data produced by the different components of the pipeline. And it was very easy to just use the latest version of the data or to use any previous version it was just a matter of specifying that version name. So migrating to Fabric how you think we can get the similar behavior there.

2) In our current workflows, we have three AML workspaces i.e. a separate one for development, test and prod environments. Now we develop the ML pipelines in dev workspace and then deploy them to test and prod workspaces via CI/CD pipelines. So is it possible to achieve the same behavior in Fabric?

Re: Machine learning pipelines in Microsoft Fabric

Anonymous — Fri, 05 Apr 2024 10:15:32 GMT

Hi hsn367

Data Versioning in Fabric:
Microsoft Fabric does not have the dataset concept as in Azure Data Factory
While Fabric’s approach to data versioning may differ from AzureML SDK’s data asset management, you can achieve similar behavior by leveraging OneLake and the data integration pipelines within Fabric.
You can use Notebooks and also Azure Devops to acheive version control in Fabric.
https://radacad.com/version-control-in-power-bi-and-fabric
https://www.linkedin.com/pulse/unraveling-past-empowering-future-versioning-timetravel-data/

CI/CD Pipeline Deployment Across Workspaces in Fabric:
Fabric’s lifecycle management tools, including Git integration and deployment pipelines, support a standardized system for collaboration and continuous delivery of updated content into production. Deployment pipelines in Fabric allow you to clone content from one stage to another, typically from development to test, and from test to production, maintaining the connections between copied items. You can have similar 3 workspaces in Fabric and achieve the same.
Introduction to the CI/CD process as part of the ALM cycle in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
The Microsoft Fabric deployment pipelines process - Microsoft Fabric | Microsoft Learn

Hope this helps. Please let me know if you have any further questions.

Re: Machine learning pipelines in Microsoft Fabric

Anonymous — Mon, 08 Apr 2024 07:19:04 GMT

hsn367 wrote:

I have a background in building machine learning pipelines in AzureML using AzureML SDK, this really helps us in orchestrating the end to end data science workflows. In workflows in our organization, we have ML pipelines written using AzureML SDK and then we have CI/CD pipelines that are supposed to publish these ML pipelines to AzureML studio.

Now moving into Fabric with this background, I have a couple of questions that I did not get answers to when going through the documentations.

1) How to orchestrate the data science workflow in Fabric. For instance we have multiple scripts for our end-to-end solution, we can easily build pipelines over it using AzureML SDK in AzureML studio but in fabric what is the alternative, how are we suppose to build ML pipelines?

2) Data drift monitoring in an important component of end-to-end data science solution, we can monitor drift of the model's data in AzureML but what is the alternative available in Fabric?

hsn367
An additional reply from the internal team for the above questions

We have pipelines in Fabric in the form of Data Factory, and you can run Notebooks with ML activities/code as part of those. Overall, we are working on strengthening our MLOps story. We have Model endpoints in PrPr and working on providing a better SDK. If you look for running MLOps in Production today, we recommend using AzureML with Fabric. AzureML has access to data in OneLake and working on improving that integration. Over time Fabric will become more complete on MLOps too, for data centric and analytics workloads. We focus on scenarios where you serve data to PowerBI today. And we are evolving into other scenarios gradually, like real time model endpoints for example.

We don't yet have drift monitoring in Fabric. On the roadmap.

Re: Machine learning pipelines in Microsoft Fabric

hsn367 — Tue, 09 Apr 2024 08:43:19 GMT

AnonymousThank you so much for all the support.

Re: Machine learning pipelines in Microsoft Fabric

AlAmeenN — Thu, 04 Jun 2026 08:08:53 GMT

Hi, Since nearly two years have passed, I wanted to check whether the recommendation has changed.

Would you still recommend using Azure ML alongside Microsoft Fabric for production MLOps workloads, or has Fabric matured to support a full end-to-end MLOps lifecycle natively?

Re: Machine learning pipelines in Microsoft Fabric

carter_gray705 — Sat, 27 Jun 2026 06:53:47 GMT

Fabric doesn have a direct replacement for azureml sdk pipelines. Most teams orchestrate ML workflows using Fabric Data Pipelines with notebooks and Spark jobs. For data drift monitoring, there's no built-in equivalent to AzureML yet, so you'll need custom monitoring or continue using AzureML for MLOps. A hybrid Fabric + AzureML approach is still a common choice.

Re: Machine learning pipelines in Microsoft Fabric

Tarun_khudiya — Sat, 27 Jun 2026 07:05:30 GMT

Fabric Notebooks as Steps: Convert your standalone python scripts (.py) into individual Fabric Notebooks (e.g., Notebook 1 for Data Prep, Notebook 2 for Model Training).Fabric Data Pipelines for Orchestration: Open the Data Factory experience and create a new Data Pipeline. Use the drag-and-drop canvas to add Notebook Activities for each step of your process.Control Flow & Dependencies: Chain your notebooks together using conditional connectors (On Success, On Failure). This mimics the Directed Acyclic Graph (DAG) structure of AzureML.Parameterization: Pass dynamic inputs (like file paths or hyperparameter values) across execution steps by defining parameters within your pipeline and notebooks.Experiment Tracking: Use the built-in MLflow integration inside your notebooks to log metrics, parameters, and register models.2) Data Drift Monitoring in FabricMicrosoft Fabric does not feature an out-of-the-box, no-code data drift monitoring dashboard identical to the automated dataset monitors in Azure ML Studio. Instead, drift monitoring is implemented programmatically:Open-Source Libraries: Implement your data drift detection logic inside a Fabric Notebook using open-source Python frameworks like Evidently AI, Great Expectations, or whylogs.Delta Tables Base: Store your baseline training data and new production inference data as Delta tables in your central Fabric Lakehouse or OneLake.Comparison Script: Write a notebook that reads both datasets from the Lakehouse, computes drift statistics (e.g., Population Stability Index or Chi-Square tests), and saves the results.Automation and Alerting: Schedule this evaluation notebook using a Fabric Data Pipeline to run periodically (e.g., daily). You can configure a Web activity or an Office 365 alert activity within the pipeline to send notifications if drift thresholds are breached.

Re: Machine learning pipelines in Microsoft Fabric

Tarun_khudiya — Sat, 27 Jun 2026 07:18:09 GMT

1. Data Ingestion (Fabric Data Factory)

This is the starting point of your pipeline. You need to pull data from various sources (SQL databases, APIs, cloud storage, etc.) into the Fabric ecosystem.

Fabric Data Pipelines: These are used to orchestrate the movement of data. You can perform Copy activities to move data or use Dataflows Gen2 to perform low-code data transformation during the ingestion phase.

Result: The data is landed in OneLake, acting as a centralized "data lake" for the entire organization.

2. Data Storage & Management (OneLake & Lakehouse)

In Fabric, you don't need to create separate storage accounts.

OneLake: Every workspace in Fabric is connected to OneLake. It is built on top of ADLS Gen2 (Azure Data Lake Storage) and uses the open Delta-Parquet format.

Lakehouse: This is the primary storage structure for ML projects. It provides a structured file system (for raw files) and a SQL analytics endpoint (for querying data like a database).

3. Data Exploration & Preparation (Synapse Data Science Notebooks)

Once the data is in the Lakehouse, you need to prepare it for modeling.

Notebooks: You use Python (PySpark/Pandas) or Spark SQL in Notebooks to perform Exploratory Data Analysis (EDA), handle missing values, engineer features, and normalize data.

Integration: Since the Notebooks are tightly integrated with OneLake, you can access your data directly without complex connection strings.

4. Model Training & Experimentation (MLflow)

Fabric natively integrates MLflow to track your machine learning lifecycle.

Experiments: As you train different versions of your models, you can log parameters, metrics, and environment configurations.

AutoML: If you want to accelerate the process, Fabric provides AutoML capabilities that automatically iterate through various algorithms and hyperparameters to find the best model for your data.

5. Model Registration (Model Registry)

Once you have trained a successful model, you save it to the Fabric Model Registry.

Versioning: The registry allows you to version your models (e.g., v1, v2), track their status (e.g., Staging, Production), and easily retrieve the best model for deployment.

6. Deployment & Scoring (Real-time Scoring & SQL)

The final step is making your model accessible to applications or end-users.

PREDICT function: You can use the PREDICT SQL function within your Lakehouse or Data Warehouse to run batch scoring on new data directly using SQL queries.

Real-time Scoring: You can deploy models as a web service to generate predictions in real-time, allowing applications to send an input and receive a

prediction instantly.

Re: Machine learning pipelines in Microsoft Fabric

Tarun_khudiya — Fri, 03 Jul 2026 04:54:04 GMT

1. Orchestrating ML Workflows in FabricSince Fabric does not use the AzureML SDK for pipelines, you must reconstruct your workflow using the following integrated pieces:Fabric Data Pipelines: Use these to coordinate the sequence of your tasks. They replace the overall orchestration function of AzureML pipelines.Fabric Notebooks: Put your machine learning scripts inside PySpark or standard Python Notebooks.Activity Orchestration: Within a Data Pipeline, add a "Notebook Activity" to trigger your scripts in a specific order (e.g., Data Prep \(\rightarrow \) Feature Engineering \(\rightarrow \) Training \(\rightarrow \) Scoring).OneLake / Lakehouse: Use the built-in Microsoft Fabric Lakehouse to store your raw, clean, and processed datasets centrally.MLflow: Use Fabric's built-in MLflow integration to track your experiment runs, parameters, and model versions.2. Managing Data Drift MonitoringData drift means that the data entering your system today looks different from the data you used to train your model in the past. Fabric does not have an automatic button for this yet. You can handle it in two ways:The Code-Based Approach: Write a script inside a Fabric Notebook using open-source libraries like Evidently AI, Great Expectations, or Alibi. Run this notebook on a schedule using a Fabric Data Pipeline to compare your new production data against your baseline training data.The Hybrid Approach: Continue using Azure Machine Learning alongside Fabric. You can keep your data in Fabric's OneLake but route the model monitoring and governance workloads back to Azure Machine Learning to leverage its advanced MLOps tools.