Forum Discussion
Machine learning pipelines in Microsoft Fabric
hsn367 wrote:
I have a background in building machine learning pipelines in AzureML using AzureML SDK, this really helps us in orchestrating the end to end data science workflows. In workflows in our organization, we have ML pipelines written using AzureML SDK and then we have CI/CD pipelines that are supposed to publish these ML pipelines to AzureML studio.
Now moving into Fabric with this background, I have a couple of questions that I did not get answers to when going through the documentations.
1) How to orchestrate the data science workflow in Fabric. For instance we have multiple scripts for our end-to-end solution, we can easily build pipelines over it using AzureML SDK in AzureML studio but in fabric what is the alternative, how are we suppose to build ML pipelines?
2) Data drift monitoring in an important component of end-to-end data science solution, we can monitor drift of the model's data in AzureML but what is the alternative available in Fabric?
hsn367
An additional reply from the internal team for the above questions
We have pipelines in Fabric in the form of Data Factory, and you can run Notebooks with ML activities/code as part of those. Overall, we are working on strengthening our MLOps story. We have Model endpoints in PrPr and working on providing a better SDK. If you look for running MLOps in Production today, we recommend using AzureML with Fabric. AzureML has access to data in OneLake and working on improving that integration. Over time Fabric will become more complete on MLOps too, for data centric and analytics workloads. We focus on scenarios where you serve data to PowerBI today. And we are evolving into other scenarios gradually, like real time model endpoints for example.
We don't yet have drift monitoring in Fabric. On the roadmap.AnonymousThank you so much for all the support.
1. Data Ingestion (Fabric Data Factory)
This is the starting point of your pipeline. You need to pull data from various sources (SQL databases, APIs, cloud storage, etc.) into the Fabric ecosystem.
Fabric Data Pipelines: These are used to orchestrate the movement of data. You can perform Copy activities to move data or use Dataflows Gen2 to perform low-code data transformation during the ingestion phase.
Result: The data is landed in OneLake, acting as a centralized "data lake" for the entire organization.
2. Data Storage & Management (OneLake & Lakehouse)
In Fabric, you don't need to create separate storage accounts.
OneLake: Every workspace in Fabric is connected to OneLake. It is built on top of ADLS Gen2 (Azure Data Lake Storage) and uses the open Delta-Parquet format.
Lakehouse: This is the primary storage structure for ML projects. It provides a structured file system (for raw files) and a SQL analytics endpoint (for querying data like a database).
3. Data Exploration & Preparation (Synapse Data Science Notebooks)
Once the data is in the Lakehouse, you need to prepare it for modeling.
Notebooks: You use Python (PySpark/Pandas) or Spark SQL in Notebooks to perform Exploratory Data Analysis (EDA), handle missing values, engineer features, and normalize data.
Integration: Since the Notebooks are tightly integrated with OneLake, you can access your data directly without complex connection strings.
4. Model Training & Experimentation (MLflow)
Fabric natively integrates MLflow to track your machine learning lifecycle.
Experiments: As you train different versions of your models, you can log parameters, metrics, and environment configurations.
AutoML: If you want to accelerate the process, Fabric provides AutoML capabilities that automatically iterate through various algorithms and hyperparameters to find the best model for your data.
5. Model Registration (Model Registry)
Once you have trained a successful model, you save it to the Fabric Model Registry.
Versioning: The registry allows you to version your models (e.g., v1, v2), track their status (e.g., Staging, Production), and easily retrieve the best model for deployment.
6. Deployment & Scoring (Real-time Scoring & SQL)
The final step is making your model accessible to applications or end-users.
PREDICT function: You can use the PREDICT SQL function within your Lakehouse or Data Warehouse to run batch scoring on new data directly using SQL queries.
Real-time Scoring: You can deploy models as a web service to generate predictions in real-time, allowing applications to send an input and receive a
prediction instantly.