Forum Discussion
Machine learning pipelines in Microsoft Fabric
hsn367 wrote:
I have a background in building machine learning pipelines in AzureML using AzureML SDK, this really helps us in orchestrating the end to end data science workflows. In workflows in our organization, we have ML pipelines written using AzureML SDK and then we have CI/CD pipelines that are supposed to publish these ML pipelines to AzureML studio.
Now moving into Fabric with this background, I have a couple of questions that I did not get answers to when going through the documentations.
1) How to orchestrate the data science workflow in Fabric. For instance we have multiple scripts for our end-to-end solution, we can easily build pipelines over it using AzureML SDK in AzureML studio but in fabric what is the alternative, how are we suppose to build ML pipelines?
2) Data drift monitoring in an important component of end-to-end data science solution, we can monitor drift of the model's data in AzureML but what is the alternative available in Fabric?
hsn367
An additional reply from the internal team for the above questions
We have pipelines in Fabric in the form of Data Factory, and you can run Notebooks with ML activities/code as part of those. Overall, we are working on strengthening our MLOps story. We have Model endpoints in PrPr and working on providing a better SDK. If you look for running MLOps in Production today, we recommend using AzureML with Fabric. AzureML has access to data in OneLake and working on improving that integration. Over time Fabric will become more complete on MLOps too, for data centric and analytics workloads. We focus on scenarios where you serve data to PowerBI today. And we are evolving into other scenarios gradually, like real time model endpoints for example.
We don't yet have drift monitoring in Fabric. On the roadmap.AnonymousThank you so much for all the support.
Fabric Notebooks as Steps: Convert your standalone python scripts (.py) into individual Fabric Notebooks (e.g., Notebook 1 for Data Prep, Notebook 2 for Model Training).Fabric Data Pipelines for Orchestration: Open the Data Factory experience and create a new Data Pipeline. Use the drag-and-drop canvas to add Notebook Activities for each step of your process.Control Flow & Dependencies: Chain your notebooks together using conditional connectors (On Success, On Failure). This mimics the Directed Acyclic Graph (DAG) structure of AzureML.Parameterization: Pass dynamic inputs (like file paths or hyperparameter values) across execution steps by defining parameters within your pipeline and notebooks.Experiment Tracking: Use the built-in MLflow integration inside your notebooks to log metrics, parameters, and register models.2) Data Drift Monitoring in FabricMicrosoft Fabric does not feature an out-of-the-box, no-code data drift monitoring dashboard identical to the automated dataset monitors in Azure ML Studio. Instead, drift monitoring is implemented programmatically:Open-Source Libraries: Implement your data drift detection logic inside a Fabric Notebook using open-source Python frameworks like Evidently AI, Great Expectations, or whylogs.Delta Tables Base: Store your baseline training data and new production inference data as Delta tables in your central Fabric Lakehouse or OneLake.Comparison Script: Write a notebook that reads both datasets from the Lakehouse, computes drift statistics (e.g., Population Stability Index or Chi-Square tests), and saves the results.Automation and Alerting: Schedule this evaluation notebook using a Fabric Data Pipeline to run periodically (e.g., daily). You can configure a Web activity or an Office 365 alert activity within the pipeline to send notifications if drift thresholds are breached.