Forum Discussion
Machine learning pipelines in Microsoft Fabric
Hi hsn367
Thanks for using Fabric Community.
Transitioning from AzureML to Microsoft Fabric involves adapting to the tools and services that Fabric offers for machine learning and data science workflows.
Orchestrating Data Science Workflows in Fabric:
1) In Fabric, you can use Fabric notebooks for data science scenarios, which allow you to ingest data into a Fabric lakehouse using Apache Spark, load existing data from delta tables, and clean and transform data using Apache Spark and Python-based tools.
2) You can create experiments and runs to train different machine learning models within these notebooks.
3) For orchestrating workflows, you can construct data analytics workflows with Fabric Data Factory data pipelines, which provide a low-code solution for data integration and ETL projects.
4) The Data Factory in Fabric allows you to build automated workflows that combine different artifacts in your workspace, such as files, notebooks, and dataflows, to create an end-to-end data analytics workflow.
Please refer to these links:
Data science tutorial - get started - Microsoft Fabric | Microsoft Learn
Construct a data analytics workflow with a Fabric Data Factory data pipeline | Microsoft Fabric Blog | Microsoft Fabric
Data Drift Monitoring in Fabric:
1) Fabric doesn't have a built-in data drift monitoring tool like AzureML. However, you can leverage various options for drift detection
2) Monitoring in Fabric is centralized through the Monitoring hub, which enables users to monitor Fabric activities, including data pipelines, dataflows, lakehouses, notebooks, and semantic models.
3) While specific features for data drift monitoring like those in AzureML may not be directly mentioned, the Monitoring hub provides a comprehensive view of all activities and could be used to track changes and performance over time.
Use the Monitoring hub - Microsoft Fabric | Microsoft Learn
Hope this helps. Please let me know if you have any further questions.