Forum Discussion
Helper IMachine learning pipelines in Microsoft Fabric
Helper IHi Anonymous
Thank you so much for the detailed response. So here is what I got from your response.
1) AzureML pipelines alternative available in Fabric is Fabric Data Factory pipelines where we can orchestrate multiple python scripts of our data science solutions.
2) Data drift monitoring is not available in Fabric yet. You mentioned monitoring hub but that does not fulfil the needs of drift monitoring.
I have a couple of more questions regarding migrating to Fabric coming from AzureML background.
1) In AzureML, we were heavily relying on AzureML SDK's data asset management for data versioning of our data science solution or to version the data produced by the different components of the pipeline. And it was very easy to just use the latest version of the data or to use any previous version it was just a matter of specifying that version name. So migrating to Fabric how you think we can get the similar behavior there.
2) In our current workflows, we have three AML workspaces i.e. a separate one for development, test and prod environments. Now we develop the ML pipelines in dev workspace and then deploy them to test and prod workspaces via CI/CD pipelines. So is it possible to achieve the same behavior in Fabric?
Hi hsn367
Data Versioning in Fabric:
Microsoft Fabric does not have the dataset concept as in Azure Data Factory
While Fabric’s approach to data versioning may differ from AzureML SDK’s data asset management, you can achieve similar behavior by leveraging OneLake and the data integration pipelines within Fabric.
You can use Notebooks and also Azure Devops to acheive version control in Fabric.
https://radacad.com/version-control-in-power-bi-and-fabric
https://www.linkedin.com/pulse/unraveling-past-empowering-future-versioning-timetravel-data/
CI/CD Pipeline Deployment Across Workspaces in Fabric:
Fabric’s lifecycle management tools, including Git integration and deployment pipelines, support a standardized system for collaboration and continuous delivery of updated content into production. Deployment pipelines in Fabric allow you to clone content from one stage to another, typically from development to test, and from test to production, maintaining the connections between copied items. You can have similar 3 workspaces in Fabric and achieve the same.
Introduction to the CI/CD process as part of the ALM cycle in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
The Microsoft Fabric deployment pipelines process - Microsoft Fabric | Microsoft Learn
Hope this helps. Please let me know if you have any further questions.