Forum Discussion
ML Deployment
Hi nbleonhard,
Thank you for sharing your real-world experience. It's interesting to see how moving from a local laptop-based workflow to Microsoft Fabric has improved your machine learning process.
I particularly liked your point about having a dedicated Lakehouse for each ML project. Separating ML workflows from data ingestion while storing reference tables and prediction history in Delta tables seems like a practical approach that also makes collaboration with BI teams much easier.
I have one question based on your implementation. As the number of ML projects grows, how do you manage shared datasets and reusable features across multiple Lakehouses? Do you maintain a centralized feature repository or rely on OneLake shortcuts to avoid data duplication?
Thanks again for sharing your experience—it's always valuable to learn from production implementations rather than just theoretical best practices.
Before Microsoft Fabric, our entire machine learning operation ran on my laptop. One of the big advantages of moving to Fabric was that each ML project got access to its own lakehouse, which now plays a central role in our entire ML lifecycle.
Here are a few of the biggest benefits we've seen:
- Decoupled workflows: It allowed us to decouple our machine learning pipeline from the data ingestion process.
- Reference tables: Setting up reference tables is super convenient. For example, I use a delta table as a crosswalk to map our raw feature names to user-friendly names.
- BI accessibility: Storing our current predictions (as well as past predictions) directly on the lakehouse is convenient for our BI team.
Hi nbleonhard,
Thank you for sharing your real-world experience. It's interesting to see how moving from a local laptop-based workflow to Microsoft Fabric has improved your machine learning process.
I particularly liked your point about having a dedicated Lakehouse for each ML project. Separating ML workflows from data ingestion while storing reference tables and prediction history in Delta tables seems like a practical approach that also makes collaboration with BI teams much easier.
I have one question based on your implementation. As the number of ML projects grows, how do you manage shared datasets and reusable features across multiple Lakehouses? Do you maintain a centralized feature repository or rely on OneLake shortcuts to avoid data duplication?
Thanks again for sharing your experience—it's always valuable to learn from production implementations rather than just theoretical best practices.