Forum Discussion
Kudo KingpinHow Do You Ensure Feature Consistency Between Training and Production?
When building machine learning models, one common challenge is maintaining consistency between the features used during model training and those used during production inference.
In Microsoft Fabric, we can use Lakehouse, notebooks, pipelines, and MLflow to build end-to-end ML workflows. However, managing reusable features across multiple models and ensuring that training and inference use the same feature logic can become challenging.
I'd like to hear how the community approaches the following:
1. **Feature Reusability:** How do you manage and reuse common features across multiple ML models in Fabric?
2. **Training vs. Inference:** How do you prevent training-serving skew and ensure identical feature transformations?
3. **Feature Versioning:** What is your approach to tracking feature definitions, source data versions, and transformation logic?
4. **Data Quality:** How do you handle missing values, outliers, and changing feature distributions?
5. **Architecture:** Would you recommend building a centralized feature management layer using Lakehouse tables and notebooks, or integrating an external feature store?
**My key question:** What is the most scalable and maintainable approach to implementing a feature-store-like architecture in Microsoft Fabric, especially when multiple models share the same features?
I'd appreciate real-world implementation examples and best practices from the community.
6 Replies
v-aatheequeCommunity Support
A good approach to maintaining feature consistency between training and production is to centralize feature engineering logic and reuse it across the ML lifecycle.
In Microsoft Fabric, you can consider the following architecture:
- Store reusable, curated features in Lakehouse Delta tables so multiple ML models can consume common features.
- Implement shared feature engineering logic using Python functions or notebooks and reuse the same logic during training and inference to minimize training-serving skew.
- Maintain feature definitions, transformation code versions, source data versions and processing timestamps for reproducibility.
- Include validation checks for missing values, duplicates, outliers and changes in feature distributions.
- Use Fabric's MLflow integration to track experiments, parameters, metrics and registered model versions.
- Use Fabric pipelines and notebooks to automate feature preparation, model training and batch scoring.
For multiple models sharing common features, starting with a centralized Lakehouse-based feature layer can be a practical and maintainable solution. If you have strict low-latency online inference requirements or need advanced feature serving capabilities, you may also evaluate an external feature store.
Official documentation:
- binitafulpagare
# Feature Store in Microsoft Fabric: How Do You Ensure Feature Consistency Between Training and Production?
Thank you for sharing this detailed explanation!
I agree that a centralized Lakehouse-based feature layer with reusable Python functions is a practical starting point for maintaining feature consistency across training and inference.
One question I have is about feature updates and versioning: If multiple ML models depend on shared features and the feature engineering logic changes, how would you manage backward compatibility and ensure that existing production models continue using the correct feature versions?
Would you recommend maintaining separate versioned feature tables or managing feature transformations through reusable, version-controlled code?
I'd be interested to hear your thoughts on this approach.
- benibix
Resolver I
A few concrete patterns that make the Lakehouse approach work in practice:
One feature package, imported by both sides. Put the feature functions in a small Python package, build it as a .whl and attach it to a Fabric Environment that both the training and the scoring notebooks use. Copying the same code into two notebooks is where skew usually starts. Shared code in one versioned library can't drift.
Put the fitted preprocessing inside the model. Scaling, encoding and imputation values learned in training (means, category lists, bins) belong in the model artifact, for example as a scikit-learn Pipeline logged with MLflow. Then scoring can't recompute them on today's data by mistake. Only the stateless feature logic needs the shared package.
Point-in-time features for training. Keep feature tables keyed by entity and an as-of timestamp. When building the training set, join each label to the feature values as they were at the label's time, not the latest ones. Otherwise the model trains on information it won't have at scoring time, and it looks better in testing than it is in production.
Versioning with what Delta already gives you. Log the Delta version of each feature table used for training (from DESCRIBE HISTORY) as an MLflow tag, along with the feature package version. Any model can then be traced to the exact data and code, and time travel can rebuild its training set.
A skew check on every scoring run. Compare each feature's distribution in the scoring batch with the training snapshot (PSI, or just null rate, mean and the share of unseen categories), write the result to a Delta table, and alert above a threshold. Most "the model got worse" cases show up there first.
For batch scoring, this Lakehouse feature layer is usually enough. An external feature store earns its cost mainly when you need online, low-latency lookups.
- binitafulpagare
Thank you for sharing these practical implementation patterns!
I especially liked the approach of using a shared, versioned Python package and logging Delta table versions in MLflow. The point-in-time feature strategy is also important for preventing data leakage during model training.
I have one follow-up question: When a feature distribution changes significantly in production, how would you decide whether to trigger model retraining or simply investigate the data pipeline?
Would you rely on PSI thresholds, model performance metrics, or a combination of both?
I'd love to hear how you would implement this monitoring and decision-making process in Microsoft Fabric.
- benibix
I check in this order, because the pipeline is cheaper to rule out than a retrain:
- Look at the inputs first. A jump on one date in null rate, row count or the category list, or one feature moving while related ones stay flat, is almost always a load problem (a unit change, a new default value, a join that dropped rows). Fix the pipeline, don't retrain.
- If the inputs look clean, check the model metric on the batches whose labels have matured. Drift with stable AUC or recall gets logged and watched, nothing more.
- Retrain when the metric drops for two or three matured batches in a row, or, if labels arrive too slowly, when PSI goes past 0.25 on the features the model leans on most.
In Fabric I write all three checks to the same monitoring table, and the notebook returns one verdict for the pipeline's If Condition.
If that settles the question, feel free to mark it as the solution.
