Scaling Machine Learning Workloads in Microsoft Fabric: How Do You Optimize Performance and Cost?
Building a machine learning model on a small dataset is relatively straightforward, but scaling the same workflow to millions of records introduces challenges related to performance, memory, compute resources, and cost. Microsoft Fabric provides Spark notebooks, Lakehouse, pipelines, and MLflow to support machine learning workflows. However, optimizing large-scale ML workloads requires careful architectural decisions. I'd like to hear how the community approaches the following challenges: 1. **Large Datasets:** How do you efficiently process millions or billions of records using Fabric Spark notebooks? 2. **Performance Optimization:** When would you use Spark MLlib instead of Python libraries such as Scikit-learn or XGBoost? 3. **Memory Management:** What strategies do you use to avoid out-of-memory errors during feature engineering and model training? 4. **Cost Optimization:** How do you balance Fabric capacity consumption with model training performance? 5. **Workflow Optimization:** How do you reduce unnecessary data processing and avoid repeatedly executing expensive notebook operations? **My key question:** What architecture and optimization techniques would you recommend for building scalable, cost-efficient machine learning pipelines in Microsoft Fabric when working with large datasets? I'd appreciate practical examples, performance optimization tips, and real-world experiences from the community.
Fabric ML Model Endpoint returns 401 from application but works in Postman with the same token
Hi, I'm currently integrating the Fabric ML Model endpoint with a .NET application and encountering a 401 Unauthorized error when calling the scoring API. The endpoint is activated and working as expected when tested directly through Postman. However, when the application calls the same Fabric scoring API, it returns HTTP 401. As far as we've verified, the same Bearer token and authentication details are being used. Endpoint pattern: POST /v1/workspaces/{workspaceId}/mlmodels/{modelId}/endpoint/score Error response: { "errorCode": "RequestFailed", "message": "Unable to process the request", "isRetriable": false } Additional details from the application logs: HTTP status: 401 Unauthorized Authorization header present on final request: True WWW-Authenticate header: Empty Postman request: Successful Application request: Fails with 401 The error response doesn't provide any additional information to identify the exact cause. I'm trying to understand whether there are any specific authentication, authorization, endpoint configuration, or HTTP client requirements when calling Fabric ML Model endpoints from an external application. Has anyone encountered a similar issue where the same token works in Postman but fails when used in the application? Are there any additional headers, permissions, redirect behaviors, or diagnostic steps that we should check to identify the cause? Any suggestions? Thanks!
How Are You Monitoring Model Drift and ML Performance in Microsoft Fabric?
Hi everyone, I'm exploring how data science teams monitor machine learning models after deploying them in Microsoft Fabric, particularly when predictions are used in business-critical applications. Training and evaluating a model is only part of the challenge. Once deployed, changes in customer behavior, incoming data, or business conditions can gradually reduce prediction accuracy. Microsoft Fabric provides MLflow-based experiment tracking, model management, and batch and real-time scoring capabilities. However, I'm interested in how teams build reliable monitoring and retraining workflows around these components. A few questions for the community: Data and Model Drift: How are you detecting changes in input data distributions and model performance over time? MLflow Integration: Are you using MLflow metrics and experiment tracking alongside custom monitoring tables or dashboards? Automated Retraining: What conditions do you use to trigger model retraining? Scheduled intervals, performance thresholds, or detected data drift? Production Validation: How do you evaluate a newly trained model before promoting it to production? Monitoring and Alerts: Are you using Fabric notebooks, pipelines, and Power BI dashboards to track model performance and notify teams when intervention is needed? I'm particularly interested in practical implementations involving predictive analytics, customer churn prediction, demand forecasting, or anomaly detection. For teams running ML models in production, what's been more effective: scheduled retraining or drift-triggered retraining with human approval? Would love to hear your experiences, recommended tools, and lessons learned.How Can Businesses Turn Data Science Insights into Automated Business Decisions?
Data science helps organizations identify patterns, predict outcomes, and make better decisions. However, one challenge many businesses face is turning analytical insights into actions that improve everyday operations. A predictive model may identify potential customer churn, forecast demand, or detect unusual activity, but how can organizations automatically act on these insights? I'd like to hear how the community approaches this challenge. A few questions: How do you integrate machine learning models with existing business applications and workflows? What tools do you use to trigger automated actions based on model predictions? How do you maintain human oversight when automating decisions? What strategies help monitor model performance and detect prediction drift? Have you used Microsoft Fabric, Azure Machine Learning, Power Automate, or custom APIs to connect analytics with operational processes? I'm particularly interested in practical implementations rather than just model development. How is your organization bridging the gap between data science and business automation?How Do You Validate ML Models for Real-World Performance?
# Preventing Data Leakage in Microsoft Fabric: How Do You Validate ML Models for Real-World Performance? Data leakage is one of the biggest challenges in building reliable machine learning models. A model may achieve excellent validation accuracy but perform poorly in production because of incorrect data splitting, future information leakage, or inconsistent feature engineering. In Microsoft Fabric, we can use Lakehouse, notebooks, pipelines, and MLflow to build end-to-end machine learning workflows. However, ensuring reliable model validation across different stages can be challenging. I'd like to understand how the community handles the following: 1. **Data Splitting:** How do you choose between random, stratified, and time-based splitting for different ML problems? 2. **Data Leakage Detection:** What techniques do you use to identify target leakage and data leakage during feature engineering? 3. **Cross-Validation:** How do you implement reliable cross-validation in Fabric notebooks, especially for large datasets? 4. **Model Validation:** How do you ensure that validation results represent real-world production performance? 5. **Experiment Tracking:** How do you use MLflow to maintain reproducibility across different experiments and validation runs? **My key question:** What best practices would you recommend for designing a leakage-resistant ML validation pipeline in Microsoft Fabric, particularly for time-dependent datasets and imbalanced classification problems? I'd appreciate practical examples and real-world experiences from the community.How Do You Ensure Feature Consistency Between Training and Production?
When building machine learning models, one common challenge is maintaining consistency between the features used during model training and those used during production inference. In Microsoft Fabric, we can use Lakehouse, notebooks, pipelines, and MLflow to build end-to-end ML workflows. However, managing reusable features across multiple models and ensuring that training and inference use the same feature logic can become challenging. I'd like to hear how the community approaches the following: 1. **Feature Reusability:** How do you manage and reuse common features across multiple ML models in Fabric? 2. **Training vs. Inference:** How do you prevent training-serving skew and ensure identical feature transformations? 3. **Feature Versioning:** What is your approach to tracking feature definitions, source data versions, and transformation logic? 4. **Data Quality:** How do you handle missing values, outliers, and changing feature distributions? 5. **Architecture:** Would you recommend building a centralized feature management layer using Lakehouse tables and notebooks, or integrating an external feature store? **My key question:** What is the most scalable and maintainable approach to implementing a feature-store-like architecture in Microsoft Fabric, especially when multiple models share the same features? I'd appreciate real-world implementation examples and best practices from the community.
Show "last refreshed" and "data freshness" status on every Power BI visual, with no DAX workaround
PROBLEM: Report viewers often can't tell whether the numbers they're looking at are current. Today, report authors have to build a workaround: a hidden table with NOW() or a refresh-time query in Power Query a card visual with a custom measure manual placement on every page Those workarounds are easy to get wrong. NOW() shows the time the report was opened, not the time the data was refreshed. A visual built on a DirectQuery or Direct Lake source can also be fresher or staler than the rest of the page. SOLUTION: Add a built-in, no-code option to the report canvas: A "Data freshness" card or badge in the Visualizations pane that reads the last successful refresh time from the semantic model automatically. A per-visual toggle in Format > General, "Show data freshness", which shows a small timestamp tooltip or footer on that visual. Threshold alerts. The author sets a limit, such as "older than 24 hours", and the badge turns amber or red when the data goes stale. Support for all storage modes: Import, DirectQuery, Direct Lake and composite models, so each source shows its own freshness. Add a native "Data freshness" visual and per-visual toggle that automatically shows the last successful refresh time of the semantic model, with optional stale-data warnings, so authors no longer need DAX or Power Query workarounds.The Book of a Paper Trader: 355 trades, one honest question. Is there an edge?
ALPACA · PAPER ACCOUNT · 27 AUG – 30 OCT 2026 Most trading dashboards show you a P&L number and stop. This report keeps asking why. Over ten weeks an automated bot won +$23,859, and still closed the book at ▼ −$4,711. The story is about where that money went, which part of the system actually works, and what is sitting on the desk right now. Profit factor 0.84 · win rate 50.1% · expectancy ▼ −$13.27 per trade · account equity $95,819 The story in five pages Cover · What am I looking at? Three headline numbers, one candle per trading day, and a “how to read” strip for R-multiple, profit factor and why PAPER matters. The Book · Is the account growing? No. Stop exits alone erase 54% of all gains. September was the bleed (▼ −$6,828), October stopped it (▼ −$13). Only crypto pays (▲ +$631, PF 1.14). The Edge · Where do wins and losses come from? Shorts carry the book: ▲ +$4,727 at 62% win vs longs ▼ −$9,438 at 44%. 40 trades slipped past −1R (▼ −$8.2K), while trailing stops made ▲ +$2,845 at 81% win. The Desk · What is open right now? 74% of equity sits in cash ($71.3K). One name (KO) is 39% of exposure. 631 orders: 43.9% filled, 52% cancelled by design. The Ledger · Let me explore it myself. Every strategy by asset class with trades, a net P&L heat bar, win rate, avg R, profit factor and an EDGE / WATCH / CUT verdict. Storytelling choices Titles state the finding, not the chart type. “From $23.9K won to $4.7K lost”, “Shorts carry the book”, “One name carries the gain”. Every title is a DAX measure, so it rewrites itself when the data changes. One question per page, in the order a trader would ask it: am I growing → why → what is at risk now → let me dig. Honest framing. A PAPER badge on every page, the 15 crypto pairs with no price data are called out instead of hidden, and the cover explains every metric before you see it. Markers inside the chart instead of legends: PEAK / LOW chips on the equity curve, a win-rate chip on every strategy, “+N smaller” for treemap tiles too small to label. How it is built Microsoft Fabric end to end. Notebooks pull Alpaca (paper) and Binance data into a lakehouse and move it bronze → silver → gold as Delta tables. Two semantic models. A Direct Lake model for daily work, plus an Import copy for Publish to web (Direct Lake can't be published publicly). 146 DAX measures, zero custom visuals. Every chart (equity curve with drawdown, waterfall, lollipop, R-multiple dot plot, treemap, order-flow sankey, cover candles) is drawn in DAX as SVG or HTML. It is fully data-driven and loads as plain images. One design system. A single token set (colour, type scale, radius, 8px spacing) generates the DAX, a tokens.css file and the Power BI theme JSON, so native and custom visuals look identical. Serif display titles, sans labels, tabular figures. Navigation and accessibility. A dark rail with an active pill, made clickable by an invisible vertical Page navigator. Gain/loss colours pass WCAG AA, ▲▼ arrows mean colour is never the only signal, and alt-text measures are written for the main visuals. Explore ▶ Live report: embedded above (start on the cover and use the left rail) ↗ GitHub (notebooks, all DAX measures, theme and design tokens): github.com/PatricRc/trading-analytics-fabric ↗ LinkedIn: linkedin.com/in/patriciorioscanepa · Portfolio: data-pat-ai.netlify.app Paper trading only. Simulated fills, no real money, not financial advice. Built by Patricio Rios with Microsoft Fabric + Power BI.Production-Ready MLOps in Microsoft Fabric: Drift, Retraining & Governance
Hi Fabric Community, I've been exploring Microsoft Fabric's Data Science capabilities and thinking about how organizations can move from experimental machine learning projects to fully automated, production-ready ML workflows. Training a machine learning model is only one part of the journey. The real challenge begins after deployment, when incoming data changes, model performance starts degrading, and business requirements evolve. I'd like to understand how experienced Fabric users are addressing these challenges. 1. Automated Model Drift Detection Suppose we deploy a customer churn prediction model using MLflow in Microsoft Fabric. After a few months, customer behavior changes and the model's prediction quality decreases. - How are you detecting data drift and concept drift in Fabric? - Are you implementing custom statistical tests using Python notebooks, or integrating external monitoring solutions? - How do you distinguish between normal data fluctuations and actual model degradation? 2. Automated Model Retraining When model performance drops below an acceptable threshold: - Can we automate model retraining using Fabric Data Factory pipelines? - How do you trigger retraining based on model performance or incoming data changes? - Do you retrain models on a fixed schedule or only when drift is detected? - How do you prevent a newly trained model from automatically replacing a better-performing production model? 3. Model Validation and Promotion Before promoting a retrained model to Production: - What validation criteria do you recommend? - Do you use MLflow experiments to compare the new model against the current production model? - How do you implement approval workflows for model promotion? - Is there a recommended approach for maintaining rollback capability? 4. Model Monitoring and Explainability For business-critical applications, prediction accuracy alone may not be sufficient. - How are you monitoring Precision, Recall, F1-score, and other business-specific metrics after deployment? - Are you integrating SHAP or other explainability techniques into Fabric workflows? - How do you maintain model lineage and explain individual predictions for audit purposes? 5. Architecture and Cost Optimization Finally, I'm interested in understanding how organizations design scalable MLOps architectures in Fabric. - Would you keep model training, monitoring, and batch inference in separate workspaces? - How do you manage Fabric capacity consumption when automated retraining runs alongside regular analytics workloads? - When would you consider integrating Azure Machine Learning instead of using Fabric Data Science alone? My Key Question Can Microsoft Fabric support a complete automated MLOps lifecycle—from data drift detection and model retraining to model validation, approval, deployment, and rollback—without requiring extensive external infrastructure?** I'd particularly appreciate insights from people who have implemented these workflows in real-world production environments. Looking forward to learning from your experiences, architectural recommendations, and lessons learned! Best regards, Binita FulpagareHow does the Fabric handle the real-time ML inference requests?
Regarding the scalability and the performance of Fabric ML endpoints. For the use case, we're planning to deploy the XGBoost/LightGBM model in Fabric and consume the prediction endpoint through the application. So, can the API support the bulk predictions. For Ex: we may send 10 records in a single request, but we could also have multiple users calling the same endpoint simultaneously. I understand that Fabric manages the underlying compute and supports the autoscaling, but wanted to clarify few things: How does Fabric handle concurrent requests to the same ML endpoints? Is the Load balancing managed automatically? Are there any concurrency, throughput, or request-size limits we should consider? How does the scaling behave when the number of concurrent request increases, and what impact does this have on the response time and capacity consumption? Are there any recommended approaches for load testing or configuring the endpoint for the production applications? Also our main concern is ensuring that the prediction service doesn't affect the existing performance of the application Has any one tested the Fabric ML endpoints with concurrent requests or used them for any similar application integration?
