Data Science topics

How Are You Monitoring Model Drift and ML Performance in Microsoft Fabric?

codeautomation — Thu, 05 Nov 2026 09:43:13 GMT

Hi everyone,

I'm exploring how data science teams monitor machine learning models after deploying them in Microsoft Fabric, particularly when predictions are used in business-critical applications.

Training and evaluating a model is only part of the challenge. Once deployed, changes in customer behavior, incoming data, or business conditions can gradually reduce prediction accuracy.

Microsoft Fabric provides MLflow-based experiment tracking, model management, and batch and real-time scoring capabilities. However, I'm interested in how teams build reliable monitoring and retraining workflows around these components.

A few questions for the community:

Data and Model Drift: How are you detecting changes in input data distributions and model performance over time?
MLflow Integration: Are you using MLflow metrics and experiment tracking alongside custom monitoring tables or dashboards?
Automated Retraining: What conditions do you use to trigger model retraining? Scheduled intervals, performance thresholds, or detected data drift?
Production Validation: How do you evaluate a newly trained model before promoting it to production?
Monitoring and Alerts: Are you using Fabric notebooks, pipelines, and Power BI dashboards to track model performance and notify teams when intervention is needed?

I'm particularly interested in practical implementations involving predictive analytics, customer churn prediction, demand forecasting, or anomaly detection.

For teams running ML models in production, what's been more effective: scheduled retraining or drift-triggered retraining with human approval?

Would love to hear your experiences, recommended tools, and lessons learned.

MLflow in Fabric notebooks: how do you structure experiments and model promotion across workspaces?

jubinsoni — Wed, 04 Nov 2026 19:04:43 GMT

Hi everyone,

We are standardizing how data scientists track experiments in Fabric notebooks. The basic pattern we use for logging a run looks like this:

import mlflow mlflow.set_experiment("churn-model") with mlflow.start_run(run_name="lgbm-baseline"): mlflow.log_param("learning_rate", 0.05) mlflow.log_metric("auc", auc) mlflow.sklearn.log_model(model, "model")

Open questions

Naming and structure: one experiment per use case, per team, or per model version?
Promotion: how do you move a registered model from a dev workspace to test and prod without retraining?
Governance: who is allowed to register models, and how do you document the data version a model was trained on?

If you have a working convention or a lightweight review process, I would love to hear how you run it. Thanks!

How Can Businesses Turn Data Science Insights into Automated Business Decisions?

codeautomation — Tue, 03 Nov 2026 19:30:11 GMT

Data science helps organizations identify patterns, predict outcomes, and make better decisions. However, one challenge many businesses face is turning analytical insights into actions that improve everyday operations.

A predictive model may identify potential customer churn, forecast demand, or detect unusual activity, but how can organizations automatically act on these insights?

I'd like to hear how the community approaches this challenge.

A few questions:

How do you integrate machine learning models with existing business applications and workflows?
What tools do you use to trigger automated actions based on model predictions?
How do you maintain human oversight when automating decisions?
What strategies help monitor model performance and detect prediction drift?
Have you used Microsoft Fabric, Azure Machine Learning, Power Automate, or custom APIs to connect analytics with operational processes?

I'm particularly interested in practical implementations rather than just model development.

How is your organization bridging the gap between data science and business automation?

Scaling Machine Learning Workloads in Microsoft Fabric: How Do You Optimize Performance and Cost?

binitafulpagare — Tue, 03 Nov 2026 17:02:34 GMT

Building a machine learning model on a small dataset is relatively straightforward, but scaling the same workflow to millions of records introduces challenges related to performance, memory, compute resources, and cost.

Microsoft Fabric provides Spark notebooks, Lakehouse, pipelines, and MLflow to support machine learning workflows. However, optimizing large-scale ML workloads requires careful architectural decisions.

I'd like to hear how the community approaches the following challenges:

1. **Large Datasets:** How do you efficiently process millions or billions of records using Fabric Spark notebooks?

2. **Performance Optimization:** When would you use Spark MLlib instead of Python libraries such as Scikit-learn or XGBoost?

3. **Memory Management:** What strategies do you use to avoid out-of-memory errors during feature engineering and model training?

4. **Cost Optimization:** How do you balance Fabric capacity consumption with model training performance?

5. **Workflow Optimization:** How do you reduce unnecessary data processing and avoid repeatedly executing expensive notebook operations?

**My key question:** What architecture and optimization techniques would you recommend for building scalable, cost-efficient machine learning pipelines in Microsoft Fabric when working with large datasets?

I'd appreciate practical examples, performance optimization tips, and real-world experiences from the community.

How Do You Validate ML Models for Real-World Performance?

binitafulpagare — Tue, 03 Nov 2026 17:00:46 GMT

# Preventing Data Leakage in Microsoft Fabric: How Do You Validate ML Models for Real-World Performance?

Data leakage is one of the biggest challenges in building reliable machine learning models. A model may achieve excellent validation accuracy but perform poorly in production because of incorrect data splitting, future information leakage, or inconsistent feature engineering.

In Microsoft Fabric, we can use Lakehouse, notebooks, pipelines, and MLflow to build end-to-end machine learning workflows. However, ensuring reliable model validation across different stages can be challenging.

I'd like to understand how the community handles the following:

1. **Data Splitting:** How do you choose between random, stratified, and time-based splitting for different ML problems?

2. **Data Leakage Detection:** What techniques do you use to identify target leakage and data leakage during feature engineering?

3. **Cross-Validation:** How do you implement reliable cross-validation in Fabric notebooks, especially for large datasets?

4. **Model Validation:** How do you ensure that validation results represent real-world production performance?

5. **Experiment Tracking:** How do you use MLflow to maintain reproducibility across different experiments and validation runs?

**My key question:** What best practices would you recommend for designing a leakage-resistant ML validation pipeline in Microsoft Fabric, particularly for time-dependent datasets and imbalanced classification problems?

I'd appreciate practical examples and real-world experiences from the community.

How Do You Ensure Feature Consistency Between Training and Production?

binitafulpagare — Mon, 02 Nov 2026 05:21:29 GMT

When building machine learning models, one common challenge is maintaining consistency between the features used during model training and those used during production inference.

In Microsoft Fabric, we can use Lakehouse, notebooks, pipelines, and MLflow to build end-to-end ML workflows. However, managing reusable features across multiple models and ensuring that training and inference use the same feature logic can become challenging.

I'd like to hear how the community approaches the following:

1. **Feature Reusability:** How do you manage and reuse common features across multiple ML models in Fabric?

2. **Training vs. Inference:** How do you prevent training-serving skew and ensure identical feature transformations?

3. **Feature Versioning:** What is your approach to tracking feature definitions, source data versions, and transformation logic?

4. **Data Quality:** How do you handle missing values, outliers, and changing feature distributions?

5. **Architecture:** Would you recommend building a centralized feature management layer using Lakehouse tables and notebooks, or integrating an external feature store?

**My key question:** What is the most scalable and maintainable approach to implementing a feature-store-like architecture in Microsoft Fabric, especially when multiple models share the same features?

I'd appreciate real-world implementation examples and best practices from the community.

Production-Ready MLOps in Microsoft Fabric: Drift, Retraining & Governance

binitafulpagare — Mon, 02 Nov 2026 05:19:12 GMT

Hi Fabric Community,

I've been exploring Microsoft Fabric's Data Science capabilities and thinking about how organizations can move from experimental machine learning projects to fully automated, production-ready ML workflows.

Training a machine learning model is only one part of the journey. The real challenge begins after deployment, when incoming data changes, model performance starts degrading, and business requirements evolve.

I'd like to understand how experienced Fabric users are addressing these challenges.

1. Automated Model Drift Detection

Suppose we deploy a customer churn prediction model using MLflow in Microsoft Fabric.

After a few months, customer behavior changes and the model's prediction quality decreases.

- How are you detecting data drift and concept drift in Fabric?

- Are you implementing custom statistical tests using Python notebooks, or integrating external monitoring solutions?

- How do you distinguish between normal data fluctuations and actual model degradation?

2. Automated Model Retraining

When model performance drops below an acceptable threshold:

- Can we automate model retraining using Fabric Data Factory pipelines?

- How do you trigger retraining based on model performance or incoming data changes?

- Do you retrain models on a fixed schedule or only when drift is detected?

- How do you prevent a newly trained model from automatically replacing a better-performing production model?

3. Model Validation and Promotion

Before promoting a retrained model to Production:

- What validation criteria do you recommend?

- Do you use MLflow experiments to compare the new model against the current production model?

- How do you implement approval workflows for model promotion?

- Is there a recommended approach for maintaining rollback capability?

4. Model Monitoring and Explainability

For business-critical applications, prediction accuracy alone may not be sufficient.

- How are you monitoring Precision, Recall, F1-score, and other business-specific metrics after deployment?

- Are you integrating SHAP or other explainability techniques into Fabric workflows?

- How do you maintain model lineage and explain individual predictions for audit purposes?

5. Architecture and Cost Optimization

Finally, I'm interested in understanding how organizations design scalable MLOps architectures in Fabric.

- Would you keep model training, monitoring, and batch inference in separate workspaces?

- How do you manage Fabric capacity consumption when automated retraining runs alongside regular analytics workloads?

- When would you consider integrating Azure Machine Learning instead of using Fabric Data Science alone?

My Key Question

Can Microsoft Fabric support a complete automated MLOps lifecycle—from data drift detection and model retraining to model validation, approval, deployment, and rollback—without requiring extensive external infrastructure?**

I'd particularly appreciate insights from people who have implemented these workflows in real-world production environments.

Looking forward to learning from your experiences, architectural recommendations, and lessons learned!

Best regards,

Binita Fulpagare

Fabric ML Model Endpoint returns 401 from application but works in Postman with the same token

vkeda29 — Thu, 29 Oct 2026 16:05:27 GMT

Hi,

I'm currently integrating the Fabric ML Model endpoint with a .NET application and encountering a 401 Unauthorized error when calling the scoring API.

The endpoint is activated and working as expected when tested directly through Postman. However, when the application calls the same Fabric scoring API, it returns HTTP 401.

As far as we've verified, the same Bearer token and authentication details are being used.

Endpoint pattern:

POST /v1/workspaces/{workspaceId}/mlmodels/{modelId}/endpoint/score

Error response:

{ "errorCode": "RequestFailed", "message": "Unable to process the request", "isRetriable": false }

Additional details from the application logs:

HTTP status: 401 Unauthorized
Authorization header present on final request: True
WWW-Authenticate header: Empty
Postman request: Successful
Application request: Fails with 401

The error response doesn't provide any additional information to identify the exact cause.

I'm trying to understand whether there are any specific authentication, authorization, endpoint configuration, or HTTP client requirements when calling Fabric ML Model endpoints from an external application.

Has anyone encountered a similar issue where the same token works in Postman but fails when used in the application?

Are there any additional headers, permissions, redirect behaviors, or diagnostic steps that we should check to identify the cause? Any suggestions?

Thanks!

Clinical and IoT data on OneLake: Fabric data science or Azure ML for predictive models?

kiara-mehra35 — Mon, 26 Oct 2026 13:30:54 GMT

Hi all,

While working on a Fabric healthcare analytics platform at DataToBiz, we made some design choices on clinical, patient monitoring and IoT data that I’d like to compare with how people here would handle them.

The sources were SQL Server, SharePoint, clinical systems, ERP/CRM systems, APIs, files and IoT devices, which had been feeding legacy BI tools through manual ETL with reporting delays of 2 to 3 days. We pulled everything into OneLake with Data Factory pipelines, then used Data Factory and Synapse for transformation, validation and preparation. For ML, we used Synapse Data Science and Azure ML for predictive modeling on patient outcomes, resource forecasting and anomaly detection. Power BI handles reporting with automated refreshes and row-level security, Power Automate covers workflows and alerts, and Purview provides lineage and auditing, with role-based access on OneLake to meet HIPAA and GDPR requirements.

It works, but a few decisions are still open for me.

On the ML side, we ended up using both Synapse Data Science and Azure ML. If you were starting now, would you do everything in Fabric’s data science experience, or keep Azure ML for deployment and monitoring? Where do you draw the line, especially for models that regulators or auditors might ask about?

For the IoT data and anomaly detection, we processed it in batches. Would you move that to Eventstreams and an Eventhouse for near real-time detection, or is batch good enough for most monitoring use cases? What made you decide?

For sensitive data, how do you split security between OneLake access controls, the Warehouse or Lakehouse, and RLS in the semantic model? I’m particularly interested in how you prove access and lineage to an auditor in practice, whether Purview and the audit logs are enough or you need to build more.

And for the move from SQL Server and the legacy BI tools, did you use mirroring or Data Factory copy pipelines? I’d like to hear how you handled the cutover without disrupting reporting.

If you’ve built something similar, or tried an approach and dropped it, I’d really like to know what worked and what didn’t.

Thanks, and I’d love to hear how you’ve handled it.

Promoting MLflow models from Dev to Prod in Fabric: handling versions and batch scoring

jubinsoni — Fri, 16 Oct 2026 17:18:00 GMT

Hi everyone,

I'm setting up an ML workflow in Fabric and would like to hear how others are handling model promotion across environments before we settle on a pattern.

Current setup

Dev workspace -> notebooks for feature engineering + training (MLflow autolog) -> model registered as an ML model item, a few versions so far Prod workspace -> nightly pipeline -> notebook -> batch scoring into a Delta table -> Power BI report reads the scored table

The scoring notebook currently looks roughly like this:

import mlflow from synapse.ml.predict import MLFlowTransformer model = MLFlowTransformer( inputCols=feature_cols, outputCol="churn_score", modelName="churn-model", modelVersion=7 # <- hard-coded, this is the part I don't like ) scored = model.transform(spark.read.table("silver.customer_features")) scored.write.mode("overwrite").saveAsTable("gold.customer_churn_scores")

Where I'm unsure

1. Moving models between workspaces. Do you re-train in each workspace from the same code, or move the trained model version itself (e.g. via deployment pipelines / Git, or by logging to a shared registry)? Re-training feels safer for reproducibility, but it means Prod may not score with exactly the model we validated in Dev.

2. Referencing the "current production" model. Hard-coding the version number in the scoring notebook means a code change for every promotion. Is anyone using MLflow aliases/tags, or a small config table that the pipeline reads, to point at the approved version?

3. Feature consistency. Training features are built in a Dev notebook and scoring features in a Prod notebook. How are you keeping those in sync? Shared notebook/library in an Environment, a feature table in the lakehouse, something else?

4. Rollback and lineage. If a new version performs worse, what's your rollback path, and how do you record which model version produced which scores (e.g. a model_version column on the output table)?

Not looking for one "right" answer. Mostly curious what has worked (or not) for teams running this in production. Thanks in advance!

How does the Fabric handle the real-time ML inference requests?

vkeda29 — Thu, 15 Oct 2026 06:39:27 GMT

Regarding the scalability and the performance of Fabric ML endpoints.

For the use case, we're planning to deploy the XGBoost/LightGBM model in Fabric and consume the prediction endpoint through the application.

So, can the API support the bulk predictions. For Ex: we may send 10 records in a single request, but we could also have multiple users calling the same endpoint simultaneously.

I understand that Fabric manages the underlying compute and supports the autoscaling, but wanted to clarify few things:

How does Fabric handle concurrent requests to the same ML endpoints?
Is the Load balancing managed automatically?
Are there any concurrency, throughput, or request-size limits we should consider?
How does the scaling behave when the number of concurrent request increases, and what impact does this have on the response time and capacity consumption?
Are there any recommended approaches for load testing or configuring the endpoint for the production applications?
Also our main concern is ensuring that the prediction service doesn't affect the existing performance of the application

Has any one tested the Fabric ML endpoints with concurrent requests or used them for any similar application integration?

How does Fabric handle concurrent real-time ML inference requests?

vkeda29 — Thu, 15 Oct 2026 06:25:46 GMT

## Regarding the scalability and performance of Fabric ML model endpoints.

For our Use case, we're planning to deploy an XGBoost/LightGBM model in Fabric and consume the prediction endpoint through the application.

Can the API support bulk predictions. For example, we may send 10 records in a single request, but we could also have multiple users calling the same endpoint simultaneously.

I understand Fabric manages the underlying compute and supports automatic scaling, but I wanted to clarify a few things:

How does Fabric handle concurrent requests to the same ML endpoint? Is the load balancing managed automatically?

Are there any concurrency, throughput, or request-size limits we should consider?

How does the scaling behave when the number of concurrent requests increases, and what impact does this have on response time and capacity consumption?

Are there any recommended approaches for load testing or configuring the endpoint for production applications?

Our main concern is ensuring that the prediction service doesn't affect the existing performance of the application.

Has anyone tested Fabric ML endpoints with concurrent requests or used them for a similar application integration?

Microsoft Fabric ML Model Endpoint remains "Activating" indefinitely after activation request

fabric_user16 — Mon, 12 Oct 2026 15:12:36 GMT

Hi,

I'm using Microsoft Fabric ML Model Endpoints as part of an automated DEV → UAT promotion pipeline.

Scenario

The pipeline performs the following steps:

Download model artifacts
Validate SHA256
Register model versions in UAT
Create/register UAT model versions successfully
Activate the matching UAT model endpoint version

Registration completes successfully.

e.g result

{

"endpoint_http_status": 202,

"endpoint_action_requested": true

}

Problem

POST /activate

→ HTTP 202 Accepted

the endpoint remins in

Activating state forever

Questions

Is there a known issue where Fabric ML Model Endpoints can remain in Activating indefinitely after a successful activation request?
Is there a recommended maximum time for an endpoint to remain in Activating before it should be considered failed?
Is there a supported way to determine why endpoint provisioning is stuck?
Should an endpoint that remains in Activating be:
- left alone,
- deactivated and reactivated,
- or recreated entirely?
Are there diagnostic logs available for endpoint provisioning failures?

Any guidance from the Fabric product team or community would be greatly appreciated.

Thank you.

How are teams using AI agents to automate data science workflows?

codeautomation — Wed, 07 Oct 2026 21:21:28 GMT

I am exploring how AI agents can support modern data science workflows by automating repetitive tasks and improving collaboration between data teams.

Some potential use cases:

Automating data exploration and preprocessing steps
Generating insights from datasets using natural language queries
Assisting with feature engineering and model evaluation
Triggering ML pipelines based on business events
Monitoring model performance and identifying data quality issues

I would like to understand how data science teams are combining Microsoft Fabric, notebooks, ML workflows, and AI agents in real-world projects.

What approaches, tools, or architecture patterns are you using for AI-assisted data science workflows?

How are teams using AI agents to automate data science workflows?

codeautomation — Mon, 28 Sep 2026 20:26:46 GMT

I am exploring how AI agents can support modern data science workflows by automating repetitive tasks and improving collaboration between data teams.

Some potential use cases:

Automating data exploration and preprocessing steps
Generating insights from datasets using natural language queries
Assisting with feature engineering and model evaluation
Triggering ML pipelines based on business events
Monitoring model performance and identifying data quality issues

I would like to understand how data science teams are combining Microsoft Fabric, notebooks, ML workflows, and AI agents in real-world projects.

What approaches, tools, or architecture patterns are you using for AI-assisted data science workflows?

Roadmap

Surya_27 — Sun, 20 Sep 2026 17:56:58 GMT

Roadmap for become a data analyst in 2026 or 2027 in pov of this AI era

Challenges in Deploying Machine Learning Models into Production Workflows

codeautomation — Sat, 19 Sep 2026 02:30:54 GMT

Hi everyone,

I am exploring how teams are moving machine learning models from experimentation into reliable production workflows.

In many projects, the challenge is not only training the model but also managing the complete lifecycle:

- Preparing and maintaining quality datasets

- Tracking experiments and model versions

- Deploying models for real-world usage

- Monitoring performance after deployment

- Handling model updates over time

I would like to understand how the community is approaching this with Microsoft Fabric.

A few questions:

What are the recommended patterns for managing ML model lifecycle in Fabric?

How are teams handling model versioning and experiment tracking?

Are you using Fabric notebooks, MLflow, or external platforms for managing production ML workflows?

What challenges have you faced when moving data science projects from development to production?

Would love to hear practical experiences and approaches from the community.

FABRIC REST API Connections - Create Connection

Charline_74 — Wed, 16 Sep 2026 14:29:45 GMT

Does anyone know how to run this API via a PowerShell script to create a SQL Server cloud connection for a semantic model using Service Principal authentication?

Here's what I have so far:

Invoke-PowerBIRestMethod -Url "https://api.fabric.microsoft.com/v1/connections" -Method Post -Body $body

But I don't understand how to construct the body

Thanks in Advance,

Charline

Certificate validation error between Fabric and Huggingface

pborah — Wed, 26 Aug 2026 17:25:07 GMT

Hi,

I'm tyring to download a couple of transformer/sentence-transformer model weights into a notebook and I keep running into certificate mismatch issues between fabric and huggingface. I was wondering if anyone else has ran into this issue or if this is a known issue that Microsoft plans on handling in the very near future?

Environment

Microsoft Fabric Notebook (PySpark)

Python 3.11

PyTorch 2.2.1

Canada tenant

sentence-transformers==2.7.0 successfully installed

What works

Package installation succeeds

%pip install sentence-transformers==2.7.0

Import succeeds

from sentence_transformers import SentenceTransformer

version check succeeds

import torch import sentence_transformers print(torch.__version__) print(sentence_transformers.__version__)

What fails

from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2")

Observations

The notebook appears able to reach Hugging Face and download metadata files. For example, the following files are successfully cached:

config.json config_sentence_transformers.json modules.json README.md sentence_bert_config.json

However, the model weight file never downloads. The Hugging Face cache contains *.incomplete with a file size of 0 MB.

Error

The relevant error message appears to be

"No CA certificates were loaded from the system"

Is downloading Hugging Face model weights from Fabric notebooks currently supported? If so, is there a known issue with certificate validation? Is there a recommended workaround?

If not. then is there a recommended approach for generating semantic text embeddings within Fabric notebooks utilizing some kind of ANN based algorithm instead of classic techniques such as TF-IDF or SVD followed by some type of clustering?

The goal of this notebook is semantic clustering of approximately 500 short "type" descriptions for taxonomy development.

Thanks!

Data science roadmap

powerbidev123 — Sun, 02 Aug 2026 13:37:35 GMT

OneLake is often highlighted as one of the key capabilities of Microsoft Fabric, especially for centralizing and managing enterprise data. I'm interested in hearing from those who are using it in production environments.

For those with hands-on experience:

What challenges have you encountered while implementing or managing OneLake?
Have you experienced any performance or scalability issues as your data volumes increased?
How do you structure and organize data across workspaces, domains, or business units in a large enterprise?
What governance, security, and access control practices have worked well for your organization?
Are there any lessons learned or best practices you wish you had known before adopting OneLake?

I'd love to hear about real-world experiences, whether positive or challenging. Your insights could be incredibly valuable for others planning or expanding their Microsoft Fabric adoption.