Forecasting With Confidence using Prophet for Data Science in Microsoft Fabric
In this edition, we’re exploring how Prophet actually thinks about time. By the end, you should have a much clearer sense of how trends, seasonality, and real-world disruptions fit together, and why that framing often matters more than complex math. We will also explore interpretation, because getting numbers is easy, but knowing what they really mean is where the real value shows up.
How are teams using AI agents to automate data science workflows?
I am exploring how AI agents can support modern data science workflows by automating repetitive tasks and improving collaboration between data teams. Some potential use cases: Automating data exploration and preprocessing steps Generating insights from datasets using natural language queries Assisting with feature engineering and model evaluation Triggering ML pipelines based on business events Monitoring model performance and identifying data quality issues I would like to understand how data science teams are combining Microsoft Fabric, notebooks, ML workflows, and AI agents in real-world projects. What approaches, tools, or architecture patterns are you using for AI-assisted data science workflows?Challenges in Deploying Machine Learning Models into Production Workflows
Hi everyone, I am exploring how teams are moving machine learning models from experimentation into reliable production workflows. In many projects, the challenge is not only training the model but also managing the complete lifecycle: - Preparing and maintaining quality datasets - Tracking experiments and model versions - Deploying models for real-world usage - Monitoring performance after deployment - Handling model updates over time I would like to understand how the community is approaching this with Microsoft Fabric. A few questions: What are the recommended patterns for managing ML model lifecycle in Fabric? How are teams handling model versioning and experiment tracking? Are you using Fabric notebooks, MLflow, or external platforms for managing production ML workflows? What challenges have you faced when moving data science projects from development to production? Would love to hear practical experiences and approaches from the community.Sequence Modeling for Data Science in Microsoft Fabric
In this edition, we're exploring how sequence modeling changes the way one thinks about data, why treating rows as independent often leads to shallow insights, and how RNNs and GRUs step in to model memory. By the end, you should have a clear intuition for when simple recurrence is enough, when gating really matters, and how to start applying this mindset in Microsoft Fabric (without overengineering anything!)Forecasting with Autoregression, ARIMA & AIC / BIC for Data Science in Microsoft Fabric
In this edition, we’re exploring forecasting through Autoregression, ARIMA, and the model selection tools AIC and BIC. By the time you’re done reading, you’ll understand how data can actually learn from its own patterns, how ARIMA helps bring structure to unpredictable trends, and how AIC and BIC keep your models grounded by balancing accuracy with simplicity.Model Explainability That Goes Beyond the Notebook for Data Science in Microsoft Fabric
A machine learning model can be remarkably accurate and still leave people completely unconvinced. That is one of the strange realities of working in data science. You can spend weeks cleaning data, testing algorithms, tuning parameters, and validating performance, only to reach the moment when somebody looks at a prediction and asks a very reasonable question: “Why?” Suddenly, accuracy alone does not feel like enough. A probability score might tell you what the model believes, but it does not necessarily help the person receiving that prediction understand how the model arrived there. That gap between prediction and understanding is exactly where model explainability becomes important. What you will learn: In this edition, we’re exploring model explainability in production and how SHAP, FastAPI, and Plotly can work together to make predictions easier to understand. By the time you’re done with this, you’ll have a clear view of how SHAP can explain individual model decisions, how those explanations can be served dynamically through an API, and how interactive visualizations can turn technical attribution values into something far more approachable. Source: Sahir Maharaj (https://sahirmaharaj.com) It is tempting to think of explainability as a chart you generate after training a model, but production explainability is really an ongoing capability. Every new observation can produce a different prediction, and every prediction may have a different explanation. A customer predicted to leave a service might receive a high risk score because of declining engagement, while another customer receives the same score because of repeated service issues. The prediction may look identical at the surface, but the reasoning behind it can be completely different. That difference matters when somebody needs to decide what to do next. This is where SHAP becomes particularly valuable because it gives you a structured way to describe feature contributions. Imagine a model predicting whether a machine is likely to fail. The prediction itself might tell you that failure risk is 82 percent. Useful, certainly, but incomplete. SHAP can help reveal that rising operating temperature, vibration intensity, and unusually long operating hours are pushing the prediction upward, while another factor such as recent maintenance is reducing the predicted risk. Suddenly the prediction becomes much easier to reason about. You still have a probabilistic model, but you also have something resembling an explanation of its behavior. As a data scientist, I find that this distinction between global and local understanding is especially important. During model development, I often want to understand the model globally. Which variables matter most overall? Is the model relying heavily on variables I expected it to use? Are there surprising relationships hiding in the data? In production, however, people often care about something much more specific. They want to know why this customer, this transaction, this machine, or this observation received the prediction it did. That is a local explanation, and SHAP is particularly useful for helping bridge that gap. Source: Sahir Maharaj (https://sahirmaharaj.com) You can think about the difference through a credit-risk example. Globally, a model might rely heavily on payment history, debt ratios, and income stability. That tells you something useful about the overall model. But suppose one applicant receives a higher risk prediction than expected. A general feature importance chart does not really answer the person's question. You need to understand what happened for that particular prediction. Perhaps the debt ratio increased the risk score significantly, while long-term employment reduced it. That level of explanation provides much more context than simply showing which variables matter across thousands of predictions. Explainability can also become an important debugging tool for you as the person building the model. Suppose a model performs well according to the metrics you are monitoring, but its explanations repeatedly show that one unexpected variable dominates many predictions. That should make you curious. import numpy as np import pandas as pd import shap import matplotlib.pyplot as plt from sklearn.ensemble import RandomForestClassifier from sklearn.model_selection import train_test_split np.random.seed(42) n = 800 X = pd.DataFrame({ "usage_drop": np.random.normal(20, 10, n), "support_calls": np.random.poisson(3, n), "login_days": np.random.randint(1, 30, n), "tenure_months": np.random.randint(1, 72, n), "monthly_cost": np.random.normal(80, 20, n) }) score = ( 0.08 * X["usage_drop"] + 0.45 * X["support_calls"] - 0.05 * X["login_days"] - 0.02 * X["tenure_months"] + 0.015 * X["monthly_cost"] ) y = (score + np.random.normal(0, 1, n) > np.median(score)).astype(int) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=42 ) model = RandomForestClassifier(n_estimators=150, random_state=42) model.fit(X_train, y_train) explainer = shap.TreeExplainer(model) values = explainer(X_test) class_values = values[:, :, 1] if values.values.ndim == 3 else values shap.plots.beeswarm(class_values, max_display=5) plt.show() shap.plots.waterfall(class_values[0], max_display=5) plt.show() Imagine that somebody submits new information to a prediction service. The model receives those features, processes them, and generates a result. If explainability is part of the design, the same interaction can also produce SHAP information describing the important factors behind that specific prediction. Instead of storing explanations for every imaginable case beforehand, you generate them when they are needed. FastAPI fits naturally into this kind of workflow because it gives the prediction and explanation process a consistent interface. Rather than another application needing to understand your model directly, it can send the required information to an API. The API can coordinate the prediction process, calculate the explanation, and return the relevant output. The application consuming the result does not need to know the mathematical details of SHAP or how the model itself works internally. It simply receives information in an agreed structure. That separation can make a production system much easier to manage. From what I observe when building data science workflows, this separation becomes valuable surprisingly quickly. During early experimentation, everything often exists together. Data preparation, prediction, explanation, and visualization may all happen inside one environment. That is perfectly reasonable when you are exploring an idea. Production introduces more responsibilities. The application requesting a prediction should not need access to the entire modeling environment, and your model should not depend on somebody manually generating explanations. Creating a clear service boundary helps each part of the system do its job without becoming unnecessarily tangled with everything else. Dynamic explanations also give you flexibility over how much information you return. A technical monitoring application may want detailed SHAP values for every available feature. A decision-support interface may only need the five strongest contributors. Another application may need a short summary explaining the strongest positive and negative influences. The explanation engine can be the same while the presentation changes according to the audience. That matters because explainability is not only a modeling problem. It is also a communication problem. Source: Sahir Maharaj (https://sahirmaharaj.com) Performance deserves careful attention here as well. Calculating an explanation adds work beyond producing the prediction itself, and depending on the model, the amount of additional work may not be trivial. That does not mean explanations should be avoided. It means you should think about when they are required. Some systems may need an explanation for every single prediction. Others may generate explanations only when a user requests additional detail. You might also decide that certain high-impact decisions always receive explanations while low-impact automated predictions do not. Production design is often about finding that balance between completeness and responsiveness. There is also an important consistency issue. An explanation is only useful if it corresponds to the exact model and transformation process that generated the prediction. Imagine updating your model but accidentally continuing to generate explanations using an older version. The results could look perfectly reasonable while describing behavior that no longer matches the prediction being shown. import numpy as np import pandas as pd import shap import plotly.graph_objects as go from sklearn.ensemble import GradientBoostingClassifier np.random.seed(7) n = 700 X = pd.DataFrame({ "temperature": np.random.normal(70, 12, n), "vibration": np.random.normal(4, 1.5, n), "operating_hours": np.random.normal(900, 250, n), "maintenance_days": np.random.randint(1, 120, n) }) risk = ( 0.06 * X["temperature"] + 0.7 * X["vibration"] + 0.003 * X["operating_hours"] - 0.025 * X["maintenance_days"] ) y = (risk + np.random.normal(0, 1, n) > np.median(risk)).astype(int) model = GradientBoostingClassifier(random_state=7) model.fit(X, y) explainer = shap.Explainer(model, X) row = pd.DataFrame([{ "temperature": 92, "vibration": 6.8, "operating_hours": 1250, "maintenance_days": 75 }]) prediction = model.predict_proba(row)[0, 1] explanation = explainer(row) names = row.columns.tolist() impacts = explanation.values[0] waterfall = go.Figure(go.Waterfall( orientation="h", y=names, x=impacts, measure=["relative"] * len(names), text=[f"{x:+.2f}" for x in impacts], textposition="outside" )) waterfall.update_layout( title="SHAP Explanation: What Drove This Prediction?", xaxis_title="SHAP Impact on Model Output", yaxis_title="Feature", height=450 ) waterfall.show() gauge = go.Figure(go.Indicator( mode="gauge+number+delta", value=prediction * 100, number={"suffix": "%"}, delta={"reference": 50}, title={"text": "Predicted Machine Failure Risk"}, gauge={ "axis": {"range": [0, 100]}, "steps": [ {"range": [0, 40]}, {"range": [40, 70]}, {"range": [70, 100]} ], "threshold": { "line": {"width": 4}, "thickness": 0.8, "value": prediction * 100 } } )) gauge.update_layout(height=400) gauge.show() pd.DataFrame({ "Feature": names, "Value": row.iloc[0].values, "SHAP Impact": impacts }).sort_values("SHAP Impact", key=abs, ascending=False) The third objective is turning those explanation values into something people can interpret without needing to understand every detail of SHAP. This is where visualization becomes more than decoration. Raw values can tell you exactly how features influenced a prediction, but they require effort to interpret. A Plotly visualization can organise those contributions so the strongest influences immediately stand out. Features pushing a prediction in one direction can be separated visually from those pushing it in the opposite direction, allowing somebody to understand the basic story of the prediction within a few moments. Imagine a customer churn model that predicts an unusually high probability of cancellation. A dynamic explanation might show that a recent drop in product usage is the strongest contributor, followed by repeated support contacts and a reduction in login frequency. At the same time, a long customer relationship might slightly reduce the churn prediction. That is much more meaningful than presenting a table containing feature names and decimal values. The visualization helps the reader understand not only which features mattered but also how they worked together. Source: Sahir Maharaj (https://sahirmaharaj.com) When I look at explanations intended for other people, I try not to assume that more information automatically creates more transparency. It is very easy to generate a detailed visualization containing dozens of variables simply because the values are available. The result may technically be comprehensive while being practically overwhelming. If somebody has to study a chart for several minutes before understanding the main message, the explanation probably needs refinement. In many situations, highlighting the strongest contributors and allowing additional detail to be explored when needed creates a much better experience. Interactivity can make this even more useful. Plotly allows explanations to become something people can investigate instead of merely observe. A user might hover over a feature to inspect the contribution more closely or compare several predictions to understand why similar observations received different results. Imagine comparing two loan applications that received different risk scores. An interactive explanation can make it easier to see that one prediction was strongly influenced by debt levels while another was affected primarily by inconsistent payment history. That comparison helps reveal how the model responds to different combinations of information. At the same time, visual polish should never become a substitute for clarity. A sophisticated interactive chart can create an impression of authority, and that means you have a responsibility to provide enough context around what the visualization represents. The user should understand that the chart explains the model's behavior, not necessarily the underlying truth of the real-world situation. If a feature makes a large contribution, that means the model relied heavily on that feature for the prediction. It does not automatically prove that the feature caused the outcome. Keeping that distinction clear is a major part of responsible model communication. Source: Sahir Maharaj (https://sahirmaharaj.com) So give it a try. Do not wait until you have a huge production system or a complicated machine learning platform before thinking about transparency. Start with one prediction, one explanation, and one clear visualization. Build from there as your understanding grows. The more comfortable you become explaining what your model is doing, the easier it becomes to recognise when its behaviour deserves a closer look. And that ability is valuable whether you are just entering data science or have been building models for years. If you ever need anything while exploring it, just reach out to me. I’ll be more than happy to help. Thanks for taking the time to read my post! I’d love to hear what you think and connect with you! 🙂 LinkedIn Kaggle Topmate (Free Power BI / Data Science Sessions and Resources) Website The Tech Journal (Blog) About the author Sahir Maharaj is a Lead Data Scientist who leads the design and deployment of end-to-end AI solutions that drive strategic decisions at scale. As a Microsoft MVP, he has been featured internationally, including in The Indian Express and on New York's Times Square billboards, and is a prolific content creator on LinkedIn.Certificate validation error between Fabric and Huggingface
Hi, I'm tyring to download a couple of transformer/sentence-transformer model weights into a notebook and I keep running into certificate mismatch issues between fabric and huggingface. I was wondering if anyone else has ran into this issue or if this is a known issue that Microsoft plans on handling in the very near future? Environment Microsoft Fabric Notebook (PySpark) Python 3.11 PyTorch 2.2.1 Canada tenant sentence-transformers==2.7.0 successfully installed What works Package installation succeeds %pip install sentence-transformers==2.7.0 Import succeeds from sentence_transformers import SentenceTransformer version check succeeds import torch import sentence_transformers print(torch.__version__) print(sentence_transformers.__version__) What fails from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") Or Observations The notebook appears able to reach Hugging Face and download metadata files. For example, the following files are successfully cached: config.json config_sentence_transformers.json modules.json README.md sentence_bert_config.json However, the model weight file never downloads. The Hugging Face cache contains *.incomplete with a file size of 0 MB. Error The relevant error message appears to be "No CA certificates were loaded from the system" Is downloading Hugging Face model weights from Fabric notebooks currently supported? If so, is there a known issue with certificate validation? Is there a recommended workaround? If not. then is there a recommended approach for generating semantic text embeddings within Fabric notebooks utilizing some kind of ANN based algorithm instead of classic techniques such as TF-IDF or SVD followed by some type of clustering? The goal of this notebook is semantic clustering of approximately 500 short "type" descriptions for taxonomy development. Thanks!Mastering Advanced Regression for Data Science in Microsoft Fabric
In this edition, we’re exploring two regression techniques that every data professional eventually bumps into when the simple models stop telling the full story. You’ll get a clear sense of what quantile regression actually solves, especially when your data behaves in unpredictable or uneven ways. By the time you’re done, you’ll feel more confident choosing the regression approach that truly fits the question you’re trying to answer, instead of defaulting to whatever is familiar.Bayesian Thinking for Data Science in Microsoft Fabric
In this edition, we’re exploring Bayesian modeling and how to think about uncertainty in a more realistic, practical way using PyMC3. By the time you’re done, you’ll have a clear intuition for what Bayesian thinking really means, why it’s so useful in day-to-day data work, and how it changes the way you interpret results. We’ll also explore how PyMC3 supports this mindset in a structured but approachable way.
