Causal Inference for Data Science in Microsoft Fabric

[Sahir_Maharaj]

At some point in your data career, you realize that prediction alone stops being enough. You can predict churn, revenue, or conversion rates with impressive accuracy, yet the same uncomfortable question keeps coming back: what actually caused this change? I’ve seen dashboards that look perfect on the surface but fail the moment a stakeholder asks, “If we change this lever, will the outcome really change?” Correlation feels useful until it suddenly feels dangerous. This is where causal inference becomes one of the most valuable skills a data professional can develop. It’s not about being more complex for the sake of it, but about being more honest with the data. When you understand causality, your work shifts from reporting patterns to shaping decisions. And that shift changes how people trust your insights.

What you will learn: In this edition, we’re exploring into causal inference and why it matters once you move beyond basic reporting and prediction. By the time you’re done, you’ll have a clear understanding of how causal thinking differs from traditional analytics and how to reframe everyday business questions around cause and effect instead of simple correlation. And because insights only matter if they’re understood, we’ll look at how to communicate causal findings clearly and responsibly so decision-makers know what they can trust and act on.

Read Time: 8 minutes

Source: Sahir Maharaj (https://sahirmaharaj.com)

Early on, finding strong correlations feels like progress. You pull data, run an analysis, and suddenly patterns appear that weren’t obvious before. Lines slope upward, variables move together, and the results feel satisfying because they’re clean and defensible. For a while, that’s enough. But over time, especially once your work starts influencing decisions rather than reports, you begin to feel a quiet tension. Someone inevitably asks a question like, “If we change this, will the outcome actually change?” And in that moment, correlation starts to feel incomplete. That’s usually when causal thinking begins to matter, even if it isn’t named yet. Correlation is descriptive. It tells you what happened together in the past.

Causality is explanatory. It tries to answer why something happened and, more importantly, what might happen if a decision is made differently. That distinction sounds simple, but it has deep implications. Most real business questions aren’t about curiosity; they’re about action. Leaders don’t ask for trends because they enjoy patterns. They ask because they want to know which levers are safe to pull and which ones are risky. As soon as you start thinking in terms of cause and effect, your relationship with data changes. You stop looking only at what exists and start thinking about what could have existed. Causal reasoning introduces the idea of alternate realities, outcomes that never happened but still matter. What would revenue have looked like if a promotion had never launched? What would engagement have been without a redesign? These questions are uncomfortable because the data can’t show you those worlds directly.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(7)

n = 800

seasonality = np.linspace(0, 12, n)
marketing_spend = 50 + 10 * np.sin(seasonality) + np.random.normal(0, 3, n)
sales = 200 + 5 * np.sin(seasonality) + np.random.normal(0, 8, n)

df = pd.DataFrame({
    "MarketingSpend": marketing_spend,
    "Sales": sales,
    "Seasonality": seasonality
})

correlation = df["MarketingSpend"].corr(df["Sales"])

plt.figure(figsize=(10, 6))
plt.scatter(df["MarketingSpend"], df["Sales"], alpha=0.6)
plt.title(f"Strong Correlation Observed (r = {correlation:.2f})")
plt.xlabel("Marketing Spend")
plt.ylabel("Sales")
plt.grid(True)
plt.show()

plt.figure(figsize=(10, 4))
plt.plot(df["Seasonality"], df["MarketingSpend"], label="Marketing Spend")
plt.plot(df["Seasonality"], df["Sales"], label="Sales")
plt.title("Hidden Driver: Both Variables Follow Seasonality")
plt.xlabel("Time / Seasonality")
plt.legend()
plt.grid(True)
plt.show()

You only ever see one version of reality, and everything else has to be reasoned out carefully. This discomfort is where many analyses quietly overreach. A strong trend line starts to feel like proof. A before-and-after comparison looks convincing enough to move forward. But there are almost always other explanations lurking in the background. Timing, selection effects, seasonality, external events, internal priorities, all of these can produce the same patterns. Without acknowledging them, correlation turns into a story that feels confident but rests on shaky ground. From what I’ve seen working with teams, this is often where trust gets strained, not because the analysis was sloppy, but because expectations weren’t aligned. Stakeholders tend to hear causality even when it wasn’t explicitly claimed. They assume that because two things moved together, one must have caused the other. Recognizing that gap early changes how you communicate. You become more careful with language, more explicit about limits, and more thoughtful about what you’re actually saying.

Another realization that comes with experience is that causality isn’t something you prove once and move on from. It’s never absolute. Every causal claim lives within a set of assumptions, whether they’re acknowledged or not. Strong analysis doesn’t eliminate uncertainty. It manages it. It makes the conditions under which a conclusion holds clear, rather than hiding them behind polished visuals or confident phrasing. Once this way of thinking starts to settle in, it naturally affects how you ask questions. The difference between a predictive question and a causal one becomes more obvious. Asking what predicts an outcome is very different from asking whether changing something will cause that outcome to change. The first fits neatly into historical data. The second forces you to think about actions, timing, and alternatives all at once.

Source: Sahir Maharaj (https://sahirmaharaj.com)

This is where many conversations improve before any analysis even begins. When you restate a request in causal terms, gaps surface naturally. Sometimes the “change” isn’t clearly defined. Sometimes the outcome isn’t measurable in a meaningful way. Sometimes there’s no reasonable comparison group. These moments can feel like friction, but they’re actually clarity. They reveal whether the question is answerable at all, and if so, under what conditions. Every causal question quietly contains three elements. There is something that changes, something you care about, and something to compare against. That comparison is often the hardest part because it refers to a world that never existed. Yet without it, claims drift back toward correlation without anyone realizing it. Being disciplined about this structure keeps your reasoning grounded.

Timing also starts to matter more than it used to. Causes must come before effects. It sounds obvious, but in practice it’s easy to blur that line, especially when post-change metrics get mixed into explanations. Being strict about sequence can feel pedantic at first, but it protects you from circular reasoning later. It forces you to slow down and think about what could have influenced what, and when. This kind of framing does slow things down. You spend more time thinking before analyzing, which can feel uncomfortable in fast-paced environments. But that upfront investment pays off. Fewer reworks. Fewer awkward follow-ups. Fewer moments where a confident conclusion collapses under a simple question. Over time, people begin to trust not just your results, but your judgment.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(21)

n = 1000

experience = np.random.normal(5, 2, n)
training = (experience > 5).astype(int)

performance = (
    60
    + 6 * experience
    + 4 * training
    + np.random.normal(0, 5, n)
)

df = pd.DataFrame({
    "Experience": experience,
    "Training": training,
    "Performance": performance
})

group_means = df.groupby("Training")["Performance"].mean()

plt.figure(figsize=(8, 5))
group_means.plot(kind="bar")
plt.title("Naïve Comparison: Trained vs Untrained Performance")
plt.ylabel("Average Performance Score")
plt.xlabel("Training Status")
plt.grid(axis="y")
plt.show()

plt.figure(figsize=(10, 6))
plt.scatter(df["Experience"], df["Performance"], c=df["Training"], cmap="coolwarm", alpha=0.6)
plt.title("Performance vs Experience (Color = Training)")
plt.xlabel("Years of Experience")
plt.ylabel("Performance")
plt.colorbar(label="Training")
plt.grid(True)
plt.show()

With clearer questions in place, attention naturally shifts to assumptions. This is the part that carries the most weight, even though it’s often treated as a footnote. Causal methods don’t remove assumptions; they surface them. Whether your conclusions hold depends entirely on how reasonable those assumptions are in the real world. At its core, causal inference is about comparison. You are always comparing a treated situation to an untreated one. The challenge is making that comparison fair. If the groups differ in meaningful ways before the change, any difference afterward becomes ambiguous. This is where confounding enters the picture, and in real data, confounders are almost always present.

Experience, motivation, timing, budget size, leadership support, these factors shape both decisions and outcomes, even when they aren’t neatly captured in a dataset. Based on what I’ve observed working with messy, real-world data, this is where analyses often look strongest on paper but weakest in practice. The math might be correct, but the logic underneath is fragile. Accepting this isn’t about being pessimistic. It’s about being honest. Perfect certainty is rare outside controlled experiments, and most business data is observational. Causal inference exists to help you reason responsibly within those limits, not to manufacture confidence. Once you accept that, your work becomes calmer and more precise.

Source: Sahir Maharaj (https://sahirmaharaj.com)

This is also where transparency becomes a strength rather than a liability. Stating assumptions clearly doesn’t weaken your insight. It strengthens it. Decision-makers don’t need guarantees. They need to understand conditions, risks, and trade-offs. When you provide that context, your work becomes easier to trust, even when the answers aren’t simple. Over time, you also become more selective about which questions you try to answer causally. Some questions simply can’t be answered with the data available, and recognizing that early is a sign of maturity, not limitation. Knowing when to stop is just as important as knowing how to proceed.

Source: Sahir Maharaj (https://sahirmaharaj.com)

And yes - I know... causal inference can feel like one of those topics that sounds intimidating from the outside, but once you spend time with it, it becomes surprisingly practical. It doesn’t require perfect data or complex experiments to start benefiting from it. What it really asks is that you slow down and think more carefully about the questions you’re answering and the claims you’re making. That alone can elevate the quality of your work, even before any new techniques are introduced. I find that one of the most important things to remember is that you don’t need to master everything at once. You don’t need to jump straight into advanced methods or textbook-heavy approaches. Starting small is not just acceptable, it’s recommended. Try reframing a familiar analysis in causal terms. Take a question you’ve answered before and ask yourself what would need to be true for your conclusion to actually represent cause and effect.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import statsmodels.api as sm

np.random.seed(42)

n = 1200

baseline_health = np.random.normal(0, 1, n)
treatment = (baseline_health > 0).astype(int)

outcome = (
    3 * treatment
    + 5 * baseline_health
    + np.random.normal(0, 1.5, n)
)

df = pd.DataFrame({
    "Treatment": treatment,
    "BaselineHealth": baseline_health,
    "Outcome": outcome
})

naive_effect = df.groupby("Treatment")["Outcome"].mean()

X_adjusted = sm.add_constant(df[["Treatment", "BaselineHealth"]])
model = sm.OLS(df["Outcome"], X_adjusted).fit()

adjusted_effect = model.params["Treatment"]

plt.figure(figsize=(8, 5))
naive_effect.plot(kind="bar")
plt.title("Naïve Treatment Effect (Confounding Ignored)")
plt.ylabel("Average Outcome")
plt.grid(axis="y")
plt.show()

plt.figure(figsize=(10, 6))
plt.scatter(df["BaselineHealth"], df["Outcome"], c=df["Treatment"], cmap="viridis", alpha=0.6)
plt.title("Outcome Depends Strongly on Baseline Health")
plt.xlabel("Baseline Health")
plt.ylabel("Outcome")
plt.colorbar(label="Treatment")
plt.grid(True)
plt.show()

plt.figure(figsize=(8, 5))
plt.bar(["Naïve Effect", "Adjusted Effect"], [naive_effect[1] - naive_effect[0], adjusted_effect])
plt.title("How Adjustment Changes the Estimated Effect")
plt.ylabel("Estimated Treatment Effect")
plt.grid(axis="y")
plt.show()

So as you begin experimenting with this mindset, you’ll notice how it changes the way you interact with stakeholders. Conversations shift from “what does the data show?” to “what would happen if we changed this?” That shift is powerful. It moves your work closer to real decisions and makes your insights feel more relevant and actionable. You’ll also find that causal thinking makes you more comfortable saying “it depends.” That’s not a weakness. It’s honesty. Understanding the assumptions behind an insight allows you to explain where it holds and where it might break down.

Over time, this transparency builds trust far more effectively than confident-sounding answers that don’t survive scrutiny. So, if you’re unsure where to begin, pick one small project and treat it as a sandbox. Don’t aim for perfection. Aim for clarity. Document your assumptions, talk through your reasoning, and see how the conclusions change when you challenge them. That process alone is incredibly valuable. And if at any point you feel stuck or unsure, you don’t have to figure it out alone - I’m here and happy to support you as you explore this way of thinking and put it into practice!

Thanks for taking the time to read my post! I’d love to hear what you think and connect with you 🙂

• LinkedIn
• Kaggle
• Topmate (Free Power BI / Data Science Sessions and Resources)
• Website
• The Tech Journal (Blog)

About the author

Source: Sahir Maharaj

Sahir Maharaj is a Lead Data Scientist who leads the design and deployment of end-to-end AI solutions that drive strategic decisions at scale. As a Microsoft MVP, he has been featured internationally, including in The Indian Express and on New York's Times Square billboards, and is a prolific content creator on LinkedIn.