If you have ever wondered why modern language models feel so good at keeping context, attention is usually the reason. I still remember the first time this really clicked for me as a data scientist. It was one of those moments where things suddenly felt obvious in hindsight. Models did not have to plod through sequences step by step anymore. They could slow down, look back, and decide what actually mattered. That shift changes how you think about deep learning. Sequences stop feeling rigid and start feeling flexible. You stop worrying so much about what the model might forget and start thinking about what it should focus on instead. If you work with data, especially anything sequential or text based, this idea shows up everywhere. Attention is not magic, even though it can feel like it at first. Once you understand it, transformers stop being intimidating and start feeling surprisingly logical.
What you will learn: In this edition, we’re exploring attention mechanisms and why they became such a big deal in modern deep learning. By the time you reach the end, you will have a clear sense of what problem attention was meant to solve, why older sequence models struggled, and how attention fits naturally inside transformer models using a torch-style way of thinking.
Read Time: 8 minutes
Source: Sahir Maharaj (https://sahirmaharaj.com)
Let me start with something I have seen many times in practice. You build a sequence model and it works great at first. Short inputs, clean patterns, decent results. Metrics look stable and you feel confident pushing forward. Then you increase sequence length or add richer context and things slowly start to break down. Nothing is obviously wrong. The model still trains. Loss still goes down. But performance flattens or degrades in subtle ways. In my experience, this is usually the moment where the model is being asked to remember too much at once. Traditional sequence models try to compress the past into a single moving state. Over time, that state becomes overloaded.
This shows up clearly in real data science problems. Language, user behavior, event streams, and time series all depend on relationships that stretch across time. An early decision or event can explain something much later. When a model cannot easily reconnect those dots, it is not because the data is bad. It is because the model cannot decide what deserves attention. Attention exists because treating everything as equally important does not scale. Instead of dragging the entire past forward at every step, attention lets the model look back when it needs to. Relevance becomes dynamic rather than fixed. From what I have observed, this single change improves both performance and stability.
There is also a very human element here. When you analyze data, you do not replay every record in your head. You jump to turning points, strange outliers, or moments you already know matter. Attention gives models the ability to do the same thing. So as we move forward, keep this in mind. Attention was not introduced to make models more complex. It showed up because real world data needed a better way to focus.
import torch
import matplotlib.pyplot as plt
import numpy as np
torch.manual_seed(0)
sequence_length = 80
signal_position = 10
x = torch.zeros(sequence_length)
x[signal_position] = 1.0
hidden_state = 0.0
decay = 0.95
rnn_memory = []
for t in range(sequence_length):
hidden_state = hidden_state * decay + x[t]
rnn_memory.append(hidden_state)
queries = torch.randn(sequence_length)
keys = torch.randn(sequence_length)
values = x.clone()
scores = torch.outer(queries, keys)
attention_weights = torch.softmax(scores, dim=1)
attention_output = attention_weights @ values
plt.figure(figsize=(12,5))
plt.plot(rnn_memory, label="Sequential memory signal")
plt.plot(attention_output, label="Attention-based signal")
plt.axvline(signal_position, linestyle="--", alpha=0.5)
plt.legend()
plt.title("Why Attention Exists: Memory Decay vs Relevance Retrieval")
plt.xlabel("Sequence position")
plt.ylabel("Signal strength")
plt.show()Once the why makes sense, the how becomes much easier to digest. I usually explain attention as structured comparison rather than heavy computation. At each step, the model is basically asking a question. Given what I am trying to predict right now, which parts of my input should influence that decision the most. Each element in the sequence is turned into a learned representation. These representations capture meaning, not just position. Attention then compares these representations with each other and assigns importance scores. What matters is not the exact formula. What matters is that importance changes with context. This is where queries, keys, and values come in as a mental shortcut.
Source: Sahir Maharaj (https://sahirmaharaj.com)A query represents what the model is currently looking for. Keys represent what information is available. Values contain the actual content. When I first thought about it this way, attention stopped feeling abstract and started feeling natural. Another important point is that attention is learned. There are no hard rules telling the model what to focus on. Patterns of relevance emerge from data. Over time, the model gets better at deciding what matters in different situations. From a data science point of view, this feels very familiar. You start with a question, scan the data, notice patterns, and adjust your focus. Attention does the same thing, just at scale and with gradients. Once this clicks, transformers stop feeling mysterious. They are not guessing. They are weighting evidence in a structured and learnable way.
import torch
import matplotlib.pyplot as plt
import numpy as np
torch.manual_seed(1)
tokens = 20
embedding_dim = 16
embeddings = torch.randn(tokens, embedding_dim)
W_q = torch.randn(embedding_dim, embedding_dim)
W_k = torch.randn(embedding_dim, embedding_dim)
W_v = torch.randn(embedding_dim, embedding_dim)
Q = embeddings @ W_q
K = embeddings @ W_k
V = embeddings @ W_v
scores = Q @ K.T / torch.sqrt(torch.tensor(embedding_dim, dtype=torch.float32))
attention = torch.softmax(scores, dim=1)
context = attention @ V
plt.figure(figsize=(8,6))
plt.imshow(attention.detach().numpy(), cmap="viridis")
plt.colorbar()
plt.title("Attention Heatmap: What Each Token Pays Attention To")
plt.xlabel("Key positions")
plt.ylabel("Query positions")
plt.show()Next is where everything comes together. Transformers make a bold choice. They remove recurrence entirely and rely on attention to move information around. The first time I worked with transformers, this felt risky. Over time, it became obvious why it works so well. Instead of processing sequences step by step, transformers look at the entire sequence at once. Attention allows every element to interact with every other element directly. Distance in the sequence no longer limits learning. Early information does not fade just because it came first. Multi-head attention pushes this idea further. Rather than learning one notion of relevance, the model learns several at the same time. One head might focus on structure, another on meaning, another on position.
I often compare this to looking at the same dataset through different analytical lenses. As layers stack, representations become more abstract. Early layers capture simple relationships. Deeper layers capture richer structure and context. This layered understanding is why transformers handle complex tasks so well. From a torch perspective, this is actually very approachable. Attention layers are explicit modules. You can inspect them, reason about them, and modify them. I have found this clarity incredibly helpful when debugging or experimenting. At a high level, attention is the connective tissue of the transformer. Without it, the architecture loses what makes it powerful.
import torch
import matplotlib.pyplot as plt
import numpy as np
torch.manual_seed(2)
tokens = 12
embedding_dim = 24
heads = 3
head_dim = embedding_dim // heads
x = torch.randn(tokens, embedding_dim)
W_q = torch.randn(heads, embedding_dim, head_dim)
W_k = torch.randn(heads, embedding_dim, head_dim)
W_v = torch.randn(heads, embedding_dim, head_dim)
all_head_outputs = []
for h in range(heads):
Q = x @ W_q[h]
K = x @ W_k[h]
V = x @ W_v[h]
scores = Q @ K.T / torch.sqrt(torch.tensor(head_dim, dtype=torch.float32))
attn = torch.softmax(scores, dim=1)
output = attn @ V
importance = attn.mean(dim=0)
all_head_outputs.append(importance)
plt.figure(figsize=(10,5))
for i, head in enumerate(all_head_outputs):
plt.bar(np.arange(tokens) + i*0.2, head.detach().numpy(), width=0.2, label=f"Head {i+1}")
plt.title("Multi-Head Attention: Different Heads, Different Focus")
plt.xlabel("Token position")
plt.ylabel("Average attention weight")
plt.legend()
plt.show()At this point, it is worth slowing down and talking honestly about where attention really shines in practice. I have seen people treat attention as a silver bullet, something you add and everything magically improves. That expectation usually leads to disappointment. Attention is powerful, but only when the problem actually benefits from dynamic relevance. Attention tends to work best when relationships matter more than raw order. Language is the obvious example, but it shows up in many other places too. User behavior, event sequences, transaction histories, and even sensor data often depend on interactions across time rather than simple progression. In these cases, attention gives the model freedom to connect distant but meaningful signals.
That said, not every problem needs this level of flexibility. For short sequences or highly regular patterns, simpler models often perform just as well and are easier to train. I have personally seen teams overcomplicate pipelines by reaching for transformers when the problem did not demand it. Attention adds capacity, but it also adds cost. Another thing to keep in mind is interpretability. Attention is often described as explainable, but that is only partially true. While attention weights can offer hints about what the model focuses on, they are not explanations on their own. I have learned to treat them as clues rather than answers. There is also the issue of scale. Attention grows more expensive as sequences get longer. This matters in real systems where latency and memory are constraints. Understanding this tradeoff helps you choose architectures more deliberately rather than defaulting to whatever is popular.
What I find most valuable about attention is not just the performance gain. It is the mental model it introduces. Thinking in terms of relevance and focus changes how you frame problems, even before you touch a model. As a data scientist, this has influenced how I approach feature engineering, problem decomposition, and evaluation. I ask different questions earlier in the process. What should matter here. When should it matter. And how might that change over time. If you take anything from this objective, let it be this. Attention is a tool, not a shortcut. When used intentionally, it can unlock clarity. When used blindly, it can add complexity without value.
Source: Sahir Maharaj (https://sahirmaharaj.com)If there is one thing I hope you take away from this, it is that attention is far more approachable than it first appears. You do not need to understand every equation or architecture detail to start using it thoughtfully. What matters is the shift in mindset, moving from trying to remember everything to deciding what deserves focus. If you are curious to try this out, start small. Pick a simple problem where context actually matters. Short text, small sequences, something you already understand well. Play with the idea of relevance and see how your thinking changes. In my experience, those small experiments build confidence quickly and make the bigger ideas easier to digest later.
Most importantly, do not feel like you have to figure this out alone. If you ever need a second opinion or a nudge in the right direction, feel free to reach out to me. I am always more than happy to help, talk through a problem, or share what I have learned along the way!
Thanks for taking the time to read my post! I’d love to hear what you think and connect with you! 🙂
- Kaggle
- Topmate (Free Power BI / Data Science Sessions and Resources)
- Website
- The Tech Journal (Blog)
About the author
Source: Sahir Maharaj
Sahir Maharaj is a Lead Data Scientist who leads the design and deployment of end-to-end AI solutions that drive strategic decisions at scale. As a Microsoft MVP, he has been featured internationally, including in The Indian Express and on New York's Times Square billboards, and is a prolific content creator on LinkedIn.
Super User