Understanding What an LLM is and How it Works [3/4]: Attention at its Core

Great to see you again! You’re reading the third article in the series. If you missed the previous whick brokedown GPTs, catch up here. These materials come from my guest lecture in Wendy Morgan’s Generative AI for Film course at San Francisco Film School. In that session, “attention” was the crowd-pleaser: it shows how LLMs actually focus on the right words. Scroll down to see why “Attention is All You Need” became an iconic paper title, minus the math headache :) Please let me know what you think? And hope you enjoy further diving in the core dimensions of attention.

Attention, And The Crux of Transformer Models

1. What is “Attention” in an LLM? Intuition for attention as the model’s way of focusing on relevant words (context matters!).
2. Examples of Attention in Action: Using analogies (eg. ambiguous words like “mole” or “bank”) to show how attention clarifies meaning from context.
3. How Attention Mechanisms Work (Conceptually): A peek under the hood of queries, keys, and values, explained without heavy math.
4. Multi-Head Attention & Why It’s Powerful: (Briefly) The idea that the model has multiple attention “heads” looking at different aspects of the sequence.

What is “Attention” and Why Does Context Matter?

If Transformers are a car, then the attention mechanism is the engine. The pioneering research paper that introduced Transformers was even titled “Attention is All You Need.” But what is “attention” in this context? Simply put, attention allows the model to decide which other words (tokens) in the input are the most relevant to a given word when producing the next representation. It’s a way for the model to focus on the parts of the input that matter for understanding a particular token.

Think of how we humans use the word “attention”: if you’re reading a screenplay and trying to interpret a character’s line, you pay attention to the scene’s context, the previous dialogue, the setting, the character’s relationships, to understand that line correctly. LLMs do the same thing, but mathematically. At each layer, for each token, the model figures out how strongly to weight (attend to) every other token’s embedding when updating that token’s embedding.

Why is this needed? Because the meaning of a word often depends on context. Consider the word “bank” in these two sentences:

• “Alice deposited money in the bank.”
• “The fisherman sat on the bank of the river.”

You and I know that “bank” means two very different things here (financial institution vs. riverbank) based on context (“money” vs “river”). Initially, an embedding for “bank” might be a sort of average of all meanings of “bank.” The attention mechanism will help adjust that embedding by looking at the other words in the sentence (context). In the first sentence, attention will see words like “money, deposited” and shift “bank”’s representation toward the financial sense. In the second, seeing “river, fisherman” will shift it toward the river edge sense. Attention is the model’s way of disambiguating and refining word meanings on the fly using context.

This applies not just to ambiguous words, but any situation where one part of the text influences another. Pronouns, for example: “John went to the store. He bought milk.” The model needs to know “he” refers to John. Attention will cause the token “he” to strongly look at “John”’s token when processing, so it carries that info forward.

Examples: Attention in Action (Disambiguation and Context-Blending)

Let’s use a concrete example from the model’s perspective. The word “mole” can mean a small animal, a unit in chemistry, or a spot on the skin. Suppose the input text is:

1. “The American shrew mole is a tiny mammal found in North America.”
2. “One mole of carbon dioxide contains 6.022×10²³ molecules.”
3. “The dermatologist said to remove the mole on my arm.”

Initially, when the word “mole” is converted to an embedding vector, it’s the same for all three cases (because at the embedding stage, no context has been applied yet, it’s just like a dictionary lookup). That embedding is a generic one representing “mole” in some averaged sense. The model then applies attention:

• In sentence 1, attention will notice “American shrew” (a type of mole animal) and adjust the “mole” token’s vector to emphasize the animal sense.
• In sentence 2, attention will latch onto “carbon dioxide” (plus the number, etc.) and realize this is the chemistry mole (6.022×10²³ of something).
• In sentence 3, attention sees “dermatologist” and “arm”, and switches “mole” to the skin-spot sense.

The outcome: after the attention block, the vector for “mole” in each sentence is now different, each one has been pulled in a direction that represents the appropriate meaning. The model has effectively “understood” the word in context by paying attention to the right clues around it.

An example to “visualize” (and imagine): the word “model” in two different contexts. On the one hand, we have this portion of a phrase “a machine learning model,” and on the other hand we have another portion of a phrase which is, “a fashion model”. The Attention mechanism will use surrounding words to update the embedding of “model.” In the first case, “model” relates to AI (think algorithms), and in the second one, “model” refers to a person who walks the runway. This shows that before attention, “model” might have a generic embedding; after attention, the word “model” in the AI context might align with concepts like data or algorithms, whereas the “model” in the fashion context aligns with clothing or appearance. In short, the same word token is adapted to two different meanings via context.

One more neat capability: long-distance dependencies. If we have a long sentence, or even multiple paragraphs, attention can connect words that are far apart. For example, if a character is mentioned in chapter 1 of a script and referred to indirectly in chapter 5, the model can (to an extent) use attention over many layers to bridge that gap. In practice, there’s a limit because the model has a fixed “attention window” of a certain number of tokens, but within that window it doesn’t matter if the relevant context word is 5 words away or 50 words away, attention can still link them. This is why GPT can handle fairly large contexts (some models handle thousands of tokens of context).

How Attention Mechanisms Work (Conceptually, No Heavy Math)

We’ve talked about what attention does (focus on relevant context). Let’s gently peek at how it might do this internally. You might hear terms like “queries, keys, and values”:

• Every token produces a Query vector and Key vector (and a Value vector) by multiplying its embedding by some weight matrices (Remember? These are learned parameters). You can imagine the Query as the question “What am I looking for?” and the Key as “What do I have to offer to others?” for each token.
• The model then takes a token’s Query and compares it with every other token’s Key (via what’s called “dot products” between the query and each key which gives a similarity score. No worries, we’re not covering this here). But what this is means is that if another token’s Key is a good match for this Query, then “this token has something the querying token wants to pay attention to.”
• These scores are turned into weights (via a softmax, which again is not covered here but is another operation that basically emphasizes the largest scores and diminishes the smaller ones).
• Then we use those weights to take a weighted sum of the Value vectors from all the other tokens. The Value vector is basically the information content of a token that it can share. So a token ends up with a weighted combination of other tokens’ values. Effectively, it has pulled in information from tokens it found relevant.

All this is a lot of detail, but the takeaway: each token ends up with a summary of the things it deemed important from the other tokens. This updated representation replaces or augments the token’s own embedding for the next layer. Through many layers, these attention operations can distribute information globally across the sequence.

For instance, if our token is “bank” and context has “money” nearby, the Query of “bank” will likely strongly match the Key of “money”, so “bank” will pull in the Value of “money” (and maybe “deposit”) heavily, thus shifting “bank” toward the finance meaning in that layer. The mechanism ensures “bank” attended to “money” more than, say, to an unrelated word like “the”.

And guess what, this happens for every token in parallel. That’s the unlock. Each word simultaneously gathers contextual info from others. The computations are optimized so it’s not as slow as it sounds, but that’s the gist.

In practice, the model doesn’t label these interactions as “meaning” or “grammar” explicitly, it’s just learning patterns. But effectively, some attention heads might specialize. For example, one head might always pick up “this noun is the subject of that verb” connections, another might track “quotation who said what”, another might handle “coreference” (linking pronouns to names), etc.

Multi-Head Attention & Its Benefits

Transformers actually use multi-head attention, meaning they perform several attention operations in parallel (with different weight matrices, so they can focus on different things). For instance, say we use 8 heads. When processing a token, it’s like the token sends out 8 separate “queries”, each querying a different aspect:

• Head 1 might attend to the direct object of a verb,
• Head 2 might attend to the subject,
• Head 3 might attend to an associated adjective or adverb,
• Head 4 might look far back for topic-related words,
• Head 5 might do something more structural like attend to the period or comma (yes, punctuation gets attention too!),
• etc.

Each head pulls in its own value summary. These are then concatenated and mixed together through another linear transform. The result is that each token’s vector now contains a blend of information gleaned from multiple perspectives. Multi-head attention is like having a committee of specialists, each focusing on a different interpretation of the context, and then pooling their insights.

The outcome is very robust context integration. This is one major reason why Transformers outperform older models: they can capture complex relationships in one layer that an RNN (the former way of doing for example translation) might need many time steps to capture (or might miss entirely).

To keep our focus, we won’t dive deeper into the math of attention. Visually and conceptually, you can remember it as lines connecting words, in diagrams, we often draw lines from each word to the words it’s attending to, with thicker lines meaning stronger attention. If you’ve seen those colorful diagrams from GPT model visualizations, that’s what they show: which words are paying attention to which other words, and how strongly.

By now, we’ve covered how attention helps the model understand context and assign meaning dynamically. With attention, the model builds a rich representation of each word in context, which is how it “knows” that in this sentence, 🐇 “mole” is a creature, but in that sentence, 🧪 “mole” is a unit.

Next, we’ll move to our final section for this series, where we ask: How does the model store more static knowledge, like facts? The attention helps with context in the input, but what about things the model just knows (from training) even if not explicitly stated in the prompt? For that, we look at the other side of the Transformer: the Feed-Forward networks, which turn out to be the model’s sort-of long-term memory.

Resources

• [Class Content] Understanding What an LLM Is and How It Works
• [Class] Generative AI in Film & TV: Emerging Roles and Future Opportunities
• Carlos Lozano — AI Agent Business