Understanding What an LLM is and How it Works [2/4]: What does GPT mean? Introduction to Transformer Models

Glad you’re back (or welcome if you just joined)! This is the second article in our four-section LLM series. If you skipped the Pre-Class Premiere, you might want to start there.

I created these notes while guest lecturing for Wendy Morgan’s Generative AI for Film class at the San Francisco Film School. The brief: make “Transformers” and “next-word prediction” crystal-clear for filmmakers who hate equations :) Please let me know what you think? And enjoy diving into GPT fundamentals!

What does GPT stand for? Introducing Transformers

1. Breaking Down “GPT”: What Generative Pre-trained Transformer means, and how these models are trained.
2. Tokens, Embeddings, and Next-Word Prediction: How text is encoded into numbers (tokens & vectors) and how predicting one word at a time can generate whole scripts.
3. Transformer Architecture Overview: The key components (the “magic” sauce): the Transformer blocks that include attention mechanisms and feed-forward layers.

Breaking Down “GPT”: Generative Pre-Trained Transformer

Let’s decode the name GPT. It stands for Generative Pre-Trained Transformer. Each word tells us something:

• Generative: it means the model generates new text. In other words, it can create sentences, dialogues, or entire stories that didn’t exist before. It’s essentially a super advanced auto-complete. If you give it a writing prompt, it continues from there, producing original text.
• Pre-Trained: this indicates the model has already been trained on a massive amount of data before you ever use it. Think of it as an AI that has read essentially the entire library of internet text (books, articles, websites) and learned patterns from it. The “pre-trained” aspect of it also implies you can further fine-tune it on specific tasks if needed, but out-of-the-box it has a wealth of general/tacit knowledge.
• Transformer: this is the core architecture, the type of neural network that the model is built on. Transformers are a special design of neural network that excel at handling sequences (like sentences) and have become the main engine of modern AI. In fact, the Transformer architecture is the key invention that caused the recent boom in AI capabilities. [Easy joke] And no, this “Transformer” is not a robot that turns into a Optimus Prime or a Decepticon, though the name was prolly chosen with a wink to that. In AI, it “transforms” data from one representation to another.

A Transformer model can be used for lots of things, not just text generation. There are Transformer models for translating languages (the OG use case for transformers!), for speech-to-text, text-to-speech, for generating images from descriptions (Midjourney and DALL·E use transformer-based backbones). We’ll focus on the text-generating kind of Transformer here though. Text-gen capabilities such the ones underlying chatbots like ChatGPT​. In this type of flavor, think of the model of being trained to do a simple-sounding task: given some text, predict what comes next​.

Now, you might think “Predicting the next word… is that really all it does?” The answer to that question is… Yes! That’s literally its goal. But don’t let the simplicity fool you. When you repeat that next-word prediction over and over, you get whole paragraphs and stories.

Here’s how it works in practice:

1. You feed the model a starting text say: “Once upon a time, in a distant galaxy,”
2. The model guesses the most likely next word, maybe it predicts: “there”, as the next word
3. We add that word “there” to the text and ask it again for the next word, and so on.
4. And by iteratively predicting one word at a time, the model can generate a complete passage​.

This is exactly what happens when you see ChatGPT spitting out a story word by word on your screen, a loop of next-word predictions, based on everything it’s seen so far​.

To visualize: imagine you’re writing a script collaboratively with the world’s fastest improviser. You write the first line, then this partner writes the next word, then you accept it and it writes the next, and so on. The partner is our GPT model. It has been pre-trained by reading zillions of scripts and stories, so it has an intuition for what a plausible next word or line is. That’s how it can “riff” coherently and often brilliantly, just by that simple mechanism of next-word guessing.

Tokens, Embeddings, and Next-Word Prediction

Now let’s dig into how the model sees text. Computers can’t directly understand words like “galaxy” or “explosion”, they need numbers.

So the first step is this process called tokenization. A token is a piece of text, often a word or a sub-word fragment, that the model uses as a basic unit. For example, the sentence “To date, the cleverest thinker of all time was …” might be split into tokens like To| date|,| the| cle|ve|rest| thinker| of| all| time| was| …​ . Notice some tokens can be whole words (“thinker”) or just pieces (“cle”, “ve”, “rest” make up “cleverest”). This is because many models use a subword tokenizer that breaks rare words into pieces but keeps common words intact.

Each token is then converted into a vector of numbers, known as an embedding. You can think of an embedding as the model’s internal representation of that token’s meaning. It’s like the model places each token as a point in a high-dimensional meaning space. In this space, tokens with similar meanings or usage tend to cluster together. For instance, the words “king” and “queen” would be closer to each other than “king” and “penguin”, because their contexts are more similar (royalty vs an animal).

To visualize image that: Each token is assigned a numeric embedding vector. Tokens with related meanings end up with vectors that are close by in this high-dimensional space (symbolized by arrows and for words like “jump”, “leap”, “skip” all pointing in similar directions). In essence, the embedding stage gives the model a mathematical way to know that, for example, “happy” is similar to “joyful” and far from “sad”​.

These embeddings are crucial. They let the model do math with meaning. Which is incredible. And these vectors can capture analogies. A classic and widely used example is the following: vector(“king”) - vector(“man”) + vector(“woman”) ≈ vector(“queen”). Wait what?

In other words, the model has a “gender direction” encoded in its space, if you take the “male-to-female” difference and apply it to a male word like “king”, you get something close to the female counterpart (“queen”). In another case, if you take for example the embedding for “uncle”, subtract “man” and add “woman”, you get a vector near “aunt”. This shows how certain directions in the vector space correspond to meaningful features like gender. We’ll see later that other directions can represent things like tense (present vs past) or factual connections.

Ok great. So the model starts by turning the input text into a sequence of vectors. But, how does it predict the next word from these vectors?

That’s where the Transformer architecture comes into play, with its two main components: Attention and Feed-Forward layers. We’ll get to those in a second (we’ll cover them in the upcoming two articles). But at a high level, here’s the process under the hood when the model is generating text:

1. Tokenize the input text into tokens.
2. Embed each token into a vector (using a learned embedding lookup table of sorts).
3. The sequence of token vectors goes through a series of Transformer layers. Each layer has two parts: an Attention block (which allows tokens to interact and share information based on context) and a Feed-Forward block (also called MLP, we’ll cover it next, that further transforms each token’s vector in isolation)​. These blocks are repeated many times (that’s the “deep” in deep learning, many layers).
4. After many such layers, we get a final vector representation for each position. The model then applies one more layer transformation (it’s linear + called “softmax”) to produce a probability distribution over the next-token possibilities​. Essentially, it asks: “Given everything I’ve processed, what is the likelihood of each word in my vocabulary coming next?”
5. We take the highest-probability word (or sample from the distribution) as the next word in the output. Then we append that word to the input and repeat the process for the following word, and so on.

In essence, the model is always answering one question: “Given the text so far, what should the next word be?”. And thanks to those Transformer layers (attention and feed-forward), it considers the full context and all the knowledge baked into its weights to answer that question. That’s what we’ll look at next in the two following articles.

Before moving on, it’s worth noting how huge these models are. They’re pre-trained on vast datasets (imagine feeding billions of words into training). Training adjusts all those billions of weight knobs (remember the ‘console knob’ example we had on pre-class article? If not go there!), so that the model gets really good at the next-word game. By the time it’s done, the model isn’t just parroting phrases, it has actually learned grammar, facts, style, and even some reasoning implicitly from the data. Which is simply, wild.

Transformer Architecture Overview (Layers at a Glance)

A full Transformer architecture is a stack of repeated layers. Each layer has primarily two parts:

• An Attention block, which allows each token’s vector to adapt based on (or dependent on) other tokens in the sequence.
• A Feed-Forward (MLP) block, which allows each token’s vector to transform based on learned patterns independent of other tokens. MLP standing for “Multi Layer Perceptron” fyi, it’s just another term for feed-fwd block.

Think of a Transformer layer like a scene in a play with an “ensemble” type of cast: first, in the Attention phase, everyone on stage looks around and pays attention to what others are doing or saying; then, in the Feed-Forward phase, each actor internally processes that information and perhaps delivers their next line accordingly. Then the next “scene” (layer) happens, and they do it again, building on what came before. By the end of many such scenes, each actor (token) knows exactly how to behave in context of the whole play.

In technical terms, after the embedding step, the token vectors go through an Attention block where they communicate with each other, exchanging information about context. Then the updated vectors go through an Feed-Forward/MLP block where they each get individually “interrogated” and refined (we’ll explain that with an analogy later). This pair (Attention + Feed-Forward) is repeated over and over: eg. GPT-3 has 96 layers! All these layers give the model incredible representational power, enabling it to encode very complex relationships.

Don’t worry if this is abstract, the following two articles will zoom in on Attention and the Feed-Forward network (MLP) with analogies. For now, remember that Transformers process words by successively refining their vector representations, mixing in context from other words (that’s attention) and applying learned transformations (that’s the feed-forward part)​. By the time we reach the top of the stack, the model has an embedding for each position that embodies everything it “knows” about that word in context, which it then uses to decide the best next word.

One more important thing that we’ll not detail here: Transformers typically also add positional information to these token vectors (since a bag of words alone doesn’t tell order). They use a clever trick to encode position (like 1st word, 2nd word…) into the vectors so that word order matters. We won’t go deep into that, but just be aware that the model does know the sequence order.

Alright! Now that we have the high-level idea, let’s dive into the heart of the Transformer: the Attention mechanism (upcoming section), followed by how the model stores knowledge in the Feed-Forward layers (the following section from that).

Next steps: Keep the momentum! In our upcoming section on: “Attention, A Transformer’s Cardiac Mechanism,” we’ll unpack how attention lets models focus on the right words.

Resources

• [Class Content] Understanding What an LLM Is and How It Works
• [Class] Generative AI in Film & TV: Emerging Roles and Future Opportunities
• Carlos Lozano — AI Agent Business