How LLMs Actually Work

A question typed into a chatbot returns a fluent, well-organized answer in seconds. It can feel like talking to someone who read the whole internet. But there is no person on the other end, and no giant lookup table either. What is happening is stranger, more mechanical, and more interesting than either picture.

This guide explains how large language models, or LLMs, actually work, using everyday language and analogies instead of equations. It covers what tokens are, how training shapes a model, what a transformer does, what a context window is, and why all of this produces text that sounds smart even when it is wrong.

What is a large language model?

A large language model is a neural network trained on huge amounts of text to predict the next piece of text in a sequence. By repeating that prediction one small step at a time, it can write paragraphs, answer questions, summarize documents, and produce code. It works by recognizing statistical patterns in language, not by looking up facts or understanding the world the way a person does.

Each part of that definition is worth unpacking. "Large" refers to the number of internal settings the model has, often billions or more, and the volume of text used to train it. "Language" names its home turf: text and, increasingly, other media converted into similar form. "Model" means it is a mathematical representation of patterns, not a program with hand-written rules.

For the bigger picture of where LLMs fit among machine learning and other kinds of artificial intelligence, a companion explainer covers what AI really is. This article zooms into the engine.

The core idea: predicting what comes next

Everything an LLM does comes back to one task: given some text, predict what text is likely to come next.

A tiny version exists on most phones. Typing "See you" prompts the keyboard to suggest "later," "soon," or "tomorrow." The phone is predicting the next word from patterns in how people write. An LLM is the same idea scaled up by an enormous factor and given a much deeper ability to use context.

Suppose the model sees: "The capital of France is". The most likely next word is "Paris," because that pattern appeared countless times in the training text. Suppose it sees: "Once upon a". It predicts "time." Suppose it sees the first half of a customer-service email followed by "Dear". It predicts a greeting and a name.

Now stretch this out. The model picks a next word, adds it to the text, then predicts the word after that, and keeps going. A full paragraph is a chain of predictions, each one building on everything that came before, including the words the model itself just produced. That is why answers appear gradually on screen.

Why prediction produces so much capability

It sounds too simple to explain what these systems can do. The key is that predicting text well requires absorbing a great deal about the world and about language. To predict the next word in a physics explanation, it helps to have picked up some physics. To finish a line of code, it helps to understand the syntax and the purpose. To continue a story, it helps to track characters and plot.

The training process never says "learn physics" or "learn grammar." It says "get better at predicting the next word," and those skills emerge as useful ways to do that. This is one of the surprising findings of recent years: scale up simple prediction, with enough data and computing power, and broad abilities appear.

Tokens: the building blocks of text

LLMs do not read words or letters the way people do. They read tokens.

A token is a chunk of text, often a common word, part of a word, a punctuation mark, or a few characters. A short, common word like "the" might be one token. A longer or rarer word might be split into several pieces. Numbers, code, and text in languages that appear less often in training data may be chopped into more tokens per word.

Before the model sees a message, a tokenizer converts the text into a sequence of tokens, and each token is mapped to a number. The model works with these numbers, and when it produces output, it generates numbers that get converted back into text.

Why tokens matter in practice

Tokens are not just a technical detail. They affect the user experience in several practical ways.

• Limits are measured in tokens. The amount of text a model can handle at once is counted in tokens, not pages or words.
• Pricing for developers is usually per token. For an AI used through an API, the cost typically depends on how many tokens go in and out.
• Odd mistakes trace back to tokenization. A model may struggle to count letters in a word or reverse a string, because it never saw the individual letters as separate units. It saw chunks.
• Different languages cost differently. Text that breaks into more tokens uses more of the model's capacity.

A rough rule of thumb in English is that a token is a bit shorter than a word. For exact ratios, most providers publish a token counter.

Inside the model: parameters and neural networks

A neural network is a large mathematical structure made of layers of connected units. Each connection has a number attached to it, called a weight or parameter. When a model is described as having billions of parameters, those are the numbers.

The parameters work like a vast set of dials. Each dial slightly changes how the model responds to input. No single dial means anything that could be described in words, such as "the dial for French cooking." The knowledge is spread across enormous numbers of dials working together, which is why a model cannot be opened up and read the way a book can.

When a message is sent, the tokens flow through the network layer by layer. Each layer transforms the information, and at the end the network outputs a set of probabilities: how likely each possible next token is. The model then chooses one, using a process described shortly.

Training: how the dials get set

Before training, the dials are random and the model produces gibberish. Training is the long process of tuning them until the output becomes useful.

Step one: pretraining on massive text

In pretraining, the model is shown a gigantic collection of text, which may include web pages, books, articles, code, and other sources. For each snippet, the training process hides the next token and asks the model to guess it. The model's guess is compared with the real answer. Then a mathematical procedure nudges the dials slightly so the same guess would be a bit better next time.

This repeats across an astronomical number of examples. It takes enormous computing power, typically thousands of specialized chips running for a long stretch. By the end, the model has absorbed grammar, vocabulary, facts, styles, common reasoning patterns, and the structure of many kinds of documents.

A model fresh out of pretraining is often called a base model. It is a powerful text continuer, but not a helpful assistant. Given a question, it might continue with more questions, because that is a plausible continuation of a list of questions. It does not know that an answer is wanted.

Step two: teaching it to follow instructions

To turn a base model into a useful assistant, developers continue training with examples of good conversations: questions paired with high-quality answers written or reviewed by people. This is often called instruction tuning or supervised fine-tuning. The model learns the format of being helpful: when asked something, respond with an answer, not a continuation of the question.

Step three: learning from human preferences

The next stage adjusts behavior using feedback about which responses people prefer. In one common approach, people compare pairs of responses and indicate which is better. The system learns what humans tend to rate as helpful, clear, honest, and safe, and the model is trained to produce more of that kind of response.

This is where much of the personality and manners come from: the polite tone, the refusal to help with dangerous requests, the habit of adding caveats. Different companies tune these differently, which is why chatbots built on similar technology can feel distinct.

Step four: safety and testing

Developers also test the model for harmful outputs, try to manipulate it into misbehaving, and add training or filters to reduce problems. This is imperfect and ongoing. Models can still be tricked or can still say things that are wrong or inappropriate.

The key point: training ends, then the model is frozen

When training finishes, the dials are fixed. In a chat, the model is not learning from the user in that moment. It is applying what it already absorbed. This is why a model has a knowledge cutoff, a point past which it knows nothing unless it is connected to a search tool. It is also why correcting a chatbot in conversation helps for that conversation, but does not teach the underlying model for everyone else.

The transformer: the design that changed everything

Most modern LLMs use an architecture called the transformer. The name sounds like a robot movie, but the core idea is elegant and worth understanding in simple terms.

The problem it solved

Language depends on context. In the sentence "The trophy did not fit in the suitcase because it was too big," what does "it" refer to? The trophy. Change one word to "small," and "it" now means the suitcase. To handle language well, a system has to connect words that may be far apart in a sentence or paragraph.

Earlier approaches processed text roughly one word at a time and struggled to keep track of long-range relationships. They also were slow to train, because each step depended on the last.

Attention: deciding what to focus on

The transformer's breakthrough was a mechanism called attention. For each token, the model figures out which other tokens in the text matter most for understanding it, and blends in information from them accordingly.

Picture a reader highlighting a page. When the model processes the word "it," attention lets it look back across the sentence and weigh "trophy" and "suitcase" differently depending on the clues. Each layer of the network does this, and models use many attention "heads" in parallel, each potentially picking up different kinds of relationships: grammar, reference, topic, tone.

Why it scaled so well

Transformers can process all the tokens in a passage at once instead of strictly one after another, which suits modern chips designed for parallel math. That made it practical to train on far more data with far larger models. Scale turned out to matter a lot, and the transformer was a design that could take advantage of it.

The details are less important than the takeaway: the model does not read left to right like a person. It looks at the whole context at once and learns which parts relate to which.

From probabilities to words: how the model picks

At each step, the model produces probabilities for every possible next token. For "The capital of France is", "Paris" might get a very high probability, and other tokens get tiny ones. How the system turns that into an actual choice affects the output.

Always picking the top choice

If the model always chose the single most likely token, answers would be consistent but often dull and repetitive. This is sometimes called greedy decoding.

Sampling with randomness

Most chat products add some randomness, so the model samples from the probability distribution instead of always taking the top one. Occasionally it chooses a less likely token, which can lead to more varied, creative text.

A setting often called temperature controls this. A low temperature makes the model stick close to the most likely words, which suits factual or code tasks where consistency is wanted. A higher temperature allows more surprising choices, which can suit brainstorming or creative writing. Some tools expose this setting, and many hide it.

Why the same question gets different answers

Because of this randomness, asking the same question twice can produce different wording or even different conclusions. That is not a bug. It is how generation works. When a consistent result is needed, saving the output that works is more reliable than expecting to reproduce it exactly.

The context window: the model's working memory

The context window is the amount of text, measured in tokens, that a model can consider at one time. It includes everything: the instructions, the conversation so far, any documents pasted in, and the model's own previous replies, plus the response being generated.

Think of it as a desk of limited size. Everything on the desk is visible to the model. Anything that does not fit gets dropped or never makes it on. The model does not have a separate long-term memory of earlier chats unless the product adds one.

What happens as conversations get long

Context windows have grown a lot over time, but they are still finite. When a conversation exceeds the limit, the product may cut the oldest messages, summarize them, or return an error. Even within the limit, quality can suffer: models sometimes pay less attention to material buried in the middle of a very long input, or lose track of early instructions.

Practical habits that help:

• Put the most important instructions clearly and restate them if the conversation drifts.
• Start a fresh conversation for a new topic, instead of piling unrelated tasks into one thread.
• Paste only the relevant parts of long documents when possible.
• For long projects, keep a short running summary of decisions and facts that can be pasted back in.

This is especially relevant for something like worldbuilding. A fantasy setting with hundreds of names and rules will not fit in a model's head by itself. The canon has to be supplied each time, which is why a maintained reference document is a common recommendation, a point expanded in AI worldbuilding.

Context is not memory

Some products offer a memory feature that saves notes about a user between chats and inserts them into the context window later. That is a layer on top of the model, not something the model does natively. Where it matters, the product's settings show what is stored and how to delete it.

Why LLMs sound so smart

The fluency of these models is the source of both their appeal and their risk. Here is why they sound the way they do.

They learned from human writing at enormous scale. The text they absorbed was written by people who explain, argue, joke, and organize ideas. The model reproduces those patterns, including a confident expository style.

They were tuned to be helpful and readable. The preference training described earlier rewards clear, well-structured, polite answers. That produces a consistent, authoritative-sounding voice.

Fluency is cheap for them. Producing grammatical, well-organized text is exactly what they are optimized for. Producing correct facts is a harder, only partly aligned goal.

People read minds into language. It is natural to assume that coherent speech comes from understanding. With human speakers, that assumption is usually safe. With models, it is not.

The practical lesson is to separate how an answer sounds from whether it is true.

Why LLMs make things up

Given the mechanism, hallucination is less surprising. The model generates the most plausible continuation. Usually that overlaps with the truth, because true statements are common in the training data. But "plausible" and "true" are different, and when the model lacks solid information, it still produces something plausible, because it must produce something.

Common triggers include obscure topics with little training data, questions with false assumptions built in, requests for precise citations or quotes, and tasks needing exact numbers. The model may invent a source that looks real because source-shaped text is easy to generate.

This deserves its own full treatment, which appears in why AI makes things up. The short version: any specific, checkable claim is best treated as a lead to verify, not as a finished fact.

What LLMs can do beyond predicting text

Plain next-word prediction is the core, but modern products wrap extra capabilities around it.

Tools and search

Many assistants can call outside tools: a web search, a calculator, a code interpreter, a file reader. The model decides when to use a tool, writes the request, reads the result, and incorporates it into its answer. This helps a lot with recent information and exact math, because the facts or calculations come from a tool instead of the model's memory. When these tools are chained into multi-step tasks, the result is what people call agents, covered in AI agents explained.

Retrieval from documents

Some systems search a collection of files or a company knowledge base and feed the relevant passages into the context window before the model answers. This approach, often called retrieval-augmented generation, grounds the answer in specific sources and can reduce made-up details. It does not eliminate them, because the model can still misread or overstate what it retrieves.

Reasoning modes

Some models are trained or configured to work through a problem step by step before giving a final answer, sometimes using extra computation at the time of the question. This can improve results on math, logic, and multi-step planning, at the cost of slower and more expensive responses. The details vary by provider and change often, so the current documentation is the best source for how each product handles it.

Multimodal input and output

Many models now handle images, audio, and sometimes video alongside text. Technically, those inputs are converted into token-like representations the same network can process. The same prediction-based idea applies, extended beyond words.

Common misunderstandings, corrected

"It is searching the internet when it answers." Not unless search is enabled. By default it generates from patterns stored in its parameters.

"It stores the conversation and learns from it live." The model itself does not update during chat. Providers may use conversations for future training depending on their policy and the user's settings.

"It understands what it says." It models language patterns very well. Whether that amounts to understanding is debated, but it clearly lacks grounded experience and a reliable sense of its own uncertainty.

"A bigger model is always better." Larger models often perform better, but smaller models can be faster, cheaper, and good enough for focused tasks. Training quality, data, and tuning matter as much as size.

"Asking it to be honest means it won't make things up." Risk can be reduced with good instructions, such as asking it to say when it is unsure, but hallucination cannot be switched off with a magic phrase.

"It remembers everything said earlier." Only what remains in the context window, plus any memory feature the product adds.

What this means for how LLMs are used

Knowing the mechanism points directly to better habits.

Give context, because the model only knows what is on the desk

Since the model has no knowledge of a user's situation beyond the context, it helps to spell out the goal, audience, constraints, and relevant facts. Concrete templates appear in prompting that works.

Ask for the format needed

Because the model continues patterns, showing an example of the desired output is powerful. Pasting two sample product descriptions and asking for a third leads the model to follow the pattern.

Break big tasks into steps

Long, complicated requests are more likely to go wrong than a sequence of smaller ones. Asking for an outline first, approving it, then requesting each section is a reliable approach.

Verify the checkable parts

Names, dates, numbers, quotes, links, and citations are the places where plausible-but-wrong text hides. Checking them against reliable sources is worthwhile.

Use tools when accuracy matters

For current events, enabling search and reading the cited pages helps. For arithmetic, a calculator or confirmed code execution is better. For facts about a specific business, pasting in the real information is best.

Expect variation

If one answer is off, regenerating or rephrasing can help. Comparing two or three outputs for consistency is useful, since disagreement between attempts is a warning sign.

Mind privacy

Anything pasted goes to the provider. Reviewing data settings and avoiding sensitive information is advisable, as covered in what not to tell a chatbot.

A quick example, from prompt to answer

To make the pieces concrete, consider tracing a simple request. Suppose the prompt reads: "Write a two-sentence description of a daily word puzzle game."

1. Tokenization. The sentence is split into tokens and converted to numbers.
2. Context assembly. The product adds hidden instructions, such as how the assistant should behave, and combines them with the message and any earlier conversation. All of it sits in the context window.
3. Processing. The tokens pass through the transformer layers. Attention helps the model work out that "daily word puzzle game" is the subject and "two-sentence description" is the format.
4. Probabilities. The model outputs a probability for every possible first token of the reply.
5. Selection. One token is chosen, with some controlled randomness. Perhaps it is "A."
6. Repeat. That token is appended, and the process runs again for the next token, then the next, until the model produces an end signal or hits a length limit.
7. Detokenization. The chosen tokens are converted back into readable text and streamed to the screen.

At no point did the model "decide what it wants to say" and then type it out. The message emerged one token at a time, each shaped by everything before it. That is also why it can start a sentence confidently and then paint itself into a corner, and why asking for an outline first often improves results.

Limits worth remembering

• No built-in fact-checking. There is no internal step that verifies a claim against reality.
• Sensitivity to wording. Small changes in phrasing can produce noticeably different answers.
• Inconsistent reliability. A model can solve a hard problem and fail an easy one in the same session.
• Bias from data. Patterns in training text, including unfair ones, can show up in output.
• Knowledge cutoff. The model's built-in knowledge ends at a point in time.
• Finite context. Long inputs can exceed the window or dilute attention.
• Hidden instructions. Products add behind-the-scenes prompts that shape behavior, and users usually cannot see them.

How to keep learning as things change

Model names, sizes, context lengths, and features turn over quickly, so memorizing numbers is of limited use. The concepts above have stayed stable even as products changed, and the following can be checked live when they matter:

• The provider's documentation for context window size, supported inputs, and tool features.
• The pricing page for how usage is measured and billed.
• The privacy and data controls for what is retained and whether it is used for training.
• Recent independent comparisons, checking the date, plus a quick test using real tasks.

For readers curious about the competitive pressures that drive these changes, the AI arms race explains who is building what and why.

Quick takeaways

• An LLM is a neural network trained to predict the next token in a sequence of text. Repeating that step produces whole answers.
• Tokens are the chunks of text the model reads and writes. Limits and pricing are usually measured in tokens.
• Training has several stages: pretraining on massive text, instruction tuning, and feedback-based tuning to make the model helpful and safer. After training, the model is frozen.
• Transformers use attention to decide which parts of the context matter for each token, which lets models handle long-range relationships in language.
• The context window is the model's limited working space. Anything outside it is invisible unless a memory or retrieval feature adds it back.
• LLMs sound smart because they learned from fluent human writing and were tuned to be clear and helpful, but fluency does not guarantee accuracy.
• Hallucinations are a natural result of generating plausible text, so names, numbers, quotes, and citations are worth verifying.
• Tools like search, calculators, and document retrieval make answers more reliable on facts, math, and recent events.

FAQ

How do large language models work in simple terms?

They are trained on huge amounts of text to predict the next token, which is a small chunk of text. At answer time, they generate a response one token at a time, each chosen based on everything that came before. Patterns of grammar, facts, and reasoning are encoded in billions of numerical settings that were tuned during training.

What is a token in an LLM?

A token is a piece of text that the model reads and writes. It can be a whole short word, part of a longer word, a punctuation mark, or a few characters. Models measure input, output, and limits in tokens, and developers are usually billed per token. In English, a token is typically a little shorter than a word.

Do LLMs actually understand language?

They are very good at modeling how language works, and they can produce answers that look like understanding. They do not have lived experience, and they cannot reliably tell when they are wrong. Whether this counts as understanding is a debate, but for practical purposes, they are best treated as powerful pattern-based tools whose output needs checking.

What is a context window?

It is the total amount of text, measured in tokens, that the model can consider at one time, including messages, pasted documents, earlier replies, and the answer being written. If a conversation grows beyond the limit, older material may be dropped or summarized. Models also tend to attend less reliably to details buried in very long inputs.

Why do chatbots give a different answer each time?

Most chat products add controlled randomness when choosing each next token. A setting often called temperature governs how much. That variation helps with creativity but means answers are not perfectly repeatable. For consistency, saving the preferred version or asking for a structured format with clear instructions helps.

Do LLMs learn from conversations?

Not in real time. The model's internal settings are fixed after training. Within a single chat, it uses what has been said, up to the context limit. Whether a provider uses conversations to train future models depends on its policy and the account settings, so the data controls are worth checking before sharing anything sensitive.

Why are LLMs sometimes confidently wrong?

The model generates text that fits learned patterns, and confident, well-formed language is a pattern it has learned well. When it lacks solid information, it still produces a plausible-sounding answer instead of stopping. Asking for sources, enabling search tools, and verifying key details are the best defenses. The explainer on why AI makes things up goes into more depth.

Is a bigger model always smarter?

Not always. Larger models tend to be more capable on hard tasks, but quality also depends on training data, tuning methods, and design choices. Smaller models can be faster, cheaper, and entirely adequate for focused jobs such as classification, short rewrites, or simple questions. Testing on real tasks is better than assuming bigger is better.