What is a Large Language Model, really?

You've probably chatted with one – ChatGPT, Claude, Gemini. But when someone says "large language model", what's actually inside? Here's a plain-language tour of the machinery, without the jargon. By the end you will know enough to pick a model size, set temperature sensibly, and decide whether a task belongs on your machine or in the cloud.

Source note: this is a conceptual explainer – no vendor benchmarks claimed. Concrete numbers (sizes, speeds, VRAM) link to our tested guides: how much VRAM, best GPU, and the lab benchmarks.

Three words, unpacked

Language model – a program trained to predict the next word in a sequence. That's it. Not to "think", not to "understand" in a human sense. It's a next-word machine. Everything else is a consequence of doing that one job astonishingly well.

Large – the model has billions of adjustable numbers (parameters) and was trained on a huge slice of the internet's text. That scale is what turns "predict the next word" into something that can answer questions, write essays, and debug code.

Tokens: the real alphabet

Models don't read letters or words directly – they read tokens. A token is a chunk of text that's typically a few characters long. For example:

"VelsTech Solutions writes about tech"
  -> "Vels" | "Tech" | " writes" | " about" | " tech"

The model predicts one token at a time, and each predicted token is appended to the input before predicting the next. That's why longer responses take longer: every word is a fresh prediction.

Two consequences beginners miss. First, cost and speed are counted in tokens, not words – cloud APIs bill per million tokens and local speed is measured in tokens per second, so a “short answer” in words can still be hundreds of tokens with context attached. Second, the context window is also measured in tokens: a 128K window sounds huge until a pasted document eats 90K of it. Thinking in tokens instead of words is the single habit that makes VRAM, cost, and context math click – our VRAM guide works entirely in these units.

Why size matters: 7B vs 14B vs 35B vs 70B

Parameters are the model's adjustable numbers – loosely, its memory of patterns. More parameters usually mean better reasoning, writing, and code, but also more VRAM and slower generation on the same card. In practice: a 7B Q4 fits ~5.5 GB and runs anywhere; a 14B Q4 needs ~10.5 GB and feels like a genuinely useful assistant; a 35B MoE stores ~20 GB of weights but fires only ~3B per token, so it behaves like a 7B for speed while reasoning better; a 70B Q4 wants ~40–45 GB – cloud territory for most home GPUs. That ladder is why our GPU picks are organized by VRAM (8 → 16 → 24 GB) instead of brand – the size you can host decides the quality you get.

Bigger is not always better for your task. A 7B at the right temperature beats a 70B at the wrong context length, and a 35B MoE on your own GPU beats a frontier API for anything private, metered, or offline. Pick the smallest model that handles your hardest daily task – then spend the savings on context and patience. Exact fit math lives in the VRAM calculator.

How it learns

During training, the model reads massive amounts of text and repeatedly tries to predict the next token. When it's wrong, its internal numbers are nudged so it's slightly less wrong next time. Repeat that a few trillion times and you get a model that has absorbed patterns of grammar, facts, reasoning, and style – not because it memorized them, but because they're the patterns that best predict text.

A useful mental model: the LLM has learned the "shape" of human text – how concepts usually connect – and generates by following that shape.

Context: what it "sees"

When you send a prompt, everything you write – plus everything the model has generated in the same conversation – becomes the context window. The model only "knows" what's in that window. It doesn't have a memory of previous chats, and it doesn't browse the live web unless a tool tells it to.

That's why the same question can get different answers: the model isn't recalling "the" answer, it's generating the most probable continuation given your specific context.

Temperature: creativity dial

At each step the model doesn't pick just one next token – it assigns a probability to many and samples from them. The temperature setting controls how random that sampling is:

• Low (0–0.3): predictable, deterministic answers. Great for code and math.
• Medium (0.7): balanced, natural conversation.
• High (1+): creative, varied, sometimes nonsensical. Good for brainstorming.

Practical rule: low temperature for anything you will act on (code, instructions, summaries you forward), medium for conversation, high only for ideas you plan to curate. Most “the model is dumb” reports are temperature mismatches – a brainstorming setting grading a math test. When answers feel randomly wrong, lower the dial before blaming the model. Tips for steering output live in better prompts.

What happens when you press enter: prefill then decode

Two phases run every turn. Prefill absorbs your prompt in one parallel gulp – fast on GPUs, and why long pastes pause before the first word. Decode then generates one token at a time – the streaming you watch, measured in tokens per second. A 7B Q4 decodes ~5–10 tok/s on a modern CPU (readable) and ~40+ tok/s on a modest GPU (instant); our lab's 35B MoE held 25–29 tok/s even at 262K context because only ~3B parameters fire per token. If chat feels instant but long docs stall, your prefill is the bottleneck; if every word drips, your decode is offloading to RAM – drop a size or shorten context. See ROCm vs Vulkan for how backends split these two speeds.

Why it still gets things wrong

Because it's predicting text, not checking facts, an LLM can confidently produce statements that are wrong – or entirely invented (often called a hallucination). It's not lying; it's just being very smooth at writing something that looks right.

The practical takeaway: use LLMs as a fast first draft, a tutor, or a coding assistant – but verify anything important before you trust it.

Verification workflow that works: ask for the reasoning steps, then check the load-bearing claims against a primary source (docs, paper, your own test) – never against another model answer. For code, run it; for facts, open the cited page; for numbers, recompute one row by hand. Frontier agents (GPT-6 Astra, Claude Opus 5.5) hallucinate less on benchmarks but still invent with confidence when context runs thin – our news explainers keep vendor-reported caveats for exactly this reason. Local models hallucinate more on niche facts and less on your own documents – which is why RAG over your files beats a bigger model for private Q&A. The rule never changes: smooth is not true.

Where LLMs live: your machine or someone's cloud

Same machinery, two homes. Cloud (ChatGPT, Claude, Gemini) runs frontier models on massive clusters – best quality, zero setup, metered per token, needs internet, prompts leave your machine. Local (Ollama, LM Studio, llama.cpp on your GPU) runs 7B–35B open models – private, offline, free per token after hardware, quality a step behind frontier autonomy. Most readers end up hybrid: local for private, metered, offline work; cloud for hard reasoning and agents. Full tradeoff with cost math lives in local vs cloud AI; the five-minute path to your first local chat is getting started with local AI.

Summary

A large language model is a next-word predictor. It is scaled up with billions of parameters and trained on enormous amounts of text. Everything it does – writing, reasoning, answering – emerges from that single, simple job.

FAQ

Is a large language model the same as artificial intelligence?

No – an LLM is one kind of AI, trained to predict the next token. AI is the bigger umbrella: vision, robotics, agents, and more.

Do LLMs actually understand what they're saying?

Not like humans. They compute the most likely next token from training patterns – often indistinguishable from understanding, until verification.

How big a model can I run at home?

7B Q4 fits ~5.5 GB, 14B Q4 ~10.5 GB, 35B MoE ~20 GB weights but 7B-like speed. Check how much VRAM and the calculator for your card.

Why do answers stream word by word?

Decode generates one token at a time – streaming is the computation made visible. Prefill absorbs your prompt first; decode drips the answer.

Sources

• llama.cpp – tokens, sampling, and inference engine
• Ollama – running open models locally
• VelsTech Solutions lab benchmarks – tested sizes and speeds
• How much VRAM do you need
• How to get started with local AI
• Local vs cloud AI