Definitions written for someone choosing a tool, not sitting an exam. Each entry says what the term means and — more usefully — what it changes about cost, hardware or how badly things fail.
39 terms · 7 groups
A large language model — a system trained on text to predict what comes next.
Everything a chat assistant does rests on next-token prediction at scale. That single mechanism explains both the fluency and the failure modes: the model is optimising for plausible continuation, not for being right.
The learned weights inside a model, usually counted in billions.
A 7B model has roughly seven billion weights. Parameter count sets the floor on memory: at 4-bit quantisation you need about half a byte per parameter, so 7B needs roughly 4GB before any context is loaded. It is a rough proxy for capability and a precise one for hardware.
An architecture where only part of the model runs on each token.
A mixture-of-experts model may hold hundreds of billions of parameters but activate only a fraction per token. It is fast for its size, but you still need memory for the whole thing — which is why MoE models look cheap on paper and expensive on your GPU.
A model whose weights you can download and run yourself.
Open weights is not the same as open source: you may get the file without the training data, the training code, or permission to use it commercially. Read the licence before you build on it.
Further training of an existing model on your own examples.
Fine-tuning changes the weights, so it shifts style and format reliably. It is the wrong tool for adding facts — for that, put the facts in the prompt or retrieve them. Most teams reach for fine-tuning too early and for retrieval too late.
Training a small model to imitate a larger one.
A distilled model is cheaper and faster and keeps much of the behaviour on the tasks it was distilled for. It usually keeps less of the general reasoning, which is why a distilled model can look excellent in a demo and thin on your actual work.
The unit a model reads and writes — roughly three quarters of a word in English.
Pricing, context limits and speed are all measured in tokens, not words or characters. Code and non-English text tokenise less efficiently, so the same page of text can cost noticeably more in one language than another.
How much text a model can consider at once, input and output together.
A one-million-token window does not mean the model uses all of it well. Attention degrades over long inputs, and cost grows with what you send, so a large window is an option you pay for rather than a feature that works for free.
The decline in answer quality as a conversation grows long.
Relevant details get buried among irrelevant ones and the model starts contradicting its earlier self. Starting a fresh conversation with a clean summary usually beats pushing a long one further.
Standing instructions that apply to the whole conversation.
It sets role, tone and hard rules. It is instruction rather than law: a determined user prompt can push against it, which is why a system prompt is not a security control.
Showing the model two or three worked examples instead of describing the task.
Usually the cheapest quality win available. Examples pin down format and edge cases far more reliably than adjectives — one good example beats a paragraph of "be concise and professional".
How much randomness the model uses when picking each next token.
Low values make output repeatable and dull; high values make it varied and less reliable. For extraction and code, keep it low. Note that some newer models drop the setting entirely.
Forcing the answer to match a schema, usually JSON.
The difference between "please reply in JSON" and a real schema constraint is that the second one cannot fail to parse. If a program consumes the answer, use the constraint.
Running a trained model to produce an answer.
Training happens once and costs a fortune; inference happens every time anyone uses the thing and is what actually shows up on your bill.
Storing weights at lower precision so the model fits in less memory.
Going from 16-bit to 4-bit cuts memory roughly fourfold for a modest quality loss. Below 4-bit the loss stops being modest. It is the single technique that decides whether a model runs on your machine.
Memory on the graphics card — the usual hard limit for running a model locally.
You need room for the weights, the KV-cache and a runtime buffer at the same time. If it does not fit, it does not run; there is no graceful degradation, just an out-of-memory error.
Memory holding the conversation so far, so it is not recomputed each token.
It grows with context length, which is why a model that loads happily can still run out of memory partway through a long chat. It is the reason "it fits" and "it fits with the context I actually use" are different claims.
The common file format for locally-run models.
One file holds the weights plus the metadata a runtime needs. The file size on the repository is the honest starting point for whether it will fit — more honest than the parameter count in the name.
Generation speed — how fast text appears.
Bounded by memory bandwidth far more than by raw compute, which is why an older card with fast memory can beat a newer one with slow memory. Anything above reading speed feels instant; below it feels broken.
The standard way hosted models are priced, input and output billed separately.
Output typically costs several times more than input. Comparing headline numbers is misleading unless you know your own input-to-output ratio — a summarisation workload and a code-generation workload price out completely differently on the same model.
Reusing an already-processed prefix instead of paying for it again.
When a long system prompt or document repeats across calls, caching can cut input cost dramatically. It only helps if the repeated part comes first and stays byte-identical.
Confident output that is simply untrue.
Not a bug to be patched but a consequence of predicting plausible text. It is most dangerous where the answer looks like a fact: names, numbers, citations, API signatures. Anything of that kind needs checking against a source.
Tying an answer to supplied source material.
A grounded answer can be checked, because you know what it was allowed to read. Ungrounded answers can only be trusted or not — and a model asked about something absent from its sources will often fill the gap rather than decline.
Retrieval-augmented generation — fetch relevant documents, then answer from them.
The standard way to give a model knowledge it was not trained on. Answer quality is usually limited by the retrieval step, not the model: if the right passage is not fetched, no amount of model quality recovers it.
A numeric representation of text where similar meanings land close together.
The mechanism behind semantic search and retrieval. Embeddings from different models are not comparable, so switching model means re-embedding everything you stored.
Storage built for finding the nearest embeddings quickly.
The retrieval half of RAG. For a few thousand documents an ordinary database with a vector column is usually enough; the specialised systems earn their keep at a scale most projects never reach.
When test questions have leaked into a model’s training data.
The score goes up without the ability going up. It is the main reason a model can top a leaderboard and disappoint on your own work, and the reason a private evaluation set is worth more than any public number.
Letting a model call functions — search, code execution, an API.
The step from text generator to something that can act. The model chooses which tool to call and with what arguments; your code decides what actually runs, which is where the safety boundary belongs.
A loop where the model plans, acts through tools, observes and repeats.
Agents suit tasks that are hard to specify in advance and easy to verify afterwards. Where verification is slow or absent — SEO, strategy, anything judged weeks later — a long autonomous run produces volume rather than progress.
A coding assistant that reads the repository, edits files and runs tests on its own.
It works well because the feedback loop is closed inside the machine: the compiler and the test suite tell the model whether it was right, thousands of times, without asking a human.
Model Context Protocol — a shared way to expose tools and data to assistants.
One server can serve any client that speaks the protocol, instead of writing an integration per assistant. It is plumbing, and its value is that it is boring and reusable.
Hostile instructions hidden in content the model reads.
A web page or document can carry text aimed at the model rather than the reader. Once a model has both untrusted input and real capabilities, treat everything it reads as untrusted — the model cannot reliably tell data from instructions.
An image or video generator that starts from noise and removes it step by step.
More steps generally means more detail and more time. It is the architecture behind most current image tools, and the reason generation speed and quality trade against each other so directly.
A description of what you do not want in the output.
Supported by most image generators and by almost no text models. Often faster than rewriting the positive prompt when a specific artefact keeps appearing.
Generating moving footage from a written description.
Clips are short, expensive and hard to control precisely; consistency between shots is the open problem. Usable for short inserts and B-roll rather than anything needing continuity.
Reproducing a specific voice from a short sample.
Technically straightforward now, legally not. Consent and likeness rights vary by country and the rules are moving faster than the tools.
The terms under which you may use a model’s weights.
Ranges from genuinely permissive to "research only" to permissive-until-you-get-large. It is the first thing to check before building a product on an open-weights model, and the thing most often skipped.
Whether a provider may use your inputs to improve its models.
Defaults differ between consumer and business tiers of the same product, and can be changed by settings you have to find. If you send anything confidential, the answer needs to come from the terms, not from an assumption.
How long a provider keeps what you send.
Separate from whether they train on it. Retention exists for abuse monitoring and support, so "we do not train on your data" and "we delete your data" are two different promises.
There is no entry for anything we could not explain in terms of a decision someone actually makes. A glossary that defines every term in the field is a glossary nobody finishes reading — so this one covers what changes a choice between tools, and stops. If a term you needed is not here, the contact page is the fastest way to get it added.