context window
The most tokens one request can hold: instructions, history, documents and the answer, together.
Like the memory one function call is allowed.
Walkthrough coming
See also: token
GenAI terms, in plain words
Every term the walkthroughs use, with what it means, what it's like if you already know software, a tiny example, and where it's taught properly. The same cards open when you tap a word inside a walkthrough.
How an LLM works
KV cache
Memory a model server uses to keep its attention results for text it has already processed, so it doesn't compute them again. Long prompts and many users fill it quickly.
Like a build cache that lives in GPU memory.
sampling
How the model picks each next token from many likely ones. Settings such as temperature control how much randomness it uses, so the same input can give different answers.
Like a dial between "same answer every time" and "brainstorm".
token
A chunk of text the model reads and writes, often part of a word. Limits and prices are counted in tokens, not words.
Like bytes on a network bill.
Walkthrough coming
Prompting
schema
The rules for a structured answer: which fields, which types, which values are allowed.
Like a table definition in a database.
priority: urgent | standard | fast_track
structured output
Making the model answer in a fixed, machine-readable shape, such as JSON that matches a schema.
Like a form instead of a free-text letter.
{"priority": "urgent", "reason": "..."}
Taught in: Build an Eval Harness You Can Ship On, step A3 →
See also: schema
RAG
BM25
A classic keyword ranking formula. It scores documents by how often query words appear, and gives more weight to rarer words.
Like the ranking behind a traditional search engine.
HNSW
A common vector index that links vectors in a layered graph, so a search can jump quickly to close matches. It's fast, but usually keeps much of its data in memory.
Like taking an express train, then a local one, to reach a nearby stop.
Tools and agents
agent harness
Everything around the model that lets an agent run: the loop, session state, sandbox, tool connections and tracing. A managed harness is one a vendor runs for you.
Like a managed runtime for your agent.
Taught in: Rent or Build the Agent Loop, step B1 →
agent loop
The code that calls the model, runs any tools it asks for, passes back the results, and repeats until the task is done.
Like a while loop where the model decides what happens next.
Taught in: Rent or Build the Agent Loop, step A1 →
compaction
Shrinking older conversation turns, often by summarizing them, so a long run still fits in the context window.
Like squashing old commits but keeping the recent ones.
Taught in: Rent or Build the Agent Loop, step A5 →
idempotency key
A unique ID sent with a request so that repeating it has no extra effect. The second call returns the first result.
Like a payment API's protection against double charges on a retry.
Taught in: Rent or Build the Agent Loop, step B3 →
session
The saved state of one agent run or conversation: its messages and tool results, and sometimes its files. It lets work resume after an interruption.
Like a checkpoint file for a long-running job.
Taught in: Rent or Build the Agent Loop, step A2 →
Evals
baseline
A saved record of which cases passed with the last good version, to compare a change against.
Like a snapshot test.
baselines/v2.json
calibration
Checking a judge against grades people already gave, and trusting it only if it agrees often enough.
Like checking a new test against known bugs before relying on it.
22 of 25 agree → 88%
CI gate
A check in the build pipeline that blocks a change from merging when it fails.
Like a required status check on a pull request.
exit 1 → merge blocked
confusion matrix
A table counting each label against each answer the model gave, so you see which mistakes it makes, not just how many.
Like a test report broken down by test category.
urgent row: 6 called urgent, 3 called standard
eval harness
A test suite for an AI feature. It runs the model on labeled cases, grades every answer and reports how many passed.
Like pytest, where each expected value came from a person.
grader
Code that decides whether one answer matches its label.
Like an assert.
"urgent" == "urgent" → pass
judge
A second model that grades free text against a written rubric (LLM-as-a-judge). Trust it only after calibration.
Like a code reviewer with a checklist.
{"verdict": "fail", "rule_violated": "uninhabitable_home"}
Taught in: Build an Eval Harness You Can Ship On, step C2 →
See also: rubric calibration
label
The answer a person says is right for one case.
Like the expected value in a unit test.
C-1042 → urgent
majority vote
Running each case several times and counting it as passed when most runs pass.
Like rerunning a flaky test before you believe it.
✓ ✓ ✗ → pass
margin of error
How far a score could be off just because you tested a sample of cases, not every case there will ever be.
Like the luck in a 100-question exam.
74% on 100 cases → about ±9 points
pass rate
The share of cases the grader passed.
Like the score on an exam.
74 of 100 → 74%
precision
Of the cases the model gave a label (say urgent), the share that really have it.
Like: of the alarms that rang, how many were real fires.
Walkthrough coming
See also: recall
recall
Of the cases that really have a label (say urgent), the share the model gave that label.
Like: of the real fires, how many alarms rang.
6 of 9 urgent caught → 67%
Taught in: Build an Eval Harness You Can Ship On, step B2 →
See also: precision
regression
A case that passed before a change and fails after it, even if the total went up.
The same as in any test suite.
rubric
The written rules a grader or judge applies, specific enough that two people would grade the same way.
Like a code review checklist.
urgent if the home can't be lived in tonight
Security
Bring your own cloud (BYOC)
A vendor's software runs inside your cloud account while the vendor operates it. What stays in your account varies by vendor.
Like a managed appliance installed in your own data center.
Customer-managed key (CMEK)
An encryption key you control in your own key service, which the vendor uses to encrypt your data. Revoking it cuts off access to that data.
Like keeping the key to a storage unit you rent.
Identity provider (IdP)
The system that signs in your staff and vouches for who they are, such as corporate single sign-on.
Like the front desk that issues everyone's badge.
sandbox
An isolated environment for running code the model wrote. It limits what that code can reach, but secret and network limits depend on its setup.
Like a throwaway container, locked down as far as you configure it.
Taught in: Rent or Build the Agent Loop, step A4 →
Zero data retention (ZDR)
An arrangement where a vendor doesn't store prompts and responses after returning the answer, apart from exceptions in its terms. It usually has to be approved, and some features don't qualify.
Like a no-logging policy on a proxy, set by contract rather than by config.
Cost and latency
Model-as-a-Service (MaaS)
Offering a model to others through a hosted API. Some open-weight licenses set separate terms for businesses that do this.
Like reselling software as a hosted product.
MTok
One million tokens, the usual unit for model prices. $2 per MTok means $2 per million tokens.
Like price per GB on a storage bill.
Provisioned Throughput Unit (PTU)
Azure's unit for reserving fixed model capacity, instead of paying per token.
Like reserved instances compared with on-demand.
Time to first token (TTFT)
How long a user waits before the first word of a streamed answer appears.
Like time to first byte on a web request.
No terms match. Try a shorter word or pick All.