02 How LLMs Actually Work and How to Pick One/2.2 Decoding Parameters and Embeddings: Controlling Output and Representing Meaning
§ 2.2section 2 of 5·~12 min
Decoding Parameters and Embeddings: Controlling Output and Representing Meaning
The previous section ended with the model producing a score for every token in its vocabulary. This section covers two things built on top of that machinery:
- Decoding: how those scores become an actual token, and which knobs you control.
- Embeddings: how models turn text into vectors that capture meaning. Embeddings power semantic search, RAG, deduplication and clustering.
1. From scores to a token
At each step the model outputs logits, one raw score per vocabulary token. A softmax turns those logits into a probability distribution. A decoding strategy then picks one token from that distribution.
logits -> (temperature) -> softmax -> (top-k / top-p filtering) -> sample or pick max -> next token
The knobs
| Parameter | What it does | Practical effect |
|---|---|---|
| Greedy (always pick the top token) | Deterministic choice of the most likely token | Repeatable but can be repetitive or get stuck in loops on long outputs |
| temperature | Divides the logits before softmax. Below 1 sharpens the distribution, above 1 flattens it. At or near 0 it behaves like greedy. | Low values give focused, consistent output. High values give more varied output and more errors. |
| top_k | Keeps only the k most likely tokens, then samples | Cuts off the long tail of unlikely tokens. Mostly exposed by open-model runtimes. |
| top_p (nucleus sampling) | Keeps the smallest set of tokens whose probabilities add up to p, then samples | Adapts to the model's confidence. Few candidates when it is sure, more when it is unsure. |
| max_tokens | Hard cap on generated tokens | Controls cost and latency. Output is cut off mid-sentence if the cap is too low. |
| stop sequences | Ends generation when a given string appears | Useful for templated outputs and for avoiding extra text |
| frequency / presence / repetition penalties |