Autoregressive Sampling: Greedy Decoding vs Temperature Scaling
Agent-Assisted Engineering Check
Prerequisites
Day 48 Decoder-Only Forward Pass and Day 41 Softmax.
Learning Objective
Formulate next-token prediction p_i = softmax(z_i / T) and explain how temperature T controls the entropy of the output distribution.
Instructions
Given logits z = [2.0, 1.0, 0.0]: (1) Compute greedy argmax index 0. (2) Compute probabilities at T=1.0: exp([2,1,0]) = [7.389, 2.718, 1.0], sum=11.107 -> p = [0.665, 0.245, 0.090]. (3) Compute at T=0.5: z/0.5 = [4.0, 2.0, 0.0], exp([4,2,0]) = [54.598, 7.389, 1.0], sum=62.987 -> p = [0.867, 0.117, 0.016]. (4) Explain T -> 0 as greedy argmax and T -> inf as uniform.
Practice & Prediction Log
1. Compute softmax probabilities for z = [2.0, 1.0, 0.0] at T=2.0.
2. Why does low temperature (e.g. T=0.2) reduce hallucinations while increasing repetitive loops?
3. What runtime error occurs if T=0.0 is passed directly into division z / T?
z / 2.0 = [1.0, 0.5, 0.0]. exp([1, 0.5, 0]) = [2.718, 1.649, 1.0], sum=5.367.
At T=2.0, p = [0.506, 0.307, 0.186]. Lower temperature squashes probability mass onto the highest logit, reducing surprise but causing cyclic repetition. T=0.0 causes ZeroDivisionError: float division by zero (code must route T=0 directly to argmax).
Derives temperature scaling analytically and explains the trade-off between diversity and repetition.
Authoritative Source Resources & Provenance
The Curious Case of Neural Text Degeneration
Relevant Section / Range: Section 2.1 Text Generation Decoding Strategies and Section 3.3 Sampling with Temperature
Verified Version & Date: ICLR 2020 / arXiv:1904.09751v2 (Verified: 2026-10-30)
Primary source introducing Nucleus (Top-p) sampling and analyzing the failure of greedy decoding (loops, repetition).
Softmax probabilities, cumulative probability distributions from Days 25-27 and 41-42.
Top-p truncation algorithms and worked probability cutoffs are completely explained in the lesson.
Compared Candidate Sources & Selection Analysis (2)
Introduces top-k sampling; Holtzman et al. dynamic nucleus threshold is compared directly against top-k.
Practical library guide; Holtzman provides the foundational mathematical reasoning.
nanoGPT: sample.py autoregressive generation script
Relevant Section / Range: sample.py: temperature and top_k settings passed to model.generate (model.py: GPT.generate)
Verified Version & Date: Git master (Verified: 2026-10-30)
Compact historical code reference for the named implementation functions; CS336 spring2025 provides the lead component exercises. Do not infer top-p, atomic persistence or cloud budget enforcement from nanoGPT.
PyTorch inference, torch.multinomial, and Day 48 decoder-only forward pass.
The generation loop is implemented in runnable local scripts.
Compared Candidate Sources & Selection Analysis (2)
C++ implementation with high performance; nanoGPT is readable Python matched to our curriculum.
Encoder-decoder model; nanoGPT decoder-only autoregressive loop is the direct fit for modern LLMs.