Day 27: Information Theory: Entropy, Cross-Entropy, and Language-Model Loss

Bits: $0.5\to1$, $0.25\to2$, $0.125\to3$, $0.8\to0.3219$, $0.1\to3.3219$. Nats: $0.6931, 1.3863, 2.0794, 0.2231, 2.3026$. $2$ bits $=1.3863$ nats. Smaller $p$ means larger surprisal, so an unlikely token costs roughly ten times more loss than a likely one - the asymmetry that punishes overconfident wrong predictions.

(a) $H = 0.5+0.5+0.375+0.375 = 1.75$ bits ($\approx1.2130$ nats). (b) $H = \log_2 6 \approx 2.5850$ bits. (c) The die is more uncertain; entropy uses only the probabilities, so relabeling outcomes does not change it.

(a) $H(p,q)=2$ bits, so $D_{KL}(p\|q)=2-1.75=0.25$ bits. (b) $D_{KL}(p\|q)\approx0.0294$ bits vs $D_{KL}(q\|p)\approx0.0290$ bits: not equal - KL is asymmetric, direction matters. (c) NLLs $0.2231, 0.6931, 1.3863$ nats; mean $\approx0.7675$ nats; perplexity $e^{0.7675}\approx2.1544$. (d) One zero-probability observation makes the loss diverge to $+\infty$; clipping $q\ge\varepsilon$ bounds the damage (e.g. $-\ln 10^{-9}\approx20.72$ nats) and smoothing prevents zeros upstream.

(a) $H(p)\approx1.8464$ bits. (b) $H(p,q)=2$ bits; $D_{KL}=2-1.8464\approx0.1536$ bits. (c) Mean NLL $=(0.6931+0.9163+2.3026)/3=1.3040$ nats; perplexity $e^{1.3040}\approx3.6840$. (d) $-\ln 0=\infty$ so one unlucky token makes the whole-corpus loss diverge; add-1 smoothing or an $\varepsilon$ floor removes literal zeros.

Post-attempt numeric self-check (only after preserving the second unaided submission):

Replace the eight None placeholders below only with plain numeric values from the already-preserved second unaided submission. Enter them in this order: (a) entropy in bits; (b) cross-entropy in bits, then KL in bits; (c) three per-token NLL values in nats in the listed token-score order, mean NLL in nats, then perplexity (unitless). Use plain numbers rounded to four decimal places. The absolute tolerance is 0.00005 (half of one four-decimal unit). The checker reports only pass/fail for (a), (b), and (c); it does not grade (d) or produce an overall score, mastery judgment, or certification.

Part (d) remains a separate manual post-attempt rubric: state that an observed q=0 makes mean NLL infinite; name a concrete safeguard that assigns positive probability; and state that it changes the scoring distribution. There is no automated semantic grading or English keyword gate. Timing, preservation, unaided conduct, and deliberate premature checker execution are policy-only; no hidden UI enforcement exists.

import math

def verify(submission):
    failed = {'(a)': 'fail', '(b)': 'fail', '(c)': 'fail'}
    if (type(submission) is not dict
            or any(type(key) is not str for key in submission)
            or set(submission) != {'a', 'b', 'c'}):
        return failed
    p = (0.55, 0.20, 0.15, 0.10)
    q = (1 / len(p),) * len(p)
    scores = (0.25, 0.5, 0.2)
    entropy = -math.fsum(pi * math.log2(pi) for pi in p)
    cross_entropy = -math.fsum(pi * math.log2(qi) for pi, qi in zip(p, q))
    kl = math.fsum(pi * math.log2(pi / qi) for pi, qi in zip(p, q))
    nll = tuple(-math.log(score) for score in scores)
    mean_nll = math.fsum(nll) / len(nll)
    targets = {'a': (entropy,), 'b': (cross_entropy, kl),
               'c': nll + (mean_nll, math.exp(mean_nll))}
    def matches(values, expected):
        if type(values) not in (list, tuple) or len(values) != len(expected):
            return False
        for value, target in zip(values, expected):
            if type(value) not in (int, float):
                return False
            try:
                if (not math.isfinite(value) or value < 0
                        or abs(value - target) > 0.00005):
                    return False
            except OverflowError:
                return False
        return True
    return {'(' + part + ')': 'pass' if matches(submission[part], targets[part])
            else 'fail' for part in ('a', 'b', 'c')}

After the function, replace the placeholders using only that preserved submission, then run submission={'a':[None], 'b':[None,None], 'c':[None,None,None,None,None]} and print(verify(submission)).