Day 24: Tensors, Broadcasting, Reshape, and Memory Ledgers

(a) E holds 50000 * 512 = 25,600,000 scalar weights. (b) Output shape (2, 3, 512): each of the 2*3 = 6 token positions is replaced by a 512-vector. (c) The output holds 2*3*512 = 3072 float32 values, i.e. 3072 * 4 = 12,288 bytes. The batch and time axes are new axes the lookup prepends to the row vector.

(a) Trailing axes 3 and 3 match, so the row [10,20,30] is added to each row: [[11,22,33],[14,25,36]]. (b) (2,1) + (1,3) stretches both: [[11,21,31],[12,22,32]], an outer-sum of shape (2,3). (c) Failure: last axes 3 vs 4 are incompatible, libraries raise a broadcast error. (d) (768,) aligns with the channel axis, so bias adds over batch and time, output stays (32,128,768). Broadcasting performs the stretch logically, without materializing the repeated data.

(a) [[1,2],[3,4],[5,6]] - row-major order is preserved, only the axis boundaries move. (b) [1,2,3,4,5,6]. (c) a.T has shape (3,2) but strides a non-contiguous layout. A no-copy .view() is stride-constrained and raises here; .reshape(6) may allocate a copy instead - in NumPy it silently allocates new memory, and in PyTorch .view() would raise while .reshape() succeeds by copying (documented behavior, not executed). (d) 32*128*768 = 3,145,728 elements; x 4 bytes = 12,582,912 bytes = 12 MiB.

(a) Row means are (2.0, 5.0) with shape (2,1) under keepdims. (b) Broadcast subtraction: [[-1,0,1],[-1,0,1]] of shape (2,3) - each row is centered by its own mean. (c) (2,3)+(4,) fails: trailing axes 3 vs 4 mismatch; the other two succeed as (2,3) and (2,3) after stretching the size-1 axes. (d) 2*3*50000 = 300,000 elements x 4 bytes = 1,200,000 bytes (~1.14 MiB). Comparison scope (float32 storage of the tensors named in this lesson only): these logits are about 98 times the (2, 3, 512) lookup output from the first activity (12,288 bytes), but smaller than the (32, 128, 768) hidden state (12,582,912 bytes) and far smaller than the embedding table E itself (25,600,000 weights = 102,400,000 bytes). No total, peak, or training-memory comparison is measured here.