advanced · Session 12
Compute complete attention: W_Q, W_K, W_V, scores, √d_head, causal mask, softmax, value mixture, multi-head concatenation, and output projection.
Study this session as a causal investigation. Before every formula or interaction, write what you expect to change and what must remain fixed. During the calculation, retain units, shapes, and intermediate values so an error can be located without restarting at random. After the result, translate the number or state into one sentence about system behavior. Always finish with a counterexample or boundary value. This discipline separates understanding a mechanism from merely recognizing its vocabulary and makes the lab reproducible by another learner. Record the evidence that changed your initial prediction.
Each representation x produces q=xW_Q, k=xW_K, and v=xW_V. Query expresses what the position seeks; key describes how it can be found; value carries information to mix.
Q=XW_Q, K=XW_K, V=XW_V
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
Dot product q·k measures alignment. In “Maya put down the book… She picked it up”, one head can learn that the query for “She” aligns with the key for “Maya”.
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
As d_head grows, dot products can become large and saturate softmax. Dividing by √d_head keeps a more stable scale.
S=QKᵀ/√d_head
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
Before softmax, future positions receive −∞. Softmax turns each permitted row into positive weights summing to 1.
A=softmax(S+causal mask)
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
Output is AV: weights choose how much of each value passes. Scores are not themselves the retrieved content.
O=AV
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
Multiple heads compute different relations, their outputs are concatenated then projected by W_O. During decoding, past K/V are cached; the new query reads that cache without recomputing the prefix.
MHA(X)=Concat(head₁…head_h)W_O
Name the input, transformed state, output, and one required assumption. Then compare your chain with the explanation above.
Reduced trace: q=[2,1], keys k_Maya=[1,1], k_book=[0,2]. The real head has d_head=4; our vectors show only 2 coordinates (a truncated illustration), but the scaling keeps the true dimension. Raw scores: 3 and 2; division by √4=2 gives 1.5 and 1. Softmax ≈ [0.62,0.38]. Output mixes 62% of v_Maya and 38% of v_book. For an earlier position, the causal mask would remove every future key.
Reading method: write the data, state every object shape, perform one transformation, and interpret the result before continuing.
Attention maps alone do not prove a causal explanation of the model’s overall behavior.
Evidence status: Established mechanisms; numerical simplifications are pedagogical.
Each representation x produces q=xW_Q, k=xW_K, and v=xW_V. Query expresses what the position seeks; key describes how it can be found; value carries information to mix.
Dot product q·k measures alignment. In “Maya put down the book… She picked it up”, one head can learn that the query for “She” aligns with the key for “Maya”.
As d_head grows, dot products can become large and saturate softmax. Dividing by √d_head keeps a more stable scale.
Before softmax, future positions receive −∞. Softmax turns each permitted row into positive weights summing to 1.
Scope: Established mechanisms; numerical simplifications are pedagogical.