Learning FlashAttention the Hard Way. Part 1: The Algebraic Foundation [D]

I'm writing a short series of tutorials on FlashAttention: from theory to efficient CUDA kernels.

Part 1 is the theoretical foundation. It walks through a modern algebraic formalism showing that FlashAttention is an associative operation, which lets you treat it as a regular reduction on the GPU and apply all the same scheduling optimizations. Some recent MLSys and CVPR papers lean on this framing, and I find it much more powerful than the original.

Overview:

• Safe softmax, Welford's variance, and FlashAttention are the same secretly-associative operation
• The twisted monoid (transport of structure), why the max-rescale coupling doesn't break associativity
• The qk_scale = log2(e)/√D you already see in FA-2 and the Triton kernels, derived from scratch
• Numerical analysis: overflow bounds, error limits, and why tiling never amplifies error
• Bird's 3rd Homomorphism Theorem as a test for whether any loop is secretly associative