FlashAttention-2 and 3

FlashAttention's first paper fixed the IO problem: tile Q, K and V through SRAM, keep online softmax state, never write the n × n map. Later versions do not change that contract. They change how the work is cut up on newer chips, and which precisions the inner loop can use.

The big idea

Think of FA1 as the recipe, FA2 as a better kitchen layout, and FA3 as new appliances that only exist in a later building. Same dish. Different time to the table.

The picture breaks if you leave FA2 running on Hopper or Blackwell when FA3 and FP8 are available. New silicon does not help a kernel that never uses its async copy units.

Walking through it

You rarely call these APIs yourself in serving. Engines select an attention backend. The generation still matters when release notes say "FA3 required for peak FP8 decode."

Training code should pin a known-good version. PyTorch scaled_dot_product_attention dispatches among math, mem-efficient, and flash backends. "Whatever SDPA picked" is not a reproducible experiment when CUDA and PyTorch minor versions move under you.

As of when this was written, if you are on Hopper or Blackwell and the stack has FA3 enabled, you usually want it for both speed and low-precision paths. Upgrading from A100 to H100 or B200 without checking the attention path leaves throughput on the table.

FlashDecoding is a different axis: it is about decode's missing query parallelism, not about GPU generation. A serving engine may run FA3-style prefill and a split-KV decode backend in the same replica.

Common mistakes

• Reading "FlashAttention" in a benchmark and assuming FA3. It might be FA2, FlashInfer, or a vendor kernel.
• Leaving FA2 on new silicon. The IO idea is the same. The Hopper async path is not.
• Treating SDPA as a pinned kernel. Log which backend ran.

Check yourself

1. What did FA2 change, and what did it not change? Answer: Work partitioning and inner-loop overhead. Not the exact softmax math.
2. You move a serving stack from A100 to H100 and TPOT barely moves. What should you check first? Answer: Whether FA3 and the FP8 attention path are actually enabled.

Key takeaways

• FA2 and FA3 keep exact attention. They change mapping and hardware use.
• FA2 is the common default. FA3 is the Hopper/FP8 kernel.
• Pin and log the backend. SDPA is a dispatcher, not one kernel.
• Pair the right generation with the right phase: FA-style prefill, split-KV decode.

Further learning

FlashAttention-2Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock n…arXiv.org

Back to the FlashAttention hub for the tiling picture, or on to quantization if you are chasing FP8 end to end.