FLOP counting for matrix multiplies

Compute costs time and money, and most of it is matrix multiplies. A FLOP (floating-point operation) is one add or one multiply. Count the FLOPs of each matrix multiply and you get a quick estimate of how long a layer or a training run will take. The formula is short enough to memorize.

The big idea

Multiplying an (m, k) matrix by a (k, n) matrix produces m · n outputs. Each output is a dot product of length k, which needs k multiplies and about k adds. Counting each multiply-add as 2 operations gives:

FLOPs = 2 · m · n · k

The shared dimension k is the one that gets summed away. It appears once in the count.

Walking through it

Multiply (1024 × 4096) by (4096 × 1024). Here m = 1024, k = 4096, n = 1024:

FLOPs = 2 · 1024 · 4096 · 1024
      = 2 · 2¹⁰ · 2¹² · 2¹⁰
      = 2³³
      ≈ 8.6 × 10⁹  (about 8.6 GFLOPs)

Now a transformer-sized case. A linear layer maps width 4096 to 16384 (four times wider), applied to 2048 tokens. The token count plays the role of m:

m = 2048, k = 4096, n = 16384
FLOPs = 2 · 2048 · 4096 · 16384 = 2³⁸ ≈ 2.75 × 10¹¹

Turn FLOPs into time by dividing by the throughput of your hardware. On a hypothetical accelerator that sustains 100 TFLOP/s on this matmul, the layer takes 2.75 × 10¹¹ / 10¹⁴ ≈ 2.75 ms.

Dense matrix multiplies dominate transformer compute. That is why accelerators are built around them, and why training cost estimates count FLOPs. The roofline model adds memory traffic to this picture.

Common mistakes

• Mixing conventions. Some papers count a multiply-add as one operation, others as two. A factor of 2 can change your estimate. Check which one a source uses.
• Using the wrong m, n or k. k is the shared inner dimension. For a batch of tokens, m is the number of tokens.
• Stopping at FLOPs. Small batches can be limited by moving memory, not by arithmetic. FLOP counts alone do not predict speed.

Check yourself

1. How many FLOPs are in (100 × 200) · (200 × 50)? Answer: 2 · 100 · 200 · 50 = 2,000,000, or 2 MFLOPs.
2. If you double the batch size, what happens to the FLOPs of a linear layer? Answer: They double, because m doubles. The weights do not change.

Key takeaways

• Count a matrix multiply as 2mnk FLOPs.
• The inner dimension k is summed away, and m · n is the number of outputs.
• Divide FLOPs by hardware throughput for a first time estimate.
• FLOPs ignore memory traffic, which can dominate at small batch sizes.

Further learning

Next, see why the starting scale of weights matters for these layers: orthogonality and initialization.