The CPU loop

A CPU core does one simple job, over and over. It fetches an instruction, works out what the instruction means, does it, and moves on. Everything else in a modern processor exists to make that loop run faster.

The big idea

Each trip around the loop has four steps:

1. Fetch the instruction at the address in the PC.
2. Decode it: which operation is it (add, load, compare, jump), and which operands does it need?
3. Execute it: do the arithmetic, read or write memory, or change the PC to jump elsewhere.
4. Advance the PC to the next instruction and repeat.

The operands live in registers, small storage slots inside the core. x86-64 has 16 general-purpose 64-bit registers. ARM64 has 31. Reading a register costs almost nothing. Reading main memory costs about 100 ns, which is roughly 400 cycles at 4 GHz. That gap is why memory dominates real performance.

Walking through it: overlapping the work

If the core finished one instruction before starting the next, most of its circuits would sit idle. Modern cores avoid that in two ways.

Pipelining works like a laundromat. While load 1 is in the dryer, load 2 is in the washer and load 3 is being sorted. A textbook pipeline has 5 stages. Each instruction still takes 5 cycles from start to finish, but once the pipe is full, one instruction completes every cycle. Modern x86 and ARM cores are roughly 14 to 20 stages deep.

Superscalar, out-of-order execution goes further. A core has several execution units and can issue several instructions in one cycle. High-end cores decode 6 to 8 instructions per cycle. The core also looks ahead through hundreds of instructions, runs any whose inputs are ready, and uses a reorder buffer to make results appear in program order. Your program cannot tell the difference.

The number to know is IPC, instructions per cycle:

Peak budget:            4 GHz × IPC 2 = 8 billion simple instructions per second per core
Tight numeric loop:     IPC 3 to 4
Typical backend code:   IPC below 1, because it waits on cache misses

A low IPC does not mean the core is lazy. It means the core is starved for data. When someone says a workload is "memory-bound", this is what they mean.

Common mistakes

• Reading IPC below 1 as "the CPU is slow". The core is usually waiting on data, not computing badly.
• Believing clock speed alone sets speed. Pipeline width, cache misses and branch mispredictions matter as much.
• Counting a line of Python as one cheap instruction. Each line expands into many machine operations.

Check yourself

• Why can a 5-stage pipeline finish one instruction per cycle when each instruction takes 5 cycles? Each stage works on a different instruction, so once the pipe is full, one finishes every cycle.
• A service runs at IPC 0.6 on a 4 GHz core. Is the core idle? No. It is mostly stalled, usually waiting for memory.

Key takeaways

• A core fetches, decodes and executes instructions, paced by a clock of about 3 to 5 GHz.
• Pipelining overlaps instructions across stages. Superscalar, out-of-order execution runs several at once.
• IPC measures how much work each cycle does. Low IPC usually means waiting on memory.
• The core is fast. Your code is often slow because it cannot feed the core data in time.

Further learning

Next, see how compilers, interpreters and JITs turn your source into these instructions in compiled, interpreted and JIT.

Tom Scott's explainer shows the fetch-execute cycle with a simple animation.