Parameter counting, width and depth

How big is this network? The answer is a count of every weight and bias. That count gives you the memory needed for the weights, and it hints at the compute. Once you can count parameters, the question of depth versus width becomes concrete.

The big idea

For a network with layer widths d₀ → d₁ → … → d_L, the parameter count is:

params = sum over layers of (d_ℓ × d_(ℓ−1) + d_ℓ)

The first term is the weight matrix, and the second is the bias vector.

Depth and width do different things:

• Depth stacks nonlinear layers. Each layer reuses the features built by the one before, so the model builds layered features.
• Width adds more units in parallel within a layer. Each unit adds capacity for one more pattern.

The universal approximation theorem says one hidden layer, given enough width and a suitable nonlinearity, can approximate any continuous function on a bounded set. That is a statement about existence, not a training recipe. The width required can be enormous, and gradient descent may never find the weights. In practice, depth usually gets there with far fewer parameters.

Walking through a count

Take a classifier for flattened 28 × 28 images, with widths 784 → 256 → 128 → 10:

layer 1: 256 × 784 + 256 = 200,960
layer 2: 128 × 256 + 128 =  32,896
layer 3:  10 × 128 +  10 =   1,290
total                    = 235,146

In 32-bit floats, that is about 235,146 × 4 bytes ≈ 940 KB of weights. A 7-billion-parameter model in 16-bit floats needs about 14 GB for its weights alone.

The widest matrix dominates. The first layer holds almost all of the parameters here, because 784 inputs times 256 units is the largest product.

Now compare doubling the width. Keep the input at 784 and the output at 10, and change the hidden layer from 256 to 512:

784 → 256 → 10:  203,530 parameters
784 → 512 → 10:  407,050 parameters

The count roughly doubles, because the 784-input layer dominates.

Width, depth and compute

A linear layer costs about 2 · batch · d_in · d_out FLOPs. Widening every layer by 2× multiplies the compute of each layer by about 4, since both its input and output double. Adding a layer adds one more matrix multiply, so doubling the depth roughly doubles the compute. If latency is the constraint, depth and width have different costs, and you should check both.

Common mistakes

• Counting parameters as memory. Training needs more: gradients, optimizer states and stored activations.
• Widening a model when latency is the limit. Width costs compute roughly with the square of the change.
• Ignoring the widest layer. It usually sets most of the memory and compute.

Check yourself

1. How many parameters does a layer with 100 inputs and 10 outputs have? Answer: 100 × 10 + 10 = 1,010.
2. You double the depth of a stack of equal-width layers. Roughly how do the FLOPs change? Answer: They roughly double, since each added layer adds one matrix multiply.

Key takeaways

• The parameter count is the sum of each weight matrix plus its bias.
• Depth builds layered features, and width adds parallel capacity.
• The widest layer usually dominates memory and compute.
• Widening every layer raises compute much faster than adding layers does.

Further learning

Scaling laws use these counts to predict how quality changes with model size and data: scaling laws.

Back to the overview: neural networks.