Weight Initialization

By the end of this lesson, you will be able to...

• explain why hidden-layer weights must start differently;

• estimate how fan-in changes the scale of an affine layer's output;

• choose Xavier or He initialization from the activation and inspect its early signal.

Idea in one sentence: Initialization is the first signal-flow decision: it must break symmetry while keeping activations and gradients large enough to learn but small enough to remain usable.

The First Batch

An image classifier receives 32 normalized images and has not yet called its optimizer. The only decisions that can explain its first activation and gradient statistics are the data transform, architecture, loss, and initialization.

Core Insight

Consider a learner starting a ten-layer ReLU classifier on a small image batch. Before the first optimizer step, the notebook shows that 98% of layer-nine activations are zero and its gradient norm is tiny. Reducing the learning rate cannot recover signal that was already lost before the update. The pressure is initialization.

It is tempting to say “random weights solve it.” Randomness does solve one problem: identical hidden units would receive identical inputs, outputs, gradients, and updates forever. But random numbers also have scale. Repeating a slightly too-large or too-small scale through ten layers turns a small first-pass mismatch into exploding values, saturation, or silence.

The stronger model has two separate jobs: use different weights to break symmetry, then select their variance from the number of incoming connections and the activation's behavior. The values below are teaching approximations, not a proof about every architecture.

Why Zero and One Scale Both Fail

For a hidden layer Z = X @ W + b, suppose each input feature has mean near zero and variance near one. One output unit sums fan_in products. If the weights have independent variance s^2, a useful approximation is:

Var(Z) ≈ fan_in * Var(X) * Var(W)
       ≈ fan_in * s^2

With fan_in = 100, weights with standard deviation 1 have variance 1; the output variance is then roughly 100. The next activation receives a very wide signal. With sigmoid or tanh, many values land in a flat saturated region where local derivatives are small. With ReLU, large values can also make later scales hard to control.

At the other extreme, standard deviation 0.01 means s^2 = 0.0001, so the output variance is roughly 0.01. Repeating that shrinkage through layers makes activations and backward signals faint. A small learning rate is not the cause; it only acts after the weak signal has been produced.

All-zero weights are a different failure. Two neurons with the same zero vector compute the same output. Their gradients are the same, so they remain clones after every update. Zero biases can be acceptable; hidden weights need distinct starting values.

The Scale Rules Earn Their Shape

To keep this forward variance near one in the simplified model, choose:

Var(W) ≈ 1 / fan_in

This is the intuition behind fan-in scaling. Xavier/Glorot initialization balances fan-in and fan-out, often using a variance near:

Var(W) ≈ 2 / (fan_in + fan_out)

It is a useful default for activations that pass positive and negative signal more symmetrically, such as tanh. He initialization accounts for ReLU discarding roughly half its inputs in the simplified random model:

Var(W) ≈ 2 / fan_in

For a ReLU layer with fan_in = 100, He gives standard deviation sqrt(2/100) ≈ 0.141; Xavier with equal fan-in and fan-out gives about 0.1. The larger He scale is not random folklore. It compensates for the activation's expected loss of half the signal.

These are starting models, not guarantees. Inputs may not be independent, layers may have normalization, residual connections change paths, and a ReLU can still create many zeros for a particular batch. The point is to match the scale rule to the mechanism that transforms signal.

Why Depth Amplifies a Small Mismatch

The variance estimate is local, but a deep network composes local layers. If each layer makes the typical signal variance half as large, six such layers leave about 1/64 of the original scale. If each layer doubles it, six layers produce about 64 times the scale. The exact values depend on data and activations, but the multiplication explains why a harmless-looking initial constant can become a visible failure only in later layers.

Backward flow has the same concern in reverse. A gradient is repeatedly multiplied by weights and local activation derivatives. Saturated sigmoid values have derivatives near zero; inactive ReLUs have a zero local derivative. When early activation statistics are already extreme, later optimization sees an unreliable credit signal. This is why inspecting only final loss can hide the cause: loss is the last symptom, while activation and gradient histograms show where scale first drifted.

For the image classifier, make a short first-batch ledger:

The numbers in this table are diagnostic categories rather than required targets. A single batch can be unusual. Compare a few deterministic batches, then change one cause at a time: input normalization, initializer scale, or architecture. Do not simultaneously change the optimizer, schedule, and initializer; that removes the evidence needed to identify the mechanism.

A Forward-and-Backward Ledger

Consider three equal-width ReLU layers with width 100. Compare three initializations:

The last row is not a promise of successful training. It is a better starting condition. Inspect activation mean, activation variance, fraction of zero ReLU outputs, and gradient norm by layer before changing optimizers or schedules. Those signals distinguish “the model has not learned yet” from “the model cannot pass useful signal at initialization.”

Choose the Rule from the Layer Contract

Use Xavier when the layer's activation is approximately symmetric around zero and the architecture has no stronger constraint. Use He for ordinary ReLU-family hidden layers. Use the framework's tested implementation rather than retyping a formula when possible, but know what its fan_in, fan_out, gain, and distribution mean.

def he_std(fan_in):
    return sqrt(2.0 / fan_in)

def xavier_std(fan_in, fan_out):
    return sqrt(2.0 / (fan_in + fan_out))

The trade-off is between a simple rule and an architecture-specific one. A universal small random number is easy to write but ignores how width and activation transform variance. A matched rule costs a little more reasoning and still needs inspection, but it gives deep signal propagation a plausible first state.

Where the Rule Stops Helping

Initialization does not repair an incompatible loss, corrupted labels, a learning rate that explodes updates, or excessive regularization. Nor should a healthy initial histogram be confused with good validation performance. It only addresses the state before meaningful learning begins.

There is also a boundary in the variance approximation: it assumes independent, centered inputs and simplified activation behavior. If an input pipeline has a large mean or huge feature-scale imbalance, initialization alone cannot make the first layer healthy. Normalize or inspect inputs as part of the same signal-flow investigation.

Check Your Understanding

Check: Why do two zero-initialized hidden neurons remain redundant after one gradient step?

Answer: They see the same input, produce the same activation, and receive the same gradient. Equal updates preserve equality.

Check: A ReLU network has tiny activation and gradient norms from its first forward/backward pass. What should be inspected before lowering the learning rate?

Answer: Weight scale, fan-in calculation, input scale, and the fraction of zero ReLU activations. The optimizer has not yet had an opportunity to create the weak signal.

Check: Why is He variance larger than Xavier's in an equal-width ReLU layer?

Answer: The simplified He rule compensates for ReLU passing only part of a zero-centered signal, preserving a more usable scale after the activation.

Trace It Yourself

An affine ReLU layer has fan_in = 64 and fan_out = 64.

1. Compute the approximate Xavier and He standard deviations.
2. State which rule is the first choice for this ReLU hidden layer.
3. Name two signals you would log on the first batch to test the choice.

Model answer: Xavier standard deviation is sqrt(2/128) = 0.125; He is sqrt(2/64) ≈ 0.177. Start with He because the activation is ReLU. Log activation variance and the fraction of zero activations; gradient norms by layer are a useful third signal. If values are still pathological, inspect inputs and architecture rather than assuming the formula failed by itself.

Resources

• [TUTORIAL] CS231n: Weight Initialization — Focus: symmetry and variance diagnostics.
• [PAPER] Understanding the difficulty of training deep feedforward neural networks — Focus: Glorot/Xavier initialization.
• [PAPER] Delving Deep into Rectifiers — Focus: He initialization for rectifiers.
• [BOOK] Deep Learning, Chapter 8 — Focus: initialization and optimization conditions.

Key Takeaways

• Random hidden weights break symmetry; their scale determines whether signal can travel through depth.
• Fan-in explains why a constant random scale fails as layers change width.
• Xavier matches more symmetric activations; He compensates for ReLU's signal loss.
• Activation and gradient distributions on the first batch are evidence for an initialization diagnosis.