GeekCoding101 make your way to geek

Weight Initialization Explained: Xavier vs He

What weight initialization is, why it prevents vanishing and exploding gradients, and when to use Xavier, He or orthogonal init, with PyTorch code.

7 min read
Daily Ai Insight Issue 13 20241215
On this page1. What Is Weight Initialization?

Weight initialization is how you set a neural network’s weights before training starts, and it decides whether gradients flow cleanly through the layers or vanish or explode on the way. The short rule is to use He (Kaiming) initialization for ReLU-family activations, Xavier (Glorot) for tanh or sigmoid, and to never start all weights at the same value, such as zero.

This is part of my Daily AI Insight series, so it’s a casual explainer, not a textbook chapter. I’ve kept the formulas to one small table so you can see what’s going on without wading through derivations.

a diagram of a weight initialization

1. What Is Weight Initialization?

Weight initialization is the process of assigning initial values to the weights of a neural network before training begins. These weights determine how neurons are connected and how much influence each connection has. Training will adjust them, but their starting points have a big impact on how well the network can learn.

Think of it as choosing your starting point for a journey:

  • A good starting point (proper initialization) puts you on the right path for a smooth trip.
  • A bad starting point (poor initialization) can lead to delays, detours, or getting lost altogether.

2. Why Is Weight Initialization Important?

The quality of weight initialization affects three things during training.

Training speed

  • Poor initialization slows learning by causing redundant or inefficient updates.
  • Good initialization speeds up convergence, so the model learns faster.

Gradient behavior

  • Vanishing gradients: if weights start too small, the signal shrinks a little at every layer. By the time gradients reach the early layers they are close to zero, and those layers barely update.
  • Exploding gradients: if weights start too large, the signal grows at every layer. Gradients become huge and training turns unstable.

The goal of every method below is the same. Keep the variance of activations and gradients roughly constant from layer to layer.

Final model performance

A well-initialized network is more likely to reach a good solution. A poorly initialized one may get stuck in a suboptimal solution or fail to train at all.

3. Everyday Examples of Weight Initialization

Example 1: The Zero Trap

Imagine you’re training a network to tell cats from dogs. If all weights start at zero, every neuron in a layer computes the same value and receives the same update. The network can never learn different features, like “whiskers” for cats or “tail shapes” for dogs.

It’s like asking a group of people to vote when everyone always gives the same answer, so no progress can be made. This happens with any identical starting value, not just zero. It’s called the symmetry problem, and it’s why weights need randomness.

Example 2: Random Chaos

Now suppose the weights are random but far too large. The network becomes chaotic, like a classroom where everyone is shouting different answers at once. Gradients become uncontrollable and learning collapses.

Example 3: The Sweet Spot

With properly scaled random values, the network starts on stable footing. It’s like giving each voter clear instructions. Everyone brings a different but manageable input to the table, and the group can reach a consensus.

4. Common Weight Initialization Methods

Random initialization

You assign small random values, for example from a normal or uniform distribution.

  • Pro: it breaks symmetry, so neurons don’t learn identical features.
  • Con: a fixed range that is too wide or too narrow makes training unstable or slow, especially as networks get deeper. The methods below fix this by scaling the range to each layer’s size.

Xavier (Glorot) initialization

Xavier initialization scales weights by both the number of inputs and the number of outputs of a layer, to keep gradient flow balanced in both directions.

  • Best for: networks with smooth, symmetric activations like tanh or sigmoid.
  • Benefit: helps gradients propagate without vanishing or exploding.

He (Kaiming) initialization

He initialization is tailored for ReLU activation functions.

  • Why it works: ReLU outputs zero for every negative input, so roughly half the signal is dropped at each layer. He initialization doubles the variance to make up for that, which keeps enough neurons active during training.
  • Best for: deep networks with ReLU or its variants (Leaky ReLU and similar).

LeCun initialization

LeCun initialization scales by the number of inputs only, with a smaller variance than He.

  • Best for: SELU activations (self-normalizing networks), and as a classic default for tanh-style layers.

Orthogonal initialization

Orthogonal initialization starts with weights that form an orthogonal matrix.

  • Pro: an orthogonal matrix preserves the length of the vectors it multiplies, so signals neither shrink nor grow as they pass through the layer. Different directions in the weight space stay independent.
  • Best for: very deep networks and recurrent networks (RNNs, LSTMs), where the same weight matrix is applied many times.

The formulas at a glance

fan_in is the number of inputs to a layer and fan_out is the number of outputs. For a uniform distribution, the bound is ±sqrt(3 × variance).

MethodWeight variancePairs well withPyTorch (torch.nn.init)
LeCun1 / fan_inSELU, tanhkaiming_normal_(w, nonlinearity="linear")
Xavier / Glorot2 / (fan_in + fan_out)tanh, sigmoidxavier_uniform_, xavier_normal_
He / Kaiming2 / fan_inReLU, Leaky ReLUkaiming_uniform_, kaiming_normal_
Orthogonal— (orthogonal matrix, optional gain)deep nets, RNNsorthogonal_

Which one should I use?

Your activationStart with
ReLU, Leaky ReLU, GELU-styleHe (Kaiming)
tanh, sigmoidXavier (Glorot)
SELULeCun
Recurrent weight matricesOrthogonal
Not sure / using framework defaultsDefaults are usually fine. Revisit if training is unstable

5. Weight Initialization in PyTorch

Frameworks already initialize layers for you. Setting it yourself is a few lines:

import torch.nn as nn
def init_weights(m):
if isinstance(m, nn.Linear):
# He / Kaiming for ReLU layers
nn.init.kaiming_normal_(m.weight, nonlinearity="relu")
nn.init.zeros_(m.bias)
model = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Linear(256, 10),
)
model.apply(init_weights)
# Xavier for a tanh layer
tanh_layer = nn.Linear(256, 128)
nn.init.xavier_uniform_(tanh_layer.weight, gain=nn.init.calculate_gain("tanh"))
# Orthogonal, common for recurrent weight matrices
rnn_like = nn.Linear(128, 128)
nn.init.orthogonal_(rnn_like.weight)

Biases are usually started at zero. Zero biases don’t cause the symmetry problem, because the random weights already make each neuron different.

6. Practical Challenges and Optimizations

Challenges

  • Different needs: different architectures and activation functions call for different initialization methods. A one-size-fits-all approach rarely works.
  • Very deep networks: in extremely deep networks, even good initialization may not keep gradients stable on its own.

Optimizations

  • Pair initialization with the activation function. He with ReLU, Xavier with tanh. This is the single most useful habit.
  • Use normalization layers. Batch Normalization or Layer Normalization can soften the effects of a poor starting point.
  • Fine-tune by hand when needed. In some cases, experimenting with the initialization range of specific layers gives better results.
  • Watch the first steps of training. If the loss goes to NaN or barely moves from the start, check initialization before you touch anything else.

7. One-Line Summary

Weight initialization is the starting point for a neural network’s training journey. Proper initialization helps the model learn efficiently, avoid gradient issues and reach better performance.

FAQ

Why can’t I initialize all weights to zero? Every neuron in a layer would compute the same output and get the same update, so they would stay identical forever. The same symmetry problem happens with any constant value. You need randomness to break it.

What’s the difference between Xavier and He initialization? Xavier scales the variance by 2 / (fan_in + fan_out) and suits tanh or sigmoid. He scales it by 2 / fan_in, which is larger, because ReLU zeroes out about half its inputs. Use He for ReLU-family activations.

Do I need to set initialization myself in PyTorch or Keras? Usually not. Both frameworks give their built-in layers a sensible default initialization. Setting it explicitly is worth it when you use a specific activation, build a very deep network, or see unstable training.

Does Batch Normalization make initialization irrelevant? No, but it makes training much more forgiving. Normalization re-scales activations as they pass through the network, which reduces the damage a poor starting point can do. A sensible initialization still helps.

How should biases be initialized? Zero is the common choice. The random weights already break symmetry, so zero biases don’t cause the symmetry problem.

Final Thoughts

Weight initialization might look like a small step in the deep learning pipeline, but it’s a critical factor in whether training succeeds. The next time you train a network, check that your initialization matches your activation function. That one choice can make or break your model’s performance. Stay tuned for more AI insights, and let’s keep exploring together!

Thanks for reading! You can find my other posts in the Daily AI Insight series, and you can reach me via the contact page.

Thanks for reading my blog post! Feel free to check out my other posts or contact me via the social links in the footer.

Keep reading

All posts