Feedforward Propagation and Layered Representation

By the end of this lesson, you will be able to...

• trace a batch through an affine layer, ReLU, and final score layer;

• predict the shape of each value in a small feedforward network;

• explain why a hidden activation is a representation for the next layer, not a final answer.

Idea in one sentence: A forward pass is a sequence of visible state changes: each layer rewrites the current representation, and the final layer reads the last rewrite as a prediction.

Core Insight

An inspection service receives two measurements for each metal part: normalized scratch severity and normalized edge damage. A model with two hidden layers is drawn on a whiteboard. It is tempting to read the drawing as “input goes in, answer comes out.” Then an implementation reports a shape error, or produces two numbers whose meaning is unclear. The boxes suddenly feel less helpful.

The reasonable first model is that a layer is a black box and shapes are bookkeeping for a library. That model works for copying a short architecture diagram. It fails as soon as we need to ask a concrete question: which values does one hidden unit see, which examples belong to a batch, and which number is the prediction?

The forward pass answers those questions. It is the model's actual computation at inference time and the set of intermediate values training will later use to assign credit. We will trace two parts through one hidden layer. The numbers are deliberately small and hand-chosen; they illustrate the mechanism, not the result of a trained production model.

The Moving Parts

For a batch of B parts with D input features, use the row-per-example convention:

X:  (B, D)       input batch
W1: (D, H)       weights into H hidden units
b1: (H,)         one bias value per hidden unit
Z1: (B, H)       hidden preactivations
A1: (B, H)       hidden activations
W2: (H, O)       weights into O output units
b2: (O,)         one bias per output unit
Z2: (B, O)       output scores, often called logits in classification

The first layer computes:

Z1 = X @ W1 + b1
A1 = ReLU(Z1)

@ means matrix multiplication. The inner dimensions must agree: (B, D) @ (D, H) produces (B, H). The bias has shape (H,), so its H values are added to every row of the batch. This repeated addition is a broadcasting convention, not a new learned bias for each part.

The second layer repeats the same pattern:

Z2 = A1 @ W2 + b2

At this point Z2 is a score. Whether it is left as a logit or passed through sigmoid, softmax, or another output function depends on the task semantics introduced in the previous lesson. Here the important fact is simpler: the output layer reads A1, not the original raw features directly.

Frameworks sometimes store a linear layer's weight with the axes reversed from this written convention. For example, PyTorch's Linear(in_features, out_features) exposes a weight shaped (out_features, in_features) and applies the equivalent operation internally. Do not memorize one storage layout as the concept. Track what each axis means and make the multiplication valid.

A Batch Trace, One State at a Time

Take two illustrative parts. Rows are examples; columns are the two input features:

X = [ [0.8, 0.2],       part P: more scratch evidence
      [0.2, 0.9] ]      part Q: more edge-damage evidence

shape(X) = (2, 2)

Our hidden layer has three units. Its toy parameters are:

W1 = [ [ 1.0, -1.0,  0.5],
       [ 0.5,  1.0, -0.5] ]

b1 = [ -0.4, -0.3, 0.1 ]

shape(W1) = (2, 3)
shape(b1) = (3,)

First compute the affine hidden scores. For part P, the first hidden unit receives:

z1[P, 1] = 0.8*1.0 + 0.2*0.5 - 0.4
          = 0.5

Doing the same calculation for every row and hidden unit gives:

Z1 = X @ W1 + b1
   = [ [ 0.50, -0.90,  0.40],
       [ 0.25,  0.40, -0.25] ]

shape(Z1) = (2, 3)

Apply ReLU element by element. It keeps positive values and replaces negative values with zero:

A1 = ReLU(Z1)
   = [ [0.50, 0.00, 0.40],
       [0.25, 0.40, 0.00] ]

shape(A1) = (2, 3)

This is the first representation change. In this teaching model, the second hidden unit responds only for the edge-heavy part Q, while the third responds only for the scratch-heavy part P. A trained network might distribute the pattern differently; these labels are an interpretation of deliberately chosen numbers, not a promise that every hidden unit has a neat human name.

Now use a one-unit output layer:

W2 = [ [1.2],
       [0.8],
       [0.3] ]
b2 = [-0.5]

Z2 = A1 @ W2 + b2
   = [ [0.22],
       [0.12] ]

shape(Z2) = (2, 1)

For P, the output score is 0.50*1.2 + 0.00*0.8 + 0.40*0.3 - 0.5 = 0.22. The calculation is not “three hidden predictions averaged together.” It is a new weighted combination of the hidden features. If this were a binary classifier, a later output step could turn each score into a binary probability; the next lesson will ask how training grades such a prediction.

So far, one batch has moved through four states:

raw measurements X (2, 2)
  -> hidden scores Z1 (2, 3)
  -> hidden activations A1 (2, 3)
  -> output scores Z2 (2, 1)

The forward pass is just this ordered computation. “Deep” adds more rewrites, not a different kind of magic.

What the Hidden Layer Has Changed

The prior lesson showed that an activation prevents stacked affine layers from collapsing into one affine map. This trace makes that claim observable. A1 retains three numbers per part, but those numbers do not merely repeat the two inputs. ReLU has removed some responses and kept others, so the output layer receives a different coordinate system.

This is what a learned representation means in the modest, useful sense: values shaped for a later decision. It does not mean that every hidden unit is a clean, stable concept, nor that larger hidden layers automatically learn better representations. The network only earns that representation through its weights, data, objective, and training process.

The shape is part of the mechanism. A layer with H = 3 hidden units produces three activations for each example, which is why W2 needs three input rows. If W2 were shaped (2, 1), it could not read A1 of shape (2, 3) under this convention. The error is informative: the architecture says the layers disagree about how many features exist at their boundary.

There is a small consistency check hidden in the batch notation. Run P alone as an input of shape (1, 2) with the same W1, b1, W2, and b2. Its final score should still be 0.22, now held in an array of shape (1, 1). Putting P beside Q in a batch changes how many rows are processed, not the calculation within P's row. If P's score changes when Q is merely added to the batch, inspect the code for an unintended operation across the batch axis. Some layers intentionally use batch-level information in training; this simple affine-plus-ReLU example does not.

Where the Simple Story Breaks

“Each layer turns inputs into better features” is useful, but incomplete.

First, the same batch axis must remain aligned with the labels. Swapping (B, D) for (D, B) can yield an immediate multiplication error, or, in some coincidental dimensions, a computation that runs while treating examples as features. A successful call is not proof that axes mean what you intended.

Second, a hidden activation is input-dependent. P and Q activate different units. Reusing A1 from P when computing Q would mix two examples and produce a score that belongs to neither. Vectorized code is fast because it processes rows together, not because it can forget which row belongs to which part.

Third, a forward pass alone cannot say whether 0.22 is good. It only says what the current parameters compute. Loss functions and targets provide the next pressure: how should the model compare that output with the inspection decision it ought to make?

Costs, Limits, and Signals

Batching gives one clear benefit: the same matrix operations can process many examples at once. The trade-off is that notation hides semantics easily. A compact expression can be numerically valid while using the wrong axis, stale preprocessing, or an output activation mismatched to the task.

Layered representations also buy flexibility, but they cost interpretability. A simple toy trace lets us name the units. In a larger network, evidence may be distributed across many dimensions, and no single activation deserves a story by itself.

Useful signals to inspect before blaming an optimizer are:

• the shape at every layer boundary;
• a single row traced by hand on toy parameters;
• summary statistics or a small sample of each activation, especially unexpected all-zero layers;
• whether the same preprocessing and feature order are used at training and inference.

These checks do not prove the model generalizes. They establish a narrower but essential fact: the function being trained is the function you think you implemented.

Common Confusions

Confusion: The input layer is a layer that learns features.

Why it is tempting: diagrams label it as a layer.

Better model: the input is the current representation supplied by the dataset. The first parameterized transformation creates the first learned hidden representation.

Confusion: A hidden activation is already a class probability.

Why it is tempting: it is a number produced inside the network.

Better model: it is an intermediate feature. Its meaning comes from how later layers use it; output semantics require a task-specific final design.

Confusion: If two arrays can be multiplied, the architecture must be right.

Why it is tempting: no runtime error feels like success.

Better model: valid dimensions are necessary, not sufficient. You must also know which axis is batch, feature, hidden unit, or output.

Check Your Understanding

Check: With X shaped (8, 4) and a hidden layer of 6 units, what shapes should W1, b1, Z1, and A1 have under the convention in this lesson?

Think first, then reveal.

Answer: W1 is (4, 6), b1 is (6,), and both Z1 and A1 are (8, 6). The eight examples stay as rows; each receives six hidden values.

Check: In the worked trace, why is A1[P, 2] zero even though P has nonzero input features?

Think first, then reveal.

Answer: Its preactivation is -0.90. ReLU acts on the weighted score after the bias, not directly on whether the original inputs were nonzero.

Trace It Yourself

An engineer changes the hidden layer from three units to four while keeping the input batch shape (5, 2) and one output score per example.

1. Give the new shapes of W1, b1, A1, W2, and Z2 using this lesson's row-per-example convention.
2. A teammate leaves W2 at shape (3, 1). Explain the failure in terms of the representation boundary, not merely “dimension mismatch.”
3. Name one inspection step that would catch a swapped batch and feature axis before training many epochs.

Model answer: W1 becomes (2, 4), b1 is (4,), and A1 is (5, 4). The output must read four hidden values, so W2 is (4, 1) and Z2 is (5, 1). A (3, 1) output matrix expects a three-value representation, while the new hidden layer provides four values for each part. A hand trace of one known row together with assertions or logs for each tensor shape can expose a swapped axis early.

Resources

• [DOCUMENTATION] PyTorch: Linear — Focus: compare its in_features and out_features contract with the matrix convention used in this lesson.
• [DOCUMENTATION] PyTorch: ReLU — Focus: confirm that ReLU applies an element-wise rectifier.
• [DOCUMENTATION] PyTorch: Broadcasting semantics — Focus: inspect why a bias vector can be applied across batch rows.
• [BOOK] Deep Learning, Chapter 6: Deep Feedforward Networks — Focus: connect the small trace to the general feedforward form.

Key Takeaways

• A forward pass is an ordered trace of representations, not a black-box jump from input to answer.
• Shapes encode the boundary between layers and must preserve the meaning of batch, feature, hidden, and output axes.
• Hidden activations are input-dependent features for a later layer; they are not automatically human concepts or final predictions.