Activation Functions and Nonlinearity

By the end of this lesson, you will be able to...

• show why any stack of affine layers without an activation is still one affine model;

• trace how a ReLU hidden layer gives a network a nonlinear representation;

• choose an output activation from the meaning of a classification or regression task.

Idea in one sentence: An activation function changes a layer's score before the next layer sees it, so depth can create decision structures that one straight boundary cannot express.

Core Insight

Return to the metal-part inspection camera from the previous lesson. Its two features are scratch severity and edge damage. A single perceptron can reject parts on one side of a straight line. It fails, however, when the required rule has separated regions: perhaps a part is rejected when exactly one of two fault indicators is present, but passed when neither or both are present. That is the XOR pattern.

The tempting repair is “add layers.” Imagine that the first layer combines the two signals, then the second layer combines the first layer's result. If both layers only perform weighted sums plus biases, the extra boxes in the diagram have not added a new kind of decision. They have only re-expressed one straight boundary.

An activation function is the missing operation. It lets an intermediate unit respond differently in different ranges of its input. The next layer can then combine those new responses. This lesson establishes that representational change. The next lesson follows the forward pass through such a network in more detail.

Why Linear Depth Collapses

Write the first layer as an affine transformation:

h = W1*x + b1

If the next layer is also only affine, its score is:

s = W2*h + b2
  = W2*(W1*x + b1) + b2
  = (W2*W1)*x + (W2*b1 + b2)

The final line has the same form as one layer: one matrix multiplying x, plus one bias. We could rename the combined terms W and b and write s = W*x + b. Adding a third or twentieth affine-only layer changes the values of those combined terms, not the class of functions the network can represent.

This is why “deep” does not automatically mean “nonlinear.” Biases do not solve the problem either; they translate a boundary but do not bend it. For a binary score based on two original features, the decision boundary remains a line.

There is still a reason to use a linear model when the relationship is plausibly linear: it is smaller, easier to inspect, and often needs less data. The failure appears when the task genuinely needs several regions or a curved boundary. XOR gives a compact example of that failure, not a claim that every practical task looks like four points in a square.

Insert a Nonlinear Gate

Now place an activation after the first affine layer:

z = W1*x + b1
h = ReLU(z)
s = W2*h + b2

ReLU, short for rectified linear unit, applies this rule to each component:

ReLU(z) = max(0, z)

A negative preactivation becomes exactly zero; a positive one passes through unchanged. Because this step is nonlinear, it generally cannot be folded into the second matrix and bias. Different inputs can turn different hidden units on or off. The final layer is still linear in the hidden representation h, but h is now a nonlinear function of the original features x.

That distinction is the useful mental model:

input features -> affine scores -> nonlinear hidden features -> final score

The hidden features are not yet human concepts such as “scratch” or “chip.” They are learned numerical detectors. Sometimes one becomes interpretable; often the useful representation is distributed across many units. The important point is that the network can first change the coordinate system in which the final linear decision is made.

Worked Path: Build XOR from Two Hidden Features

Use binary inputs only for this worked construction. Let 1 mean a fault indicator is present and 0 mean it is absent. We want to reject parts with exactly one indicator:

Create two hidden values:

h1 = ReLU(x1 + x2 - 0.5)
h2 = ReLU(x1 + x2 - 1.5)

h1 becomes positive when at least one indicator is present. h2 becomes positive only when both are present. Combine them with:

s = h1 - 3*h2 - 0.25
predict reject when s >= 0; otherwise predict pass

Trace the four cases:

This is a hand-built demonstration, not a training recipe. A real model would learn parameters from examples, and it need not learn these exact hidden features. The demonstration shows the capability that was absent before: the hidden layer separates “at least one” from “both,” and the output layer combines those two conditions into the desired rule.

Which Activation Fits Which Job?

Activations in hidden layers mainly create usable representations. Activations at the output mainly express the kind of answer the task asks for. The functions below are common choices, not universal defaults.

For example, an inspector that must select exactly one status from pass, scratch, or edge chip can produce three final scores and apply softmax. Raising the score for one class changes the normalized distribution across the others.

If a part can have both a scratch and an edge chip, the labels are not mutually exclusive. Two independent sigmoid outputs are often a better semantic match: one estimates the scratch label and one the chip label. A ReLU output would be a poor replacement in either case because a nonnegative number is not automatically a probability or a normalized class distribution.

For regression, such as predicting a continuous repair cost, the final layer is often left unsquashed so it can represent the target range. A bounded activation is appropriate only when the target itself has a known bound and the overall loss and target transformation justify it.

Trade-offs and Limits

Nonlinearity buys expressive power: several hidden units can partition the input space and let later layers combine the resulting regions. The trade-off is that the model is no longer reducible to one boundary, so its behavior, training, and activation choice deserve attention.

ReLU is simple and commonly effective in hidden layers, but it has a sharp boundary at zero. If a unit's preactivation is negative for all relevant examples, its ReLU output is always zero. That is a useful warning sign, not proof that every zero output is a bug. Sigmoid and tanh are smooth but flatten toward their extremes; their local derivatives become small there, which can make learning through many such units harder.

No activation solves a data problem by itself. A nonlinear network can still underfit if it is too small, overfit if it memorizes limited data, or fail because features and labels do not support the requested prediction. Activation choice is part of a model design, not a guarantee of generalization.

Common Confusions

Confusion: Several linear layers make a nonlinear network.

Why it is tempting: the diagram has many transformations and parameters.

Better model: composing affine transformations gives another affine transformation. An activation between layers is what prevents that collapse.

Confusion: ReLU produces a probability because it removes negative values.

Why it is tempting: its output is nonnegative.

Better model: ReLU outputs an unbounded nonnegative score. A probability requires more than being nonnegative, and a multiclass distribution also requires the components to have the right joint meaning.

Confusion: Softmax is the right output for every classification task.

Why it is tempting: it is often shown in classifier diagrams.

Better model: softmax fits mutually exclusive classes. Independent labels, such as “has scratch” and “has chip,” need outputs that can both be high.

Confusion: A nonlinear activation guarantees a good model.

Why it is tempting: it fixes the XOR limitation.

Better model: it expands the hypothesis class. Whether the learned model is accurate and useful still depends on data, architecture, training, and evaluation.

Check Your Understanding

Check: A network computes h = W1*x + b1 and then s = W2*h + b2, with no activation between them. Can it represent XOR just because it has two layers?

Answer: No. Substituting h into s produces one affine expression in x. The model is still equivalent to one linear decision boundary in the original features.

Check: In the worked XOR construction, what is the score for (1, 1) and why is it a pass?

Answer: h1 = ReLU(1 + 1 - 0.5) = 1.5 and h2 = ReLU(1 + 1 - 1.5) = 0.5. Therefore s = 1.5 - 3*0.5 - 0.25 = -0.25, which falls on the pass side.

Practice

A quality system assigns one of three mutually exclusive inspection outcomes: pass, scratch, or edge chip. Its final layer produces three scores.

1. Which output activation should the system consider first, and why?
2. The requirements change: a part may have both a scratch and an edge chip. What changes in the output design?
3. Explain why replacing either design with ReLU would not supply the same output meaning.

Model answer: For the first task, use three scores followed by softmax because the three outcomes compete: exactly one is intended. For the revised task, use independent outputs—commonly one sigmoid per label—because both labels may be present. ReLU only constrains scores to be nonnegative. It neither normalizes a mutually exclusive distribution nor represents independent label probabilities by itself.

Resources

• [PAPER] Deep Sparse Rectifier Neural Networks — Glorot, Bordes, and Bengio (2011) — a primary source on rectifier networks and sparse zero-valued representations.
• [DOCUMENTATION] PyTorch: ReLU — concise reference for the element-wise rectifier operation.
• [DOCUMENTATION] PyTorch: Softmax — reference for converting a vector of scores into normalized components.
• [BOOK] Deep Learning, Chapter 6: Deep Feedforward Networks — formal background on hidden units and feedforward networks.

Key Takeaways

• A stack of affine layers without an activation is still one affine model.
• A hidden activation creates a nonlinear representation that later layers can combine.
• Choose output activations from label semantics: mutually exclusive classes, independent labels, and continuous targets are different jobs.