Loss Functions and Error Signals

By the end of this lesson, you will be able to...

• calculate why binary cross-entropy distinguishes a hesitant correct prediction from a confident wrong one;

• choose a loss from the meaning and shape of a model's target;

• explain why a lower loss is useful evidence but not proof that a model is ready to use.

Idea in one sentence: A loss turns a prediction and its target into a graded numerical signal, so training can distinguish small mistakes from confident mistakes instead of only counting right and wrong labels.

Core Insight

The inspection model from the previous lesson outputs one score for each metal part. A part known to be defective has target y = 1. One current model produces a logit of 0.2; another produces -2.0. If we convert both scores to a binary decision with a 0.5 threshold, the first says defect and the second says pass.

It is tempting to grade the model with only that thresholded answer: correct earns 0, wrong earns 1. This works for reporting an accuracy number after a batch. It breaks as a training signal. A score just above the threshold and a score overwhelmingly above it receive the same “correct” grade. A score barely on the wrong side and a score that is confidently wrong receive the same “wrong” grade. The grade gives no smooth indication of which change would improve the model.

A loss function supplies that missing signal. It is not the prediction and not the business decision. It is a mathematical rule that grades the current output against the target in a way an optimizer can later differentiate. The next lessons explain how that influence travels through the computation graph. Here we first make the grading rule visible.

The Small Problem With a Hard Threshold

For a binary decision, turn a logit z into a number between zero and one with the sigmoid function:

p = sigmoid(z) = 1 / (1 + exp(-z))

For our positive target, a threshold-only rule is:

if p >= 0.5: correct
otherwise:    wrong

That rule is useful when a product needs a yes-or-no action. It is deliberately coarse. It changes only when p crosses 0.5.

Consider three illustrative logits for a part whose true label is 1:

The last two predictions have equal accuracy for this one example, even though 0.881 places much more probability on the correct label than 0.550. Likewise, -2.0 is a more serious error than a score just below zero, but threshold accuracy does not tell training that.

The missing property is a graded objective: a quantity that changes as the score changes. We want it lower when the model assigns more probability to what happened, and higher when it assigns little probability to what happened.

The Better Model: Negative Log-Likelihood

For a binary target y in {0, 1} and predicted probability p, binary cross-entropy (BCE) is:

BCE(y, p) = -(y * log(p) + (1 - y) * log(1 - p))

For a positive target, y = 1, the second term disappears:

BCE(1, p) = -log(p)

Plain meaning: reward the model for assigning high probability to the event that actually occurred. The technical name negative log-likelihood explains the shape: log(p) gets closer to zero as p approaches one, so its negative becomes small; assigning probability near zero to the true event produces a large penalty.

Return to the three illustrative logits. The table applies sigmoid and then BCE for the same positive target:

The values are rounded teaching calculations. Their ordering is the important evidence: moving probability toward the true label lowers the loss, even when the thresholded class does not change. This gives a later gradient calculation information to use on both sides of the decision boundary.

For a negative target, y = 0, the first term disappears instead:

BCE(0, p) = -log(1 - p)

Now a high probability of defect is penalized because the observed target was pass. The two cases are one formula, not two unrelated scoring systems.

Logits, Probabilities, and the Library Boundary

The preceding table shows sigmoid explicitly because it makes the math inspectable. In implementation, a common binary-classification interface accepts the raw logit z and the target y directly. PyTorch calls this BCEWithLogitsLoss; it combines sigmoid with BCE in a numerically stable operation.

This creates a practical boundary worth remembering:

model output:     raw logit z
training loss:    a loss designed to accept logits
human inspection: optionally convert z to sigmoid(z)
decision rule:    choose a threshold based on the task

Do not apply sigmoid and then pass that result to a loss that already expects logits. That changes the intended calculation. Conversely, a raw logit is not itself a probability: z = 2.0 is a score; sigmoid(2.0) ≈ 0.881 is the corresponding binary probability-like output under this model.

The loss evaluates the prediction under the chosen training objective. A threshold, class weighting, and calibration assessment may matter for a deployed inspection policy, but they answer different questions. This lesson keeps them separate on purpose.

Losses Must Match the Target's Meaning

The forward pass gives an output shape. The target tells us what that shape means. Choose the loss from both, not from a familiar formula alone.

For example, a part can be both scratched and chipped. Two independent binary targets match two logits and a binary loss per label. If the task instead requires exactly one category among pass, scratch only, chip only, and both, it is a mutually exclusive four-class problem; use four logits with multiclass cross-entropy.

The words “common” and “fits” matter. Loss choice is a design decision under target semantics, error costs, and data quality. A medical or safety workflow may need class weighting, a different threshold, or additional evaluation because false negatives and false positives do not have equal cost. Changing the loss cannot repair missing labels or a target definition that does not match the real decision.

Why Not Use Mean Squared Error for Everything?

Mean squared error (MSE) compares a prediction and target by squaring their numeric difference. It is a natural starting point when predicting a continuous repair cost or a physical measurement. For a binary probability target, it can be computed, but it describes the error differently from BCE.

For the same positive target in our teaching example, MSE on the sigmoid output gives:

Both columns prefer the more accurate probability. The difference is not that MSE is “illegal.” BCE is the usual fit when the output models a Bernoulli outcome, and its penalty makes extremely confident errors much more visible. The choice also changes the derivatives that the next lesson will trace, so it changes the learning signal as well as the reported number.

Consequences, Trade-offs, and Limits

A loss gives training a continuous objective, which improves on threshold-only feedback. The trade-off is compression: one scalar can summarize the batch's objective while hiding which subgroup, class, or costly failure is getting worse.

Lower training loss is therefore evidence that the current parameters fit the training objective better. It is not proof that the data are representative, the probabilities are calibrated, the chosen threshold serves the product, or the model generalizes to new parts. Compare validation loss and task metrics, inspect the kinds of errors, and keep the label definition under review.

Losses also do not encode every real preference automatically. If a missed defect is much more costly than an unnecessary rejection, that asymmetry must appear in the target policy, weighting, threshold, evaluation, or a combination of them. The signal to watch is a disagreement between a pleasing aggregate loss and the errors that matter operationally.

Common Confusions

Confusion: Accuracy can replace loss during training.

Why it is tempting: accuracy is easy to explain and matches a binary decision.

Better model: accuracy is a thresholded report. It discards how close or confident a prediction is, while a loss supplies a graded objective for optimization.

Confusion: A low loss proves a useful model.

Why it is tempting: training is explicitly trying to reduce it.

Better model: low loss only supports a claim about the chosen objective and data split. Check validation behavior, calibration when relevant, and the error costs of the real decision.

Confusion: Sigmoid plus BCE is the only binary interface.

Why it is tempting: the mathematical formula visibly starts with a probability.

Better model: the mathematics can show sigmoid first, while a stable library interface can accept logits and combine the operations internally. The required input depends on the specific loss API.

Check Your Understanding

Check: A positive example has p = 0.9 in one run and p = 0.55 in another. Which has lower binary cross-entropy, and why?

Think first, then reveal.

Answer: p = 0.9 has lower BCE because -log(0.9) is smaller than -log(0.55). Both are thresholded as positive, but BCE preserves the difference in support for the true label.

Check: A model emits two logits for scratch and edge chip; a part can have both labels. Is this a two-class softmax problem?

Think first, then reveal.

Answer: No. The labels are independent rather than mutually exclusive. Use one binary target and one binary loss contribution for each logit; both outputs may be high.

Practice

A revised inspection system produces one raw logit for each part. For a known defective part, it produces z = -1.0.

1. Compute p = sigmoid(-1.0) to three decimal places.
2. Compute the BCE for y = 1 using -log(p) to three decimal places.
3. Explain why a threshold-only score of “wrong” is less useful for training than this loss.
4. The requirements change: the system must assign exactly one of three mutually exclusive defect categories. State the output shape and common loss family that now fit.

Model answer: sigmoid(-1.0) ≈ 0.269, so BCE(1, p) = -log(0.269) ≈ 1.313. A threshold says only that the prediction is wrong; BCE says how little probability the model assigned to the true label. For exactly one of three categories, emit three logits per example and use multiclass cross-entropy with the target category.

Resources

• [DOCUMENTATION] PyTorch: BCEWithLogitsLoss — Focus: see the stable logits-to-binary-cross-entropy interface and its target contract.
• [DOCUMENTATION] PyTorch: CrossEntropyLoss — Focus: compare its multiclass logits and target requirements with independent binary labels.
• [DOCUMENTATION] PyTorch: MSELoss — Focus: contrast squared numeric error with a classification objective.
• [BOOK] Deep Learning, Chapter 5: Machine Learning Basics — Focus: connect likelihood, loss design, and generalization boundaries.

Key Takeaways

• A loss gives training graded feedback; thresholded accuracy does not distinguish hesitant and confident predictions.
• Binary cross-entropy evaluates the probability assigned to the observed binary label, while a logits-aware API can perform the sigmoid step internally.
• Choose loss and output together from the target's semantics, then validate the resulting objective against the errors that matter.