Perceptron Foundations

By the end of this lesson, you will be able to...

• trace how inputs, weights, bias, and a threshold produce a binary prediction;

• calculate one perceptron update and explain how it changes the decision boundary;

• recognize when a task cannot be solved by one linear separator.

Idea in one sentence: A perceptron learns a straight decision boundary by moving that boundary whenever a labeled example lands on the wrong side.

Core Insight

Consider a camera that inspects small metal parts. For each part, software extracts two normalized features:

• x1: scratch severity;
• x2: edge-damage severity.

The system must make one binary decision: reject or pass.

A first approach might use fixed rules:

reject if scratch severity is high
reject if edge damage is high
otherwise pass

That works while one feature alone decides the case. It becomes awkward when moderate scratch evidence and moderate edge-damage evidence should combine. We need one rule that can weigh several signals together and learn from labeled examples.

A perceptron provides exactly that small model. It produces a score from weighted inputs, adds a bias, and compares the result with zero. Its simplicity is useful because both its learning behavior and its limit are visible.

From Two Signals to One Decision

Let the input for one part be:

x = (x1, x2)

The perceptron stores:

w = (w1, w2)   weights
b              bias

It computes the score

s = w1*x1 + w2*x2 + b

and applies a hard threshold:

if s >= 0: predict +1  (reject)
if s <  0: predict -1  (pass)

The labels +1 and -1 are a convenient teaching choice. They make the update rule compact. A software interface could expose the same classes as 1 and 0.

Suppose:

w = (1.0, 0.5)
b = -2.0
x = (1.5, 2.0)

The score is:

s = 1.0*1.5 + 0.5*2.0 - 2.0
  = 1.5 + 1.0 - 2.0
  = 0.5

Because 0.5 >= 0, the perceptron predicts reject.

This calculation is the whole forward path:

two feature values
    -> multiply each by its weight
    -> add the contributions
    -> add the bias
    -> compare with zero
    -> choose one class

The score is not a probability. A score of 4 does not mean “four times more likely” than a score of 1. It only says that the point is farther on the positive side according to this model.

The Boundary Hidden Inside the Formula

The decision changes where the score equals zero:

w1*x1 + w2*x2 + b = 0

With two features, this equation describes a line. Points on one side produce positive scores; points on the other side produce negative scores.

For the previous parameters:

1.0*x1 + 0.5*x2 - 2.0 = 0

We can solve for x2:

x2 = 4 - 2*x1

That is the decision boundary. The weights control its orientation and how strongly each feature changes the score. The bias shifts where the boundary crosses the feature space. If b were zero, the line would be forced through the origin.

This is a simplified geometric model. It assumes the two extracted features already carry useful information and use compatible scales. If scratch severity ranges from 0 to 1 but edge damage ranges from 0 to 10,000, the second feature can dominate updates merely because of its units. Feature scaling is therefore part of the model boundary, not cosmetic cleanup.

Learning by Correcting a Mistake

The initial weights do not need to be correct. The perceptron changes them when an example is misclassified.

For a labeled example (x, y), with y equal to +1 or -1, use:

if y*s <= 0:
    w <- w + eta*y*x
    b <- b + eta*y

Here eta is the learning rate. It controls the size of the correction.

Why does the sign of y help?

• If a true reject example has a negative score, then y = +1. The update adds part of x to the weights and raises the bias. That pushes this example toward the positive side.
• If a true pass example has a positive score, then y = -1. The update subtracts part of x and lowers the bias. That pushes this example toward the negative side.

The rule does not ask which individual feature “caused” the mistake. It moves the boundary using the whole current example.

Worked Update: Move One Part Across the Line

Assume a simpler starting model:

w = (1.0, 1.0)
b = -3.0
eta = 0.5

The next inspected part has:

x = (1.0, 1.0)
y = +1  (the human inspector says reject)

First calculate the score:

s = 1.0*1.0 + 1.0*1.0 - 3.0
  = -1.0

The model predicts pass, but the label is reject. The example is on the wrong side.

Now apply the update:

w' = w + eta*y*x
   = (1.0, 1.0) + 0.5*(+1)*(1.0, 1.0)
   = (1.5, 1.5)

b' = b + eta*y
   = -3.0 + 0.5*(+1)
   = -2.5

Run the same part through the updated model:

s' = 1.5*1.0 + 1.5*1.0 - 2.5
   = 0.5

It now lands on the reject side.

The update corrected this example. It did not prove that every other part is now classified correctly. Moving a line for one example may move it closer to or farther from other examples. Training therefore cycles through labeled examples and keeps checking mistakes.

For linearly separable data, the classic perceptron convergence result says that repeated updates reach a separating boundary under its standard assumptions. If the classes cannot be separated by one line, updates can continue without finding a perfect solution. That difference is a property of the representation and model class, not necessarily a software bug.

So far, the model is no longer mysterious. It is a line, a side-of-line decision, and a correction that moves the line.

The Failure That One More Update Cannot Fix

Now consider XOR:

The two reject points occupy opposite corners. The two pass points occupy the other corners.

Try to draw one straight line with both reject points on one side and both pass points on the other. Any line that separates one diagonal pair splits the other pair incorrectly. Changing the weights rotates the line. Changing the bias shifts it. Neither operation turns one line into the bent or multi-region boundary XOR needs.

This is the exact place where the initial model breaks:

one weighted sum + one threshold = one linear boundary

More training cannot make the model class express a boundary it does not contain. The next lesson introduces nonlinear activation functions, which let multiple units transform the input space before the final decision.

Consequences, Trade-offs, and Limits

The perceptron buys three useful properties:

• every prediction can be traced to feature contributions;
• every update is small and inspectable;
• the decision boundary has clear geometry.

The cost is limited expressiveness. One perceptron only represents a linear split in its input features.

It can also fail for reasons other than XOR:

• poor features may hide the pattern;
• incompatible feature scales may distort updates;
• noisy or contradictory labels may prevent perfect separation;
• the hard threshold does not produce calibrated probabilities.

The signal to watch is not only training accuracy. If mistakes persist, inspect where the points lie and whether one line could separate them at all. This separates an optimization problem from a representation problem.

Common Confusions

Confusion: The score is a probability.

Why it is tempting: larger positive scores sound like greater confidence.

Better model: the score is a signed linear value. Its sign selects the class; its magnitude is not automatically calibrated as probability.

Confusion: If an update fixes the current example, training must be improving globally.

Why it is tempting: the before-and-after calculation shows a successful correction.

Better model: the update moves one shared boundary. It may change predictions for many examples, so the dataset must be checked again.

Confusion: A perceptron that fails on XOR needs a better learning rate.

Why it is tempting: training problems are often blamed on tuning.

Better model: no learning rate can place one straight line around diagonal classes. The missing capability is nonlinear composition.

Check Your Understanding

Check: A perceptron has w = (2, -1), b = -1, and receives x = (2, 1). What does it predict?

Think first, then reveal.

Answer: The score is 2*2 + (-1)*1 - 1 = 2. Because the score is non-negative, it predicts +1.

Check: The same example's true label is -1. In which direction should the update move the score for this example?

Think first, then reveal.

Answer: Downward. Multiplying the input and bias correction by y = -1 subtracts from the current positive evidence.

Practice

A content filter uses two normalized features. Its current parameters are:

w = (0.5, -0.5)
b = 0
eta = 0.2

It receives x = (2, 1) with true label y = -1.

1. Calculate the current score and prediction.
2. Apply one perceptron update.
3. Calculate the new score for the same example.
4. Explain what this result proves and what it does not prove.

Model answer:

current score = 0.5*2 + (-0.5)*1 + 0 = 0.5
current prediction = +1

w' = (0.5, -0.5) + 0.2*(-1)*(2, 1)
   = (0.1, -0.7)

b' = 0 + 0.2*(-1)
   = -0.2

new score = 0.1*2 + (-0.7)*1 - 0.2
          = -0.7
new prediction = -1

The update proves that this example moved to the correct side of the current boundary. It does not prove that the whole dataset is separable or that predictions on other examples improved.

Connections

• Logistic regression also begins with a linear score, but it combines that score with a smooth probabilistic model and a different training objective.
• The next lesson shows why stacking only linear transformations is insufficient and how nonlinearity expands the boundaries a network can represent.

Resources

• [PAPER] The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain — Focus: Rosenblatt's original 1958 formulation and the historical scope of the model.
• [DOC] scikit-learn Perceptron — Focus: the modern linear-classifier interface, weights, intercept, iterations, and constant learning-rate implementation.
• [BOOK/TUTORIAL] Neural Networks and Deep Learning, Chapter 1 — Focus: the transition from perceptrons to multilayer networks in a worked teaching sequence.

Key Takeaways

• A perceptron turns several feature values into one signed score and one binary decision.
• The weights orient the linear boundary; the bias shifts where that boundary sits.
• A mistake-driven update moves the current example toward the correct side, but must be evaluated against the rest of the data.
• Persistent mistakes can reveal a representation limit, not just an optimization problem.
• XOR makes the boundary visible: one perceptron cannot express a nonlinear split.