Support Vector Machines and Margin

By the end of this lesson, you will be able to...

• compare separating boundaries by their margin rather than by training separation alone;

• identify the support vectors that constrain an SVM boundary;

• explain how soft-margin C, feature scaling, and an RBF kernel change the model's bias.

Idea in one sentence: An SVM chooses a boundary that leaves the widest useful safety corridor between classes, while explicitly deciding how much training error that corridor may tolerate.

Core Insight

Suppose a course platform must classify two handwritten answers, 3 and 8, from two measured features: loop width and vertical height. Many straight lines separate the clean training examples. A line that merely separates them is not automatically a trustworthy choice: a small measurement error can flip a digit near that line.

The initial model says “any line with zero training mistakes is equally good.” Margin supplies the missing test: prefer the line with the largest distance to the closest examples on both sides. Those closest examples are the support vectors. They hold the corridor in place.

A Boundary Is More Than a Sign Test

For a linear classifier, write a score as:

f(x) = w · x + b
predict class + when f(x) > 0; class − when f(x) < 0

w gives the direction perpendicular to the boundary and b moves it. The set f(x)=0 is the decision boundary. The sign says which side a digit is on, but the size of the score matters too: a point close to zero is geometrically close to the boundary.

For the teaching picture, consider two possible separators of the same points:

Both classify the training points correctly. B has the larger margin, so a small shift in a borderline digit has more room before crossing the boundary. This does not prove that B will always generalize better; margin is a useful inductive bias, not a guarantee about future data. It is most persuasive when features, labels, and the evaluation split represent the future decision.

The Few Points That Set the Line

Imagine moving an easy 3 far from the boundary. As long as it stays outside the corridor, the maximum-margin line need not move. Move a point sitting on the corridor edge, however, and the line must move or the margin shrinks.

easy 3       support 3 | boundary | support 8       easy 8
   o              o    |    0     |    x               x

The points touching the margin are support vectors. In the ideal hard-margin case, their constraints can be written as:

yᵢ (w · xᵢ + b) ≥ 1

where yᵢ is +1 or −1. The arbitrary 1 fixes the scale; maximizing the geometric margin is equivalent to minimizing ||w||² / 2 while satisfying those constraints. The important intuition is not the optimization proof: nearby examples constrain the boundary; distant easy examples usually do not.

Check: If an additional 8 is far from the boundary and clearly on the 8 side, must it become a support vector?

Answer: No. It may add evidence, but it does not usually constrain the widest corridor. A point becomes decisive when it presses on the margin or violates it.

Real Data Requires a Soft Margin

Handwritten features overlap. A mislabeled example or an ambiguous 3 may sit inside the other class’s region. A hard-margin rule would either fail or contort the line to satisfy every point.

Soft-margin SVM introduces nonnegative slack ξᵢ:

minimize  1/2 ||w||² + C Σ ξᵢ
subject to yᵢ(w · xᵢ + b) ≥ 1 − ξᵢ

The first term wants a wide, simple margin. The second charges for violations. C decides their exchange rate.

• Lower C: accept more violations to preserve a wider, simpler boundary.
• Higher C: penalize violations more heavily and may accept a narrower, more training-specific boundary.

Neither direction is automatically better. If labels are noisy, a very high C can let a few suspect points dictate the line. If the boundary is genuinely detailed, very low C can miss useful structure. Choose through held-out evaluation using the error costs and threshold policy from lesson 008, rather than by choosing the best training accuracy.

The explicit trade-off is margin versus violation cost. A wide margin is valuable because it makes small feature changes less likely to change a label. But accepting violations can be unacceptable when they are real, costly cases rather than noise. The useful question is therefore not “should I maximize margin?” but “which observed errors are worth paying for to avoid a brittle boundary?” A validation error review can answer that question: list the false positives and false negatives, inspect their features and labels, and decide whether they resemble future cases.

Scaling Is Part of the Geometry

SVM margin and RBF similarity depend on distances and dot products. If loop width ranges from 0–10 but height is recorded in pixels from 0–1,000, height can dominate the geometry merely because of its unit. Standardizing features using training-set statistics puts their scales on comparable footing.

This is not a universal rule that every feature should be made identical. A feature's scale can be meaningful in some designed representations. The operational rule is narrower: decide the intended geometry, fit transformations only on training data, then apply the same transformation to validation and test data. Fitting it on all data leaks information about the evaluation distribution.

When a Straight Corridor Is Too Crude

Suppose 3 examples cluster in the middle of a two-feature picture while 8 examples form a ring around them. No straight line separates them. An RBF kernel supplies a different similarity rule: nearby points in the original feature space have stronger influence than distant ones. The model can then produce a curved boundary without manually listing every transformed coordinate.

gamma controls how local that influence is.

High gamma and high C can memorize local quirks; low values can smooth away a real pattern. A kernel is therefore not “more powerful, therefore better.” It is a stronger bias that costs interpretability and needs evaluation discipline. For large datasets, kernel SVM training can also become expensive; a linear model or a different representation may be the more practical choice.

There is another boundary to keep visible. The kernel trick changes the geometry used by the classifier; it does not explain why a student or digit belongs to a class. If the real issue is missing context, a biased label, or a feature unavailable at prediction time, a curved boundary can make validation scores look better without repairing the problem. Inspect support vectors and high-confidence errors for this reason. They are evidence about the representation and labels, not a causal explanation of the outcome.

For a production-sized decision, begin with a scaled linear baseline and a clear held-out metric. Add an RBF candidate only when error patterns suggest a smooth nonlinear relation that the representation can support. Search C and gamma on validation data, then lock the choice before the final test. This sequence is a practical preference under limited compute and review time; it prevents a large parameter search from quietly turning the final evaluation set into another tuning tool.

Practice: Defend a Boundary Choice

Two linear SVM candidates use the same standardized features. P has margin 0.9 with two training violations; Q has margin 0.3 with zero violations. Validation results show P misses 6 of 80 8s and Q misses 5, but Q produces 18 extra false positives. The review team says false positives create expensive manual corrections.

Which candidate deserves further investigation?

Model answer: P is the stronger initial candidate: it has a larger margin and only one extra miss, while avoiding 18 false positives under the stated cost. This is a decision under named constraints, not proof that P is universally superior. Inspect the confusion matrices at the deployment threshold and check whether its two violations are label errors, a representation gap, or a meaningful subgroup.

If that inspection shows that both violations are the same poorly scanned handwriting style, collect or augment representative scans before increasing C. If instead they are genuinely ambiguous digits even for people, document the review fallback. A classifier boundary cannot create information that the image does not contain.

That distinction changes the next experiment.

Resources

• [DOCUMENTATION] scikit-learn: Support Vector Machines — Focus: connect linear and kernel SVM terminology to the geometric model.
• [INTERACTIVE] scikit-learn: RBF SVM parameters — Focus: vary C and gamma and predict the resulting boundary before viewing it.
• [DOCUMENTATION] scikit-learn: SVC — Focus: inspect support-vector outputs and distinguish classifier settings from evaluation choices.

Key Takeaways

• A maximum-margin boundary is chosen for geometric slack, not merely because it separates training labels.
• Support vectors are the nearby or violating examples that constrain the boundary.
• C trades margin against training violations; gamma controls locality for RBF similarity.
• Feature scaling defines the geometry the model sees and must be fitted without evaluation leakage.
• Margin is a model bias, not a guarantee; validate it against the decision costs and future-like data.