Fine-Tuning Pre-trained Models

By the end of this lesson, you will be able to...

• distinguish a frozen feature-extraction run from a fine-tuning run by which parameters may change;

• design a staged fine-tuning plan with learning-rate groups, checkpoints, and a rollback condition;

• diagnose whether a disappointing run points to insufficient adaptation, unstable updates, or target overfitting.

Idea in one sentence: Fine-tuning is a controlled experiment that lets selected pretrained weights move toward the target task, while evidence decides how much movement is safe.

Core Insight

Suppose the warehouse team from the previous lesson has a useful starting point. Its frozen image backbone plus a new damage-classification head outperforms a target-only baseline. But the error review exposes a boundary: it misses dents seen under the warehouse's oblique camera angle. The source images had varied viewpoints; the target camera does not.

The obvious response is to unfreeze the entire model and keep training. That response is understandable. The frozen representation is not fully aligned, and more trainable weights give the model more ways to adapt.

But “more ways to adapt” is also more ways to fit accidental details of 1,200 target photographs. A large learning rate can move useful pretrained features abruptly. Unfreezing every layer makes it hard to learn which part of the representation needed change. A falling training loss can look reassuring while target validation gets worse.

Fine-tuning updates some or all pretrained parameters on the target task. Its operational job is not simply to get a lower loss. It is to make a bounded, observable change to a representation that was already useful.

What the User Sees and What the System Knows

The user sees a defect classifier that misses a real class of damage. The team needs a better target result without replacing one hidden failure with another.

At the start of a run, the system has several kinds of state:

This distinction prevents a common mistake: loading a pretrained checkpoint does not freeze it. In a framework such as PyTorch, a parameter can be loaded from a checkpoint and still update unless the run configuration excludes it from gradient-based optimization. Conversely, freezing a parameter means it stays in the forward computation but does not receive an update in that phase.

The Initial Model: Fine-Tuning Means Train Everything a Little

Full-model fine-tuning can be a sound choice when target evidence is ample and the source-target gap justifies broad adaptation. It is not the default meaning of fine-tuning.

For this small warehouse dataset, the initial model fails because it treats the checkpoint as an undifferentiated block. The head was just initialized for the target labels; its early updates can be large and necessary. The backbone already encodes useful features; its changes need a different level of caution. The model needs a run plan that names what may change, by how much, and what observation would stop the plan.

The stronger model is staged unfreezing with evidence gates. Start from the frozen result, open the narrowest plausible part of the backbone, use separate parameter groups when the framework supports them, and make validation—not training loss—the permission to continue.

Investigation Path: A Bounded Adaptation Plan

The following values are illustrative run settings, not a universal recipe. They assume the target validation split, metric, preprocessing, and label policy are fixed before the run. The 1,200 target images are not evidence that every small dataset needs these exact epochs or rates.

The key intermediate state is the transition from phase 0 to phase 1:

before phase 1
  head: trainable; backbone blocks: frozen
  optimizer: contains head parameters only

at phase 1
  head: trainable
  final backbone block: now trainable
  earlier blocks: still frozen
  optimizer: must include the newly trainable block with its declared rate

Changing requires_grad or its equivalent is not enough if the optimizer still owns only the head parameters. Conversely, adding a parameter group without checking its frozen state can update more than the plan intended. Inspect both the trainable-parameter list and the optimizer groups before the first phase-1 batch.

In PyTorch, the official transfer-learning tutorial demonstrates the two broad regimes: optimizing a pretrained network and using it as a fixed feature extractor. The exact API changes over time, but the ownership distinction remains: frozen parameters are not part of the backward updates, while fine-tuned parameters are. PyTorch's tutorial is a concrete implementation reference.

Read the Run, Not Just the Last Number

Suppose phase 0's best validation score is 0.86 overall and the oblique-camera slice is 0.61. After five epochs of phase 1, the training score rises from 0.89 to 0.98, but validation falls to 0.82 and the slice remains 0.60. These are synthetic measurements for the example.

The tempting diagnosis is “the model needs more epochs.” The trace says otherwise: training improved while the held-out target result worsened. That supports an overfitting or unstable-adaptation hypothesis. It does not prove which one. Check the learning-rate group, target split size, label noise, augmentation contract, and whether the validation slice is large enough to be reliable. Roll back to the best checkpoint rather than treating the final epoch as the model.

Now compare a different illustrative result:

This is a runbook-style inference, not a theorem about layer depth. Later layers are often a reasonable place to test adaptation because they can be more tied to the source task, but architectures and pretraining objectives differ. The target comparison earns the decision.

So far, we have seen that fine-tuning changes a controlled set of weights and must preserve a route back to the best observed target behavior. This matters because the team can learn from a failed phase instead of letting an opaque all-layer update erase the baseline.

The Failure Mechanisms Behind Three Symptoms

Symptom: the frozen run plateaus, and errors cluster in the new camera condition. Likely pressure: the existing representation is useful but not sufficiently aligned to the target variation. Next check: unfreeze the smallest late section that could express the missing distinction; compare it to the frozen checkpoint.

Symptom: validation drops immediately after unfreezing while training rises quickly. Likely pressure: updates may be too aggressive for the target evidence, or the target set may be easy to memorize. Next check: restore the best checkpoint, verify optimizer groups and learning rates, then reduce the adaptation scope or rate before rerunning.

Symptom: target validation improves, but performance on a retained relevant reference set collapses. Likely pressure: the model has lost useful earlier behavior, often called catastrophic forgetting in this context. Next check: confirm that the reference set is relevant to the intended deployment; otherwise its decline may be an irrelevant cost. If it matters, constrain the update, retain the earlier checkpoint, or reconsider the target data and objective.

The last symptom makes a boundary explicit: fine-tuning does not guarantee preservation of every source capability. Whether that preservation is a requirement depends on the model's intended job.

Mitigation, Prevention, and Trade-offs

Staged fine-tuning improves adaptation control. It costs more run management than a single “train all layers” command: parameter groups, checkpoints, validation slices, and comparisons must be maintained. It can still fail if target labels are wrong, the validation split is unrepresentative, preprocessing changes between phases, or the selected source representation is a poor fit.

The situated preference is conservative adaptation when target labels are limited, the mismatch is uncertain, and reversibility matters. With abundant target evidence and a large, documented domain shift, broader adaptation can be preferable. The signal to watch is not a universal epoch count; it is whether held-out target behavior and important error slices improve compared with the frozen reference.

Readiness Check

Before starting a fine-tuning phase, answer these questions in the run record:

1. Which blocks are trainable, and which remain frozen?
2. Which parameter groups are in the optimizer, at what learning rates?
3. What fixed target metric and error slices decide success?
4. Which checkpoint is the rollback point?
5. What result would justify unfreezing more, and what result would stop the run?

If any answer is missing, the run may still produce a number, but it will not be an interpretable adaptation experiment.

Check Your Understanding

Check: A team marks the final backbone block as trainable but forgets to add it to the optimizer. What changes during the next training phase?

Think first, then reveal.

Answer: The block participates in the forward pass and can receive gradients, but the optimizer has no instruction to update it. Its weights do not change. The run is effectively still head-only for that block, so the configuration and optimizer ownership disagree.

Check: Training accuracy rises after unfreezing, but validation and the important camera slice both fall. Is “unfreeze another block” the justified next step?

Think first, then reveal.

Answer: No. The current evidence says the present adaptation is not improving the held-out target behavior. Restore the best checkpoint and investigate scope, rates, labels, split reliability, and preprocessing before granting the model more freedom.

Practice: Write a Rollback Rule

A medical-image team has a strong frozen baseline. It wants to unfreeze the last block because one scanner type has weak recall. Write a two-phase plan with:

• the trainable groups and a relative learning-rate choice;
• one overall metric and one scanner-specific slice;
• a rollback condition; and
• one result that would justify expanding the adaptation scope.

Model answer: Train the new head at a higher declared rate and the last block at a smaller declared rate, keeping earlier blocks frozen. Compare the best checkpoint using the predeclared overall validation metric and recall for the weak scanner. Roll back if either the overall metric materially regresses or scanner recall fails to improve beyond the validation uncertainty. Consider one additional late block only if the limited update improves the scanner slice without harming the overall target result and remaining errors point to a representation mismatch rather than labeling or preprocessing failure.

Resources

• [TUTORIAL] PyTorch: Transfer Learning for Computer Vision — Focus: the operational difference between a fixed feature extractor and a fine-tuned network.
• [PAPER] How Transferable Are Features in Deep Neural Networks? — Focus: evidence about feature transfer, specialization, and source-target distance.
• [BOOK] Dive into Deep Learning: Fine-Tuning — Focus: a concrete fine-tuning workflow and its parameter choices.

Key Takeaways

• Fine-tuning changes pretrained weights; freezing and optimizer membership together determine which weights can actually update.
• A staged plan starts from the frozen reference, opens the smallest justified part of the backbone, and records a rollback checkpoint.
• Training improvement alone is not permission to continue; held-out target behavior and important error slices decide whether adaptation helped.
• Conservative or broad adaptation is a choice under data, domain-shift, and reversibility constraints—not a rule attached to a particular layer or epoch count.