Building Production Neural Networks

By the end of this lesson, you will be able to...

• design a minimal contract that connects data, a checkpoint, evaluation evidence, and inference behavior;

• distinguish an inference artifact from a resumable training checkpoint;

• audit a model handoff for preprocessing drift, train/eval state mistakes, and unverifiable selection decisions.

Idea in one sentence: A model is ready to hand off when the team can reproduce what it expects at its boundary—not merely when it has a saved weight file and a good-looking metric.

Core Insight

A warehouse classifier flags damaged packages. In a notebook it reaches 92% validation accuracy. At the scanning station it suddenly flags too few damaged boxes. The weights are unchanged, but station images are normalized differently, the code forgot model.eval(), and nobody knows which threshold was chosen from validation data.

The naive model says that deployment is “load weights and call the model.” That works only while every hidden assumption happens to stay the same. The evidence above shows that the learned weights are only one component of a prediction.

A production-minded neural network has a small, explicit prediction contract: what input is accepted, how it is transformed, which artifact and mode are used, how an output is interpreted, and which validation evidence justified that interpretation. This is not a claim to build a serving platform; it is the boundary a model artifact must carry before another system can safely call it.

The Promise We Need to Keep

For the package classifier, the promise is narrow:

given: a camera image in the declared input format
return: probability that the package is damaged
decide: flag it when probability >= a declared threshold

Each part has an owner. The preprocessing contract owns resize, color conversion, and normalization. The model artifact owns parameter values plus registered buffers such as BatchNorm running statistics. The inference path owns eval() and the output conversion. The evaluation record owns the split, metric, threshold rule, and checkpoint-selection rationale.

Keeping these boundaries explicit is useful because each observed failure points somewhere different. A shape error belongs at the input boundary. Varying scores for one image may point to train/eval mode. A changed false-alarm rate can point to the threshold or changed data—not automatically to the weights.

The Initial Design and Where It Breaks

The quick prototype commonly saves this:

torch.save(model.state_dict(), "weights.pt")

That is a good inference artifact when the model definition, preprocessing contract, label mapping, and inference mode are supplied elsewhere in a controlled way. It is not enough to resume a training run faithfully. An optimizer has state; a scheduler may have state; the epoch and best validation evidence matter; random and data-order settings can matter for an experiment.

The pressure appears after a regression. A teammate loads weights.pt, trains for five more epochs with a new optimizer, and gets a different result. Nothing is necessarily wrong with PyTorch: they resumed from parameters but not from the original training state. Another teammate evaluates the same weights through a different resize path and calls the disagreement a model failure.

The stronger design separates two artifacts because they answer different questions:

Do not add every possible file because a checklist says so. Include the state needed for the promise you are making. Weights alone can be appropriate for a bounded inference handoff; they are insufficient for a claim of faithful training continuation.

A Worked Handoff

Here is a compact contract for the warehouse example.

Walk one image through it. The station receives a 640 × 480 RGB image. It performs the declared resize and normalization, producing the tensor shape the validation loop used. The loaded model enters evaluation mode, so Dropout stops sampling masks and BatchNorm uses saved running statistics. The forward pass produces a logit; sigmoid converts it to 0.81; the declared 0.72 threshold produces flagged.

Every transition is inspectable. If another station gets 0.55 for the same raw image, compare the transformed tensors and the loaded artifact before tuning the architecture. If repeated calls on the same tensor vary, inspect mode before blaming the threshold. The trace earns the stronger model: an inference result is a chain of data and state transitions, not a weight file alone.

Checkpoints Make State Recoverable

A general PyTorch checkpoint can be a dictionary rather than a mysterious binary:

checkpoint = {
    "model_state_dict": model.state_dict(),
    "optimizer_state_dict": optimizer.state_dict(),
    "epoch": epoch,
    "best_val_loss": best_val_loss,
    "config": config,
    "split_id": "2026-07-a",
}
torch.save(checkpoint, "run-042.pt")

This is a teaching-shaped example. A real project might record scheduler state, label vocabulary, transform version, code revision, or random-state policy too. The key question is not “what fields are fashionable?” but “which state would make our current claim reproducible or auditable?”

One important boundary: reproducibility is not a promise of bit-identical results on every machine and library version. It is the stronger practice of making meaningful inputs, state, choices, and evidence explicit enough to rerun, compare, and explain a result. If exact determinism is required, it needs additional declared hardware and framework constraints.

Trade-offs and Failure Boundaries

The trade-off is upfront discipline versus short-term speed. A notebook with implicit transforms is faster to begin. A contract costs time to define, version, and test, but it makes regressions attributable and handoffs safer once experiments or callers multiply.

This contract does not solve data drift, fleet monitoring, distributed training, access control, rollout policy, or incident response. Those belong to the ML systems and operations tracks. It does establish the boundary they need: a caller should know what the artifact expects and how its output was chosen.

Watch for a mismatch between validation and inference evidence. Useful signals include a changed input tensor range, an unknown checkpoint provenance, a threshold with no validation record, a missing label map, or a model left in training mode. These are concrete reasons to stop a handoff even if the last notebook cell ran successfully.

Design Review

Before handing off a neural model, answer:

1. What exact input representation did validation and inference agree on?
2. Is the artifact intended for inference, training resumption, or both?
3. Which state beyond weights is needed for that purpose?
4. Are eval() mode and gradient behavior stated for normal inference?
5. How are logits decoded, labels mapped, and thresholds chosen?
6. Which split and metric selected this checkpoint, and where is that evidence recorded?
7. Which concern is deliberately deferred to a platform or ML systems owner?

If a question has only “it is in the notebook somewhere” as an answer, the boundary is not yet reliable.

Checks

Check: A checkpoint contains model weights but no optimizer state. Is it invalid?

Think first, then reveal.

Answer: Not for a bounded inference artifact, provided its model definition and prediction contract are available. It is insufficient for a claim that training can resume faithfully with the same optimizer behavior.

Check: The deployed model receives the same image dimensions as validation but produces different scores. What should be compared first?

Answer: Compare the actual transformed tensors, loaded checkpoint/buffers, and eval() mode. Equal image dimensions do not prove equal color conversion, normalization, or runtime behavior.

Check: Why is a threshold part of the deployment contract?

Answer: A probability becomes an action only through a decision rule. The rule changes false alarms and misses, so it needs validation evidence and a named operating constraint.

Transfer: Audit a Handoff

A team provides model.pt and says, “It is 94% accurate.” The code uses a different image resize in the service than in validation, calls model(x) without setting mode, and labels output index 1 as damaged without a stored class map. Write the first three changes you would require before integration.

Model answer: First, declare and reuse the validation preprocessing path, then compare a service tensor with a validation tensor. Second, load the intended artifact and call model.eval() with torch.no_grad() for ordinary inference. Third, store the output/label mapping and the threshold or decision rule alongside the metric evidence, including the validation split that selected it. These changes make the claimed accuracy interpretable; they do not claim to solve later concerns such as fleet monitoring or rollout safety.

Resources

• [TUTORIAL] PyTorch: Saving and Loading Models — Focus: inference state versus a general training checkpoint.
• [DOCUMENTATION] PyTorch: Module — Focus: state dictionaries, registered buffers, and train/eval mode.
• [TUTORIAL] PyTorch: Optimizing Model Parameters — Focus: the state transitions that a training loop owns.
• [TUTORIAL] PyTorch: Data Basics — Focus: making the data representation explicit.

Key Takeaways

• A deployable model is a prediction contract across input preprocessing, artifact state, mode, decoding, and decision rule—not only weights.
• Keep inference artifacts and resumable training checkpoints distinct because they require different state and support different claims.
• A small, inspectable handoff makes drift and regression diagnosable while clearly deferring platform-scale ML operations to the appropriate owner.