Serving Models with APIs

By the end of this lesson, you will be able to...

• define an inference API as a versioned contract from client input through preprocessing, model execution, and response semantics;

• trace one request to distinguish an invalid request, an unsupported input, an inference failure, and a valid low-confidence result;

• review an API design for reproducibility, compatibility, latency behavior, and honest failure handling.

Idea in one sentence: A model API is a public promise about how a request becomes a versioned inference result—or a clearly named failure—not a thin wrapper around a checkpoint.

Core Insight

Suppose a belt-scanner service sends a package photograph to the optimized defect model from the previous lesson. The scanner needs a response before it decides whether to divert the package. An engineer proposes a simple endpoint: receive an image URL, load the model, and return damaged: true.

That works for a happy-path demo. It breaks as soon as the client sends an unreadable image, a JPEG with unexpected color channels, an image larger than the declared input budget, or a request produced under an old model contract. It also hides what a client should do when the service cannot run inference in time. The boolean says nothing about the model and preprocessing version that created it, the meaning of any confidence value, or whether the input was rejected before the model saw it.

The stronger model treats serving as an inference contract. The contract owns the external input schema, validation, exact preprocessing, model and pipeline version, output meaning, latency boundary, and error categories. It gives clients a stable decision surface while keeping the model's internal tensors private.

The Promise We Need to Keep

For this warehouse system, the API promises one bounded job: classify one supported package image with a declared model pipeline, or explain why it cannot do so. It does not promise fleet autoscaling, automatic retries, or a cloud deployment recipe; those are system-level concerns outside this track.

Start by making the client-visible contract explicit.

The pipeline_version is intentionally broader than a checkpoint hash. If the resize policy, image normalization, class order, output threshold, or postprocessing changes, the client-visible behavior may change even when many model weights remain the same.

The Naive Design: /predict Returns a Label

The minimal endpoint is attractive because it concentrates on the neural network:

request -> model(image) -> label

It is insufficient because the raw client image is not necessarily the model input. The optimized artifact expects a specific decoded, resized, normalized tensor. The result may need a threshold, a label mapping, or a review decision before it has the meaning the client needs.

The minimal design also collapses different failures into one generic error. A malformed request should be fixed by the client. A valid image that exceeds the supported size may need a documented resubmission path. A temporary unavailable inference runtime may need a retry policy. A low-confidence but valid model result may need human review rather than a transport error. Those states have different owners and different safe responses.

A Better Boundary: Versioned Inference Pipeline

The server turns an external request into a model result through explicit stages:

client request
  -> schema and size validation
  -> decode and image-policy validation
  -> declared preprocessing
  -> optimized model artifact
  -> output calibration/threshold/label mapping
  -> versioned response or named failure

The following illustrative request keeps the client contract smaller than the internal tensor contract:

{
  "request_id": "scan-7f32",
  "image": {
    "content_type": "image/jpeg",
    "bytes_base64": "..."
  },
  "pipeline_version": "defect-2026-03"
}

The client has not supplied a tensor shape or normalization values. Those are server-owned details of this declared pipeline. A valid response can expose the outcome and enough context to use it safely:

{
  "request_id": "scan-7f32",
  "pipeline_version": "defect-2026-03",
  "decision": "review",
  "label": "damaged",
  "score": 0.71,
  "score_semantics": "model confidence before the review threshold",
  "latency_ms": 48
}

This does not claim that 0.71 is a calibrated probability of physical damage. Its semantics must be declared and evaluated with the model pipeline. In this example, a review decision may be a product policy that protects uncertain cases; it is separate from the model's raw score.

Worked Request Trace

Trace four requests through the same contract. The values and messages are illustrative.

The trace makes an important distinction visible. scan-7f34 is not a poor model prediction; the supported pipeline could not accept that input. scan-7f35 is not a malformed request; it is a valid request for which the service could not complete inference within the available runtime. Neither should be silently converted into damaged: false.

Now add one successful but uncertain case. A valid image gives a raw score of 0.52, and the pipeline's review policy puts scores from 0.45 through 0.60 into review. The API returns a normal result with decision: review, not a 500-series error. The model completed; the product chose not to automate this case. This is a useful boundary because it makes uncertainty an intentional interface state rather than an accidental client interpretation.

So far, we have seen that the response category identifies what happened before, during, or after inference. This matters because callers can correct inputs, retry availability problems, or route uncertain valid cases without guessing from one ambiguous failure code.

Design Alternatives and Trade-offs

There is no universal request representation. Sending image bytes in the request can make the inference attempt self-contained and auditable, but it increases payload size and decoding work. Sending an object reference can keep requests small, but the service needs explicit permissions, retrieval failure handling, and an immutable-content or version policy to make the result reproducible.

Similarly, synchronous inference is simple when the latency budget is short and the service can answer quickly. An asynchronous job interface may fit longer-running work, but it changes the client contract from “receive a prediction now” to “submit work and retrieve a later result.” Batching can improve throughput, as lesson 029 showed, but waiting for a batch can violate an interactive scanner's latency promise.

These are situated choices. For a bounded belt decision, a synchronous request with an explicit timeout and small supported image policy may be preferable. For a large non-interactive inspection queue, an object reference and asynchronous job may be more appropriate. The API design should name the constraints that lead to either choice.

Operational Consequences Inside the Boundary

The service must preserve enough structured facts to make one inference result inspectable: request id, pipeline version, validation outcome, selected response category, end-to-end latency, and a safe error code. Sensitive image content and private data need their own retention and access policy; logging every raw request is not a default requirement.

The service must also distinguish its health from a prediction result. A readiness endpoint can report whether the declared artifact and pipeline have loaded; it should not be abused as a substitute for a client inference request. The precise health and orchestration protocol belongs to production systems work, but the inference contract needs a way to avoid claiming readiness when the required artifact is unavailable.

Design Review

Before exposing an inference endpoint, check:

1. Is every accepted input shape, type, size, and source declared?
2. Does the server, not the client, own the preprocessing version for the selected pipeline?
3. Does the response name the pipeline version and the semantics of each score or decision field?
4. Are malformed requests, unsupported inputs, unavailable inference, and valid review cases distinct?
5. Is the timeout and batching behavior compatible with the caller's latency promise?
6. Can a client migrate deliberately when the response schema or output semantics change?

If the answer to any question is vague, the endpoint is still a demo, not a dependable inference boundary.

Check Your Understanding

Check: A client sends a valid JPEG, but the inference runtime is unavailable after preprocessing starts. Should the API return invalid_request?

Think first, then reveal.

Answer: No. The input is valid. Return the declared availability or timeout category so the client can use its retry or fallback policy. Calling it invalid incorrectly assigns ownership to the client.

Check: A response contains label: damaged and score: 0.71. What must the contract say before a client treats that score as a probability of damage?

Think first, then reveal.

Answer: It must state the score's semantics and provide evidence that it is calibrated for that interpretation under the declared pipeline. A model confidence or ranking score is not automatically a probability.

Practice: Review a Contract Change

An API currently returns only { "label": "damaged" }. A team wants to replace it with a new pipeline that changes image normalization and returns a score plus a review band. Write a migration plan that includes:

• what versioned request or response fields change;
• how old and new output semantics can coexist during migration;
• one invalid-input response and one valid-but-unavailable response; and
• one check that proves the new preprocessing is part of the same deployed pipeline as the new model.

Model answer: Introduce a new pipeline version and response schema that names pipeline_version, label, score, score_semantics, and decision. Keep the previous version available for a declared migration window or require clients to select the new version explicitly; do not silently change the meaning of the old label-only response. Return 400 invalid_request for a missing image field and 503 inference_unavailable for a valid request that cannot complete because the runtime is unavailable. Build and test the preprocessing code, artifact reference, threshold, and label mapping as one versioned pipeline; an integration test should send a fixture image through the API and compare the declared versioned outcome with the approved pipeline result.

Resources

• [ARTICLE] Google Cloud: Machine Learning Model Serving Patterns — Focus: the boundary between model inference, request handling, and service responsibilities.
• [ARTICLE] RFC 9457: Problem Details for HTTP APIs — Focus: structured, client-readable HTTP error responses.
• [BOOK] Designing Data-Intensive Applications — Focus: interface, failure, and compatibility reasoning that applies to networked inference boundaries.

Key Takeaways

• A serving API owns the complete inference contract: accepted inputs, preprocessing, artifact version, output semantics, timeout, and named failures.
• A valid low-confidence or review result is different from an invalid request or unavailable runtime; clients need those distinctions to act safely.
• Version the whole pipeline, not only the checkpoint, because preprocessing and postprocessing can change externally visible behavior.
• Choose payload, sync/async, and batching policies from the caller's latency and reproducibility constraints.