Feature Engineering and Representation

By the end of this lesson, you will be able to...

• turn raw operational fields into a feature contract that names transformation, availability, and leakage risk;

• explain why preprocessing is part of the model evaluated, not setup work outside it;

• reject a promising-looking feature when it could not exist at the moment of prediction.

Idea in one sentence: A model can learn only from the version of the world represented in its inputs, so every feature must be useful, reproducible, and honestly available when a prediction is made.

Core Insight

Consider a claims team that must decide at 9:00 each morning which new claims need manual review. Its database has a claim amount, submission time, claim category, past claim history, later investigation notes, and a final payment decision. A boosted model looks strong in an offline notebook when it receives all those columns.

The initial model says: “the database has the facts, so pass every column to the algorithm and let it find the pattern.” That can work for fields that are present at 9:00 and whose meaning is stable. It breaks when a column is only written after investigation, when a raw timestamp hides the relevant interval, or when training code computes a transformation differently from the scoring service.

Feature engineering is the design of the model's evidence boundary. It decides what the model can see, how it is expressed, and when that value is valid. The correct question is not “does this column improve offline accuracy?” It is “what does this value mean at 9:00, how is it produced, and could the scoring system reproduce it exactly?”

Raw Fields Are Not Yet a Decision Representation

The team starts with this small feature contract. The values are illustrative; the contract is the teaching artifact.

The raw timestamp is not automatically wrong. But a timestamp such as 2026-07-28 08:15 has no stable meaning by itself for this decision. The useful teaching representation is the interval from submission to the 9:00 scoring run. Likewise, “past claim count” must mean claims observed before the scoring cutoff, not claims that were later backfilled into the history.

Plain meaning: expose a relationship the decision can use, without smuggling in what happens later.

In this scenario: transform a stored submission time into a duration and turn history rows into a time-bounded count.

Technical name: these are engineered features. They encode an inductive bias—the assumption that recency and prior history may matter—rather than a causal conclusion that they determine claim validity.

Work the Feature Contract Before Fitting a Model

Suppose claim Q-17 was submitted at 08:15 and is scored at 09:00. A transparent transformation is:

hours_since_submission = (scoring_time − submitted_at) / 60 minutes
                         = 45 / 60
                         = 0.75

For the history feature, the contract needs an equally visible boundary:

prior_claim_count_365d(Q-17, 09:00)
  = count claims for this claimant in [09:00 − 365 days, 09:00)

The closing parenthesis matters. It excludes the current claim and any record created after scoring. It is a simplified rule, not proof that 365 days is the ideal window. The team should compare justified windows on validation data; it should not choose one because the training score is largest after trying dozens of undisclosed variants.

Now consider the investigator note. It may be strongly correlated with whether a claim needs review because the investigation itself records the evidence that prompted review. That correlation is not a discovery. At 9:00 the note does not yet exist. Including it would let the offline model see a consequence of the process it is supposed to guide.

This is leakage: information crosses the prediction-time boundary from the future or from a downstream decision. Leakage can produce excellent metrics and a useless deployment. The evidence for rejection is temporal, not aesthetic: the field is unavailable at the instant named in the feature contract.

So far, we have seen that feature engineering is not a search for more columns. It is a set of explicit claims about meaning, time, and transformation. That makes an error auditable before a model amplifies it.

Put Learned Transformations Inside the Model Boundary

Some transformations learn values from data. A standard scaler estimates a mean and standard deviation; an imputer estimates a replacement value; a category encoder learns which categories it has seen. Those fitted values must come from the training partition, then be reused unchanged for validation, test, and scoring.

For example, suppose training claims have an average amount of 200 and standard deviation of 50. A 300-unit claim becomes (300 − 200) / 50 = 2. Those numbers are illustrative. The key rule is that validation claims must not contribute to the mean or standard deviation used to transform themselves. Fitting a scaler on all data lets evaluation information influence training, even though no label column was copied.

The designed boundary is therefore a pipeline:

raw claim at 09:00
  -> time-bounded feature functions
  -> training-fitted imputation, scaling, and category encoding
  -> model score
  -> threshold and review decision

Each arrow is part of the evaluated system. If a notebook fills missing claim amounts with one rule but the scoring job uses another, the deployment is not running the model whose validation result the team approved. This is called train-serving mismatch. It is not solved by choosing a more sophisticated classifier.

There is a trade-off. Putting transformations in one reproducible boundary costs design work, tests, and ownership. It buys an input representation that can be inspected, replayed, and compared honestly. For a small one-off analysis, a manual spreadsheet may be adequate; for a prediction used repeatedly in decisions, a reproducible contract is the safer preference.

Representation Changes What Different Models Can Learn

The model family still matters, but it sees the representation rather than the original database schema. KNN and SVM are especially sensitive to numeric scale because distance and margin geometry change with units. Linear models can benefit from a ratio or explicit interaction when that relationship would otherwise require a more complex boundary. Trees can split raw thresholds, but a correctly time-bounded aggregate can still make a relevant history visible.

Do not infer that every feature needs scaling, every category needs one encoding, or every model needs hand-built interactions. Those are choices under a named model and data shape. The common requirement is narrower: decide the representation before evaluation, fit any data-dependent transformation only on training data, and apply exactly the same contract later.

Feature importance does not replace this review. A high importance for prior_claim_count_365d says the fitted model used that feature on this data. It does not show that the count was available on time, that the window is fair, or that claim history causes the outcome. The contract and error review answer different questions.

Boundaries You Can See Before They Become Incidents

Three signals deserve attention:

• A feature's offline value disappears when its timestamp is enforced. Suspect leakage or an unclear cutoff.
• A validation score changes sharply when the same transformation is fitted only on training folds. Suspect preprocessing leakage.
• Production inputs contain new categories, missing fields, or a different time reference from training. Suspect a broken feature contract.

These signals do not prove the exact cause, but they point to the next inspection. Compare feature values for a small sample of training, validation, and scoring records. Trace each value back to its source and its “as-of” time. A model can be statistically sophisticated while its inputs are temporally impossible.

Check Your Understanding

Check: The team proposes days_until_final_payment as a feature at 9:00 because it is highly predictive in historical data. Should it enter the model?

Think first, then reveal.

Answer: No. The feature is defined by a future event, so it cannot be known at scoring time. Its historical association is exactly why it is dangerous: it would leak downstream information into the prediction.

Check: A category encoder was fit once on every available claim, then cross-validation is run on the encoded data. What is wrong?

Think first, then reveal.

Answer: The encoding has already learned from validation-fold inputs. Fit the encoder inside each training fold through the pipeline, then transform that fold's validation data with the fitted encoder. Otherwise the reported score includes information that would not have been available while fitting.

Practice: Approve a Feature Contract

A model predicts missed clinic appointments at booking time. The team proposes: patient age, appointment type, no-show count in the prior year, a reminder-delivered flag, and rescheduled_after_reminder.

Which feature needs the strongest challenge, and what must the contract state for the others?

Model answer: rescheduled_after_reminder is invalid for a prediction made at booking time because the rescheduling occurs later. The reminder-delivered flag is valid only if the prediction happens after the reminder event; otherwise it has the same timing problem. For age, appointment type, and prior no-show count, document source, transformation, the exact booking-time cutoff, missing-value and unknown-category behavior, and whether each learned preprocessing step is fitted only on training data. Then evaluate the complete pipeline on a split that respects repeated patients and time order.

Resources

• [DOCUMENTATION] scikit-learn: Preprocessing data — Focus: see how training-fitted transformers apply the same representation to later data.
• [DOCUMENTATION] scikit-learn: Pipelines and composite estimators — Focus: connect heterogeneous feature transformations to one evaluated model boundary.
• [DOCUMENTATION] scikit-learn: Common pitfalls — Focus: identify leakage from fitting preprocessing before the evaluation split.

Key Takeaways

• A feature contract names each input's source, transformation, availability time, and failure behavior before model fitting.
• Raw operational columns become useful features only when their decision meaning and time boundary are explicit.
• Learned preprocessing belongs inside the training-and-scoring pipeline; otherwise evaluation can describe a different system from deployment.
• A high offline score cannot justify a feature that is unavailable, downstream, or leaked at prediction time.