Feature Store Architecture: Centralizing Data Inputs for Machine Learning

The problem a feature store solves

Two failures recur in organisations running more than a handful of models. The first is training-serving skew: a feature computed one way in a training notebook and another way in a serving path, producing a model that performs worse in production than in evaluation for reasons nobody can locate. The second is duplication: five teams independently defining customer lifetime value with five subtly different results.

A feature store addresses both by making feature definitions first-class, versioned artefacts with a single implementation used for both training and serving.

Offline and online planes

The architecture splits along access pattern. The offline store holds full history in a columnar warehouse or lakehouse and serves high-throughput batch reads for training and backfills. The online store holds only the latest values in a low-latency key-value system and serves single-entity reads at inference time under tight latency budgets.

• Registry: entities, feature definitions, owners, data types, freshness expectations.
• Transformation layer: batch, streaming and on-demand computation from one definition.
• Materialisation jobs that keep the online store in sync with agreed freshness.
• Serving API that resolves a feature vector by entity key within milliseconds.
• Monitoring for freshness lag, null rates and distribution shift per feature.

Point-in-time correctness is the hard part

Training sets must be assembled using only the feature values that were knowable at each label's timestamp. Naively joining current feature values onto historical labels leaks future information and produces optimistic evaluations that collapse in production. A feature store earns most of its value by making point-in-time joins the default rather than a discipline each team must remember.

This requires event timestamps on every feature row, an explicit definition of the acceptable lookback window, and awareness of late-arriving data. Where upstream systems backfill corrections, the store must decide whether history is immutable or restated — and record which choice applies.

Ownership, governance and when to adopt

Features need owners in the same way services do, with documented semantics, a deprecation process and access controls where the underlying data is sensitive. Lineage from source table through transformation to consuming model is what makes an audit or an incident tractable.

Adoption should be proportionate. A single model with a nightly batch pipeline does not need a feature store. The investment pays off when several teams share entities, when real-time inference requires consistent features across paths, or when regulators expect provable lineage. Below that threshold, a well-documented transformation library is the better answer.