ML and AI Pipeline Architecture: From Raw Data to Production Inference

Advanced ML and AI pipelines are staged systems: ingestion, features, training, serving, monitoring. Architecture guide for batch and streaming paths, LLM/RAG ingestion, and failure modes we see in enterprise deployments.

Teams buy models. Production teams ship systems. The model file is one artifact in a chain that starts with raw events and ends with someone trusting a score on a Tuesday morning.

Below is the architecture we implement in Python and SQL: batch and streaming, classical ML and LLM/RAG, and the stages that get skipped until something breaks in prod.

End-to-end ML pipeline (batch path)

Sources

Raw store

Features

Train

Registry

Serve

Monitor

Six stages. Skip one, pay later.

Production ML/AI stages

01
Ingestion
Events, labels, documents into raw storage. Incremental, logged, replayable.
02
Features
Reproducible transforms to offline tables or online store. Same code path training and serving use.
03
Training
Tracked experiments, frozen feature version, registered artifact with metadata.
04
Evaluation
Holdout metrics plus business KPIs. Promotion is a decision, not a deploy click.
05
Serving
Batch table, sync API, or stream processor. Pick latency requirements first.
06
Monitoring
Input drift, score drift, latency, cost. Retrain trigger optional, alerts not optional.

Batch first. Stream when you must.

Batch (start here)

Nightly or hourly train + score
Churn lists, demand forecasts, lead ranks
Easier debug, cheaper ops
Fine for most B2B decisions

Streaming (when latency demands it)

Sub-second fraud, pricing, moderation
Features computed on Kafka/Flink/Spark stream
Strict p99 SLOs, on-call culture
Our federal fraud work: 10TB+/day on this path

Streaming adds an order of magnitude in operational load. We've shipped it at scale. We still ask whether batch would have been enough.

Feature pipelines beat model tuning most weeks

The expensive mistakes are almost always train-serve skew and time leakage, not the difference between two gradient boosting libraries.

Train-serve contract

One definition of each feature: shared library, feature store, or compiled SQL template
Point-in-time correct joins for behavioral and time series data
Version feature sets with models (model v3 ↔ feature_set_v2)
Document grain, nulls, refresh cadence per feature

Warehouse SQL for historical aggregates. Python for sequential, NLP, or gnarly branching. Hybrid is normal.

Training pipeline: automate the manual ritual

Snapshot labeled data with frozen features
Train with tracked config (MLflow, W&B, or plain YAML in git)
Evaluate on temporal holdout when time ordering matters
Register artifact: data range, metrics, git SHA
Promote only if champion beat or business threshold met
Schedule retrain or trigger on drift, not on someone's calendar memory

Serving: three patterns we deploy

Batch scores written to a table. Downstream apps read. Simplest path.
Sync API with feature lookup at request time. Watch p99 and cold start.
Embedded in stream job for lowest latency. Hardest to operate.

All three need input validation, timeout/fallback, logging with a sane PII policy, and a model version on the response.

LLM / RAG is still a pipeline

RAG ingestion path (parallel to classical ML)

Docs in

Parse

Chunk

Embed

Index

Query + eval

The index rots without a refresh schedule. Treat embeddings like features: versioned, monitored, tested on a golden Q&A set before deploy. Related reads: LLMs in production, agent eval harnesses.

Failure modes we see on audits

Train-serve skew: accuracy dies in prod while offline metrics glow
Label delay: model learns stale outcomes
RAG index drift: answers get worse, nobody notices
AUC up, revenue flat: wrong metric promoted
No registry rollback: bad deploy becomes a fire drill

Build order if you're starting from zero

Batch inference on one narrow use case
Feature pipeline with tests and written grain
Eval gate before users see scores
Monitor inputs, outputs, latency
Automate retrain after the manual loop is boring and reliable
Streaming only when batch failed the latency test

Foundation patterns in Python and SQL data pipelines. Hands-on delivery via pipeline engineering and the Production Standard. Specialized layers for time series and geospatial when features need them.