Building AI Data Pipelines That Don't Break
Most RAG failures are data failures. Here's the boring infrastructure that separates demos from products.
By Creative Genius
·
·
8 min read
If your RAG demo wows in the boardroom and falls apart in production, the model is almost never the problem. It's the data pipeline feeding the model. Here's what we wire up on every serious project.
Five components every production pipeline needs
- Source-of-truth ingestion with idempotent loaders. Re-running cannot create duplicates. Use deterministic content hashes as upsert keys so you can replay the entire pipeline anytime without polluting the index.
- Versioned chunking. When you change chunk size from 512 to 1024 tokens, old vectors become stale. Tag every chunk with a
pipeline_versioncolumn and run dual-write during cutover. - Embedding generation with batching, rate-limit handling, and resume-from-failure. Never trust a single API call. We use a queue with exponential backoff and persist intermediate state every 100 chunks.
- Vector index with metadata filters. Always scope retrieval per tenant, per language, per date, per source type. A single tenant's data leaking into another's response is a P0 incident.
- Eval harness that runs after every pipeline change. Golden set of 50–200 questions with known good/bad retrievals. If recall drops below baseline, the pipeline change is blocked.
The schema we use
Every chunk row carries: id, document_id, tenant_id, content, embedding, content_hash, pipeline_version, source_url, last_seen_at, ingestion_run_id. You will thank yourself later for the last three when debugging "why doesn't the system know about the doc we uploaded yesterday."
Where teams skimp and regret it
- No dead-letter queue for failed embeddings → silent data gaps.
- No content-hash dedup → duplicate chunks dilute retrieval quality.
- No
last_seen_at→ deleted source docs stay searchable for months. - No eval before pipeline merges → regression goes to production undetected.
Bottom line
The fanciest model in the world cannot rescue a sloppy ingestion pipeline. Invest the boring engineering up front and the model becomes a commodity you can swap.