Q: How do you keep a RAG index fresh when documents change?

- Event-driven ingest: on upload/edit/delete, push a job to a queue; worker re-chunks and re-embeds only the changed doc; delete old vectors by document ID. - Keep a document version so stale chunks can be found and purged. TRADE-OFF: Re-embedding everything is simplest but expensive; incremental updates are cheap but need careful bookkeeping.

Question 1

How do you answer any AI system design question? Give the five steps.

Accepted Answer

1. Clarify scope and numbers — users, latency target, data size, how often data changes, budget.
2. Split offline from online — indexing/training on one side, serving on the other. Interviewers look for this immediately.
3. Draw the simple version — gateway → retrieval/model → LLM → response. Then add cache, queue, re-ranker, fallback only where you can justify each.
4. Name three failure modes — and what you'd do about each (retry, fallback, human review, limits).
5. End with evals and monitoring — how you'd know it's working next month, not just today.

Every answer in this topic follows the same shape: requirements → design → what breaks → trade-off. Interviewers care most about the unglamorous parts — failure modes, cost, evals, observability — so those are the lines to say out loud.

Question 2

Design a system that serves an LLM to millions of users (ChatGPT-style)

Accepted Answer

REQUIREMENTS:
Prompt in, streamed answer out; thousands of concurrent users; GPUs are the expensive part.

DESIGN:
API gateway (auth, rate limit) → request queue → GPU inference workers that batch many requests together → tokens streamed back over SSE/WebSocket → prompt/response cache for repeated queries → usage logs for billing.

WHAT BREAKS:
One very long prompt can block a batch; a single GPU node dying kills in-flight requests; cost explodes if every request hits the largest model.

TRADE-OFF:
Batching raises throughput but adds a little latency per user. Route simple requests to a small model, hard ones to a big model (model routing).

Question 3

How do you reduce latency and cost of LLM calls in production?

Accepted Answer

- Semantic cache: embed the prompt, if a near-identical prompt was answered recently, return that answer.
- Prompt caching / KV cache: reuse the computed prefix for long system prompts.
- Streaming: show the first token fast even if total time is the same.
- Model routing / cascading: cheap model first, escalate only if confidence is low.
- Shorter prompts, fewer retries, batch offline jobs at night.

TRADE-OFF:
Caches return stale or slightly wrong answers; routing needs a reliable "is this hard?" signal.

Question 4

Design an LLM gateway for a company (one entry point for all internal teams)

Accepted Answer

REQUIREMENTS:
Many teams, many providers (OpenAI, Anthropic, open models), central cost control, fallback if a provider is down.

DESIGN:
Single API compatible with the OpenAI format → auth + per-team budgets/rate limits → router picks provider/model by policy → retry and fallback to a second provider on error → logging of every request (tokens, cost, latency) → PII redaction before sending out.

TRADE-OFF:
One gateway is a single point of failure — run it replicated; provider-specific features get lost behind a common API.

Question 5

Design a system to fine-tune and deploy a model safely

Accepted Answer

DESIGN:
Dataset stored with versions → training job on a GPU cluster (LoRA/QLoRA for cheap fine-tunes) → model registry with version tags → offline eval suite must pass → canary deployment (5% traffic) → full rollout → rollback button.

WHAT BREAKS:
Model regresses on cases the eval set didn't cover; training data leaks into eval.

TRADE-OFF:
Fine-tuning bakes behaviour into weights (good for style/format), but for fresh facts use RAG, not fine-tuning.

Question 6

Design a RAG system over 10,000 internal PDFs with p95 latency under 3 seconds

Accepted Answer

CLARIFY FIRST:
How often do docs change? Single team or multi-tenant with per-document permissions?

DESIGN, TWO SEPARATE PATHS:
- Offline ingest (async): upload → parse PDF → chunk (500–1000 tokens with overlap) → embed → write to vector DB + a keyword index (BM25). Runs through a queue so uploads never slow queries.
- Online query: embed the question → hybrid retrieval (BM25 + dense) → take top ~20 → re-rank to top 3–5 → build prompt with those chunks → LLM answers with citations → cache the answer.

WHAT BREAKS:
Bad chunking splits a table in half; retrieval returns "relevant-looking" but wrong chunks; permissions leak across tenants.

TRADE-OFF:
Re-ranking improves quality but adds ~200–500 ms; bigger top-k improves recall but bloats the prompt and cost.

Question 7

How do you evaluate a RAG system?

Accepted Answer

- Separate retrieval from generation.
- Retrieval: recall@k / precision@k against a labelled set of question → correct chunk pairs.
- Generation (RAG triad): faithfulness (answer doesn't contradict the retrieved text), answer relevance (actually answers the question), context relevance (retrieved chunks were useful).
- Use an LLM-as-judge for scale, but keep a small human-labelled golden set to check the judge itself.

TRADE-OFF:
LLM judges are cheap but biased toward their own style; human labels are accurate but slow.

Question 8

Chunking strategy — how do you choose?

Accepted Answer

- Fixed-size with overlap is the default. Semantic/structural chunking (by heading, paragraph, table) is better for docs with structure.
- Store metadata with each chunk (source, page, section, date) so you can filter before searching.

TRADE-OFF:
Small chunks = precise retrieval but lost context; large chunks = context kept but noisy prompts.

Question 9

Vector DB vs traditional DB — when do you actually need a vector DB?

Accepted Answer

- Need it when you search by meaning (embeddings), not exact keywords.
- Small scale (<1M vectors) → pgvector in Postgres is enough. Large scale → dedicated store (Pinecone, Milvus, Qdrant, Weaviate) with HNSW index.

TRADE-OFF:
Approximate nearest-neighbour search is fast but not exact; keyword search still wins for IDs, names, and exact phrases — hence hybrid.

Question 10

How do you keep a RAG index fresh when documents change?

Accepted Answer

- Event-driven ingest: on upload/edit/delete, push a job to a queue; worker re-chunks and re-embeds only the changed doc; delete old vectors by document ID.
- Keep a document version so stale chunks can be found and purged.

TRADE-OFF:
Re-embedding everything is simplest but expensive; incremental updates are cheap but need careful bookkeeping.

Question 11

Design a multi-step agentic workflow (e.g. meeting scheduling, code review, or email campaigns)

Accepted Answer

DESIGN:
Orchestrator (LLM) with a plan → act → observe loop (ReAct style) → tools exposed with strict schemas (function calling / MCP) → each step's state saved so the run can resume after a crash → a max-steps limit and a budget limit → human approval for irreversible actions (send email, delete).

WHAT BREAKS:
Runaway tool loops, the agent calling the wrong tool, tool output injected with malicious instructions (prompt injection), silent partial failures.

TRADE-OFF:
More autonomy = more useful but harder to trust; put guardrails at the tool layer, not just in the prompt.

Question 12

How do you make an agent reliable in production?

Accepted Answer

- Idempotent tools so retries are safe.
- Timeouts + retries with backoff on every tool call.
- Step limits and cost limits per run.
- Tracing every step (LangSmith / Arize Phoenix / OpenTelemetry) so you can answer "where did it choose the wrong tool?"
- Checkpointing state (LangGraph-style) so long runs survive restarts.

FALLBACK:
If the agent fails, degrade to a simpler deterministic path or ask the user.

Question 13

Design a multi-agent system (planner + workers)

Accepted Answer

DESIGN:
A planner agent breaks the task into sub-tasks → a message queue hands them to specialised worker agents (search, code, write) → results collected by a coordinator → final answer assembled. Shared memory (Redis/DB) holds intermediate results.

WHAT BREAKS:
Agents disagree or duplicate work; one slow worker blocks the whole task; context explodes as everything is passed around.

TRADE-OFF:
Specialised agents are easier to test and swap, but coordination overhead and cost go up — most tasks don't need more than one agent.

Question 14

Design a unified query engine across email, calendar, documents, and chat

Accepted Answer

DESIGN:
Connectors ingest each source into a common schema with permissions → chunk + embed + metadata (source, owner, timestamp) → a query router decides which sources to search → hybrid retrieval per source → merged, re-ranked, answered with citations. Permission filter applied at retrieval time, never after.

TRADE-OFF:
Freshness (real-time API call) vs speed (pre-indexed copy); many teams do both — index for search, live-fetch for details.

Question 15

Design ChatGPT's cross-conversation memory feature

Accepted Answer

DESIGN:
After each conversation, a background job extracts durable facts (name, preferences, projects) → stores them as short records with source and timestamp in a user-scoped store → at the start of a new chat, retrieve the top relevant memories (semantic + recency) and inject into the system prompt → user can view/edit/delete.

WHAT BREAKS:
Storing sensitive data the user didn't want kept; stale facts contradicting new ones; memory bloating the prompt.

TRADE-OFF:
More memory = more personalisation but higher cost, more privacy risk, and more chance of wrong assumptions. Keep it small, keep it editable.

Question 16

How do you defend an agent against prompt injection?

Accepted Answer

- Treat all tool/web/document output as data, not instructions.
- Keep the system prompt and the untrusted content in clearly separated sections.
- Restrict what tools can do (allow-lists, read-only by default), require confirmation for side effects.
- Scan outputs for exfiltration patterns (URLs with secrets, unexpected sends).

TRADE-OFF:
No complete fix exists — you reduce blast radius with permissions, not with prompts alone.

Question 17

Design an evaluation pipeline for an LLM product

Accepted Answer

OFFLINE:
Golden dataset (a few hundred real cases with expected outputs) → run on every prompt/model change in CI → metrics: exact match where possible, LLM-as-judge with rubrics elsewhere → block deploy on regression.

ONLINE:
A/B test with a holdout, track thumbs-up/down, task completion, escalation rate.

TRADE-OFF:
Offline evals are fast and repeatable but drift from real traffic; online is real but slow and noisy. You need both.

Question 18

What do you monitor for an LLM application in production?

Accepted Answer

- Latency (P50/P95/P99), time to first token, tokens per request, cost per request, error rate, rate-limit hits, cache hit rate.
- Quality signals: judge scores on sampled traffic, user feedback, refusal rate, hallucination flags.
- Traces of every step for agents and RAG (what was retrieved, what tool was called).
- Alerts on cost spikes and quality drops — prompt regressions never show up as HTTP 500s.

Question 19

Design a content / policy violation detection system

Accepted Answer

DESIGN:
Fast cheap filter first (regex, keyword lists, small classifier) → borderline cases go to an LLM classifier with the policy in the prompt → confident violations blocked, uncertain ones queued for human review → reviewer decisions fed back to retrain the classifier.

TRADE-OFF:
False positives annoy users, false negatives cause harm; tune thresholds per category and keep humans in the loop for the grey zone.

Question 20

How do you add guardrails to an LLM app?

Accepted Answer

- Input guardrails: PII detection, jailbreak detection, topic restriction.
- Output guardrails: schema validation (JSON), toxicity check, "cite a source or refuse" rule for RAG.
- Tools: Guardrails AI, NeMo Guardrails, or your own small classifiers.

TRADE-OFF:
Every guardrail adds latency and can block legitimate requests — measure the false-positive rate.

AI & ML System Design

AI & ML System Design