Question 1

How do you improve inference speed in production LLM deployments?

Accepted Answer

Optimize across the stack, and first decide whether you're optimizing latency (TTFT, per-token speed) or throughput (tokens/sec across all requests) — they need different levers. The big ones: a proper serving engine (vLLM/TensorRT-LLM/SGLang) with continuous batching + PagedAttention, quantization (FP8/INT8/INT4), speculative decoding, prefix/prompt caching, tensor parallelism for large models, optimized kernels (FlashAttention), and better hardware (H200/B200 for bandwidth).

Levers by goal: Throughput: continuous batching + PagedAttention (keep the batch full) → latency: speculative decoding, prefix caching, quantization → capacity: tensor parallelism, better GPUs.

The highest-leverage move: "Continuous (in-flight) batching is the single biggest throughput win — it keeps the GPU in the efficient part of the roofline by keeping the batch large. For latency, speculative decoding and prefix caching help most. Always name whether you're chasing latency or throughput first; the optimizations that help one can hurt the other."

Reference in assignment: LLM Inference Optimization (youtube.com/watch?v=jV2sCj4lHYk).

Question 2

What are the key LLM optimization techniques?

Accepted Answer

Group them by what they optimize. Precision: quantization (FP16/BF16/FP8/INT8/INT4). Memory: KV cache management (PagedAttention), GQA/MQA. Throughput: continuous batching. Latency: speculative decoding, prompt/prefix caching. Model size: distillation, pruning, MoE. Compute: FlashAttention, optimized kernels. Scale: tensor/pipeline parallelism.

Categories:
- Precision: Quantization (INT8/INT4/FP8).
- Memory: PagedAttention, GQA/MQA, KV quantization.
- Throughput: Continuous batching.
- Latency: Speculative decoding, prefix caching.
- Size: Distillation, pruning, MoE.

Framing: "Every technique trades some combination of latency, throughput, memory, and quality — there's no free lunch. The art is stacking the ones that fit your workload: e.g. a high-QPS chat API leans on continuous batching + PagedAttention + FP8, while a latency-critical autocomplete leans on speculative decoding + a small model." (Cross-ref LLMOps assignment.)

Reference in assignment: "5 LLM optimization techniques" (LinkedIn, Pallavi Shekhar).

Question 3

What is model quantization (INT8, INT4, FP16, BF16), and how does it affect quality?

Accepted Answer

Quantization reduces the numerical precision of weights (and sometimes activations) to shrink memory and speed up compute. FP16/BF16 (16-bit, 2 bytes/param) is the standard baseline with minimal quality loss (BF16 has more range, better for training). FP8 (1 byte, native on H100+) gives strong quality at inference. INT8 halves memory vs FP16 with small loss; INT4 quarters it with a larger but often acceptable loss when using good methods.

Precision comparison:
- FP16 / BF16: 2 bytes/param; Quality = baseline (near-lossless).
- FP8: 1 byte/param; Quality = very good (H100+ native).
- INT8: 1 byte/param; Quality = small loss.
- INT4: 0.5 bytes/param; Quality = noticeable but usable (GPTQ/AWQ).

Key nuance: "The trade-off is precision vs footprint: lower bits = smaller and faster but more degradation. Modern methods (GPTQ, AWQ, SmoothQuant) make INT4 surprisingly usable, and quantization-aware training beats post-training quantization on quality. A 70B model is ~140GB at FP16 but ~40GB at INT4 — that's the difference between multi-GPU and a single card, which is why quantization is the first knob for fitting models on hardware."

Reference in assignment: "AI Engineering Explained" (y

Question 4

How does GGUF work?

Accepted Answer

GGUF (GPT-Generated Unified Format) is a binary file format from the llama.cpp project for storing quantized LLMs for efficient CPU/GPU inference — the successor to GGML. It packs everything into a single file (weights + metadata + tokenizer), supports many quantization levels (Q4_K_M, Q5, Q8 "k-quants"), and uses memory-mapped loading for fast startup, all cross-platform.

Mechanism: Single .gguf file → weights (quantized) + metadata + tokenizer → mmap-loaded by llama.cpp / Ollama / LM Studio → runs on CPU, Apple Silicon, or modest GPUs.

Why it matters: "GGUF is what makes local LLM inference practical — it's the format behind Ollama and LM Studio. Its wins are the single-file packaging, a spectrum of quantization levels (Q4_K_M is the popular size/quality sweet spot), and memory-mapped loading so startup is fast and you only page in what you need. It brought running capable models on consumer hardware to the masses."

Reference in assignment: outcomeschool.com/blog/how-does-gguf-work.

Question 5

What is KV cache, and how do you manage memory for it?

Accepted Answer

During autoregressive decoding, the attention keys and values for all previous tokens are cached so the model doesn't recompute them each step — turning per-token work from O(n²) into incremental O(n). The catch: the KV cache grows with batch size × sequence length and can consume enormous GPU memory — often more than the model weights for long contexts or large batches.

Managing it:
- PagedAttention (Q6): paged allocation, near-zero fragmentation.
- GQA/MQA: fewer KV heads → much smaller cache.
- KV quantization (store the cache in INT8/FP8); eviction/offloading; prefix sharing.

Key insight: "The KV cache is usually the memory bottleneck that limits your batch size and context length — for a 70B model at 8k context with batch 32, it can be 20–30GB+. That's why memory capacity is so decisive for throughput (bigger batch = more tokens/sec), and why PagedAttention and GQA are such big wins — they let you fit more concurrent requests." (Cross-ref LLM Fundamentals attention.)

Reference in assignment: outcomeschool.com/blog/kv-cache-in-llms.

Question 6

What is Paged Attention?

Accepted Answer

PagedAttention (vLLM's core innovation) manages the KV cache like an operating system manages virtual memory: the cache is split into fixed-size blocks (pages) allocated non-contiguously on demand, with a block table mapping logical token positions to physical blocks. This eliminates the memory fragmentation that wasted 60–80% of KV memory in naive contiguous allocation.

Mechanism: KV cache → fixed-size blocks (like memory pages) → allocated on demand, non-contiguous → block table maps logical→physical → near-zero waste + copy-on-write for shared prefixes.

Why it's a breakthrough: "Naive KV allocation reserves contiguous memory for the max sequence length, wasting most of it (internal + external fragmentation). PagedAttention's paging drives waste to near zero, which directly enables much larger batch sizes — and thus dramatically higher throughput. It also makes prefix sharing trivial via copy-on-write blocks. It's the reason vLLM's throughput leapt over prior servers."

Reference in assignment: outcomeschool.com/blog/paged-attention-in-llms.

Question 7

How does continuous batching improve LLM inference throughput?

Accepted Answer

Continuous (in-flight) batching batches at the iteration/token level instead of the request level. In static batching, a batch runs until every request finishes, so short requests wait idly for long ones. Continuous batching instead adds new requests and evicts finished ones every decoding step, keeping the GPU maximally utilized despite highly variable generation lengths.

Comparison: Static: wait for the whole batch to finish (fast requests idle) → Continuous: every iteration, evict done + admit new → batch stays full → GPU stays busy.

Why it's the top throughput lever: "It's the single highest-leverage inference optimization — often 2–4×+ throughput — because it keeps the batch size large, which keeps the GPU in the high-efficiency region of the roofline. Variable-length generation is the norm for LLMs, and static batching wastes huge amounts of compute waiting for the slowest request. vLLM popularized this; TensorRT-LLM calls it in-flight batching."

Reference in assignment: outcomeschool.com/blog/continuous-batching-in-llms.

Question 8

What is speculative decoding, and how does it speed up inference?

Accepted Answer

Speculative decoding uses a small, fast draft model to propose several tokens ahead, then the large target model verifies all of them in a single parallel forward pass, accepting the longest correct prefix. Because verification is parallel and the draft is usually right, you generate multiple tokens per expensive target-model pass — a 2–3× speedup with an identical output distribution (it's lossless).

Mechanism: Draft model proposes k tokens → target model verifies all k in one pass → accept the correct prefix, reject the rest → net: several tokens per target pass.

Key nuance: "The elegance is that it's lossless — the target model's verification guarantees the same distribution as normal decoding, so quality is unchanged. The speedup depends on the acceptance rate: a draft model that agrees often gives a bigger win. Variants avoid a separate draft model — Medusa adds prediction heads, EAGLE and n-gram/lookahead reuse the model itself."

Reference in assignment: outcomeschool.com/blog/speculative-decoding.

Question 9

What is model parallelism vs data parallelism in distributed training?

Accepted Answer

Data parallelism replicates the full model on each GPU and splits the batch across them, synchronizing gradients — use it when the model fits on one GPU but you want more throughput. Model parallelism splits the model itself across GPUs (because it doesn't fit on one) — this includes tensor parallelism (within layers) and pipeline parallelism (across layers).

Comparison:
- What's split: Data parallelism = the data (batch); Model parallelism = the model (weights).
- Each GPU holds: Data parallelism = full model copy; Model parallelism = part of the model.
- Use when: Data parallelism = model fits, want throughput; Model parallelism = model too big for one GPU.

Framing: "The mental split: data parallel = same model, different data; model parallel = split model, data flows through. Data parallelism is simpler and the default when the model fits; model parallelism (tensor + pipeline) is what you reach for when a model exceeds single-GPU memory. Large-scale training combines them (plus sharding) into 2D/3D parallelism."

Question 10

What is tensor parallelism, and how does it help serve large models?

Accepted Answer

Tensor parallelism splits individual layers' operations — the big matrix multiplies — across GPUs: each GPU holds a slice of the weight matrices, computes its portion, and results are combined via an all-reduce after each layer. It splits the model "horizontally" within layers, letting you serve models too large for a single GPU's memory.

Mechanism: Weight matrix split column/row-wise across GPUs → each computes its slice → all-reduce combines → repeat per layer. Needs fast interconnect (NVLink).

Key trade-off: "Tensor parallelism is how you fit a 70B or 405B model across 2–8 GPUs, but it requires a fast interconnect: there's an all-reduce after every attention and feed-forward layer, so on PCIe (~64 GB/s) communication dominates and kills performance, while on NVLink (~900 GB/s) it's negligible. Rule: always use SXM/NVLink for tensor parallelism, and keep it within a node where possible."

Question 11

What is pipeline parallelism?

Accepted Answer

Pipeline parallelism splits the model across GPUs by layers (stages) — GPU 1 holds the first N layers, GPU 2 the next N, and so on — with data flowing through the pipeline stage by stage. It splits the model "vertically" by depth, enabling very large models with less communication than tensor parallelism (only at stage boundaries).

The bubble problem: Naive pipelining leaves GPUs idle waiting for earlier stages ("pipeline bubbles"). The fix is micro-batching — split each batch into micro-batches so multiple are in flight across stages, keeping every GPU busy.

Framing: "Pipeline parallelism trades communication for latency: it's cheaper on interconnect than tensor parallelism (only stage boundaries), but introduces pipeline bubbles that micro-batching mitigates. In practice, large-model training/serving combines tensor parallelism (within a node, high-bandwidth) with pipeline parallelism (across nodes) — tensor for speed, pipeline for scale."

Question 12

What is model sharding, and when would you use it?

Accepted Answer

Model sharding splits a model's parameters — and in training, its gradients and optimizer states — across multiple GPUs so it fits when it exceeds single-GPU memory. It's the umbrella concept; tensor/pipeline parallelism shard for inference, while FSDP/ZeRO shard states for training.

When to use it:
- The model (or its training state) is too big for one GPU's VRAM.
- To reduce per-GPU memory so you can run larger batches / longer context.
- Training large models where optimizer states dominate memory (ZeRO/FSDP).

Framing: "Sharding is the general answer to 'it doesn't fit' — you distribute the model (and training state) so each GPU holds a fraction. The cost is always communication: shards must exchange data, so interconnect bandwidth becomes the constraint. Tensor parallelism, pipeline parallelism, and FSDP/ZeRO are all specific sharding strategies for different situations."

Question 13

What is FSDP (Fully Sharded Data Parallel), and how does it differ from DeepSpeed ZeRO?

Accepted Answer

Both shard model states across GPUs so you can train large models while writing data-parallel-style code. DeepSpeed ZeRO (Microsoft) shards in stages: ZeRO-1 shards optimizer states, ZeRO-2 adds gradients, ZeRO-3 adds parameters (each stage saves more memory, adds more communication), plus offload to CPU/NVMe. FSDP (PyTorch-native) shards parameters, gradients, and optimizer states — equivalent to ZeRO-3 — integrated directly into PyTorch.

Comparison:
- Origin: DeepSpeed ZeRO = Microsoft library; FSDP = PyTorch-native.
- Sharding: DeepSpeed ZeRO = staged (ZeRO-1/2/3); FSDP = params+grads+optimizer (≈ZeRO-3).
- Extras: DeepSpeed ZeRO = offload, pipeline, MoE; FSDP = simpler, tighter PyTorch integration.

Framing: "They solve the same problem — shard optimizer/gradient/parameter memory so each GPU holds a fraction — via the same core idea. ZeRO's staging lets you dial memory-vs-communication (ZeRO-1 → ZeRO-3), and it has more features (CPU/NVMe offload). FSDP is the PyTorch-native, increasingly default choice for its simplicity and integration. Pick FSDP for a clean PyTorch stack, DeepSpeed for advanced offload/features." (Cross-ref Fine-Tuning distributed training.)

Question 14

How do you select GPUs for LLM inference?

Accepted Answer

The key insight: inference is memory-bandwidth-bound, not compute-bound like training. During decode, the GPU reads the entire weight matrix from VRAM for every token, so memory bandwidth and capacity matter more than raw FLOPs. Select by: (1) size VRAM for your model (FP16 = 2 bytes/param → 70B ≈ 140GB; INT8 halves; INT4 quarters; add 20–30% for KV cache); (2) match the GPU tier; (3) weigh the software ecosystem and interconnect.

GPU comparison:
- H100: 80GB / 3.35 TB/s; Best for = standard; best ecosystem; ≤70B INT8.
- H200: 141GB / 4.8 TB/s; Best for = 70B FP16 single-GPU; long context.
- B200: 192GB / 8 TB/s, FP4; Best for = hyperscale + frontier models.
- A100 / L40S: 80GB / 48GB; Best for = budget; smaller models.

What impresses: "Lead with 'inference is memory-bound, training is compute-bound' — that reframes the whole decision. It's why the H200 beats the H100 on token/s despite identical compute (43% more bandwidth), and why memory capacity matters (bigger KV cache → bigger batch → more throughput). Two more real-world points: the H100's mature vLLM/TensorRT tuning often beats a less-optimized newer GPU, and you need SXM/NVLink for tensor parallelism."

Note: Reflects th

Question 15

How do you manage GPU memory for serving multiple models?

Accepted Answer

Several strategies depending on model sizes and isolation needs: MIG (Multi-Instance GPU — physically partition one GPU into up to 7 isolated slices for small models/multi-tenant); LoRA multiplexing (serve many fine-tunes on one shared base model since only adapter weights differ — S-LoRA / vLLM multi-LoRA); model multiplexing / lazy loading (load on demand, evict LRU); quantization to fit more; and dedicated GPUs per model (simple but wasteful).

The elegant answer: "If you're serving many fine-tunes of the same base, LoRA multiplexing is the winner — you keep one base model in memory and swap tiny adapters, serving hundreds of variants on one GPU. For heterogeneous small models, MIG gives hardware isolation. The trade-off across all of these is isolation vs utilization vs cold-start: dedicated GPUs are isolated but wasteful; multiplexing is efficient but adds swap latency."

Question 16

How do you implement auto-scaling for AI workloads?

Accepted Answer

Scale model replicas horizontally based on the right metric — usually queue depth / concurrency, not CPU. Use Kubernetes HPA or KEDA (event-driven, scale on queue length), optionally scale-to-zero for sporadic workloads (accepting cold starts), and predictive scaling for known traffic patterns.

The GPU-specific challenge: GPUs are expensive and slow to provision (loading a multi-GB model = cold start, Q19), so scaling is coarser and slower than for CPU services. Mitigate with pre-warmed pools, a sensible minimum replica count, and scaling on leading indicators (queue depth) rather than lagging ones.

Key nuance: "CPU utilization is the wrong signal for LLM serving — a GPU can be 'busy' at very different throughputs. Scale on queue depth or in-flight requests, which directly reflect user-facing latency. And because GPU cold starts are slow, you tune min-replicas and warm pools to trade cost against tail latency rather than scaling reactively to zero." (Cross-ref LLMOps.)

Question 17

What is the role of load balancing in AI serving infrastructure?

Accepted Answer

Load balancing distributes requests across model replicas — but naive round-robin is suboptimal for LLMs because request costs vary wildly (a 10-token vs a 2000-token generation). Better strategies: route to the least-loaded / least-queued replica, use prefix/session affinity (send requests sharing a prompt prefix to the same replica to reuse its KV cache), and apply model-aware routing (route to a replica that has the right model/adapter loaded).

Key insight: "The LLM twist is that requests aren't uniform, so round-robin creates hotspots — least-load routing handles the variance, and prefix-aware routing is a real win because it boosts KV-cache hit rates (shared system prompts, RAG context) and cuts both latency and compute. Load balancing for LLMs is about request cost and cache locality, not just spreading connections evenly."

Question 18

How do you implement request queuing and priority scheduling for AI services?

Accepted Answer

When demand exceeds capacity, queue requests rather than dropping or overloading, and schedule by priority — premium or latency-sensitive traffic first. Use priority queues, per-tier/per-model queues, admission control (shed load when overloaded), and a max queue time (fail fast instead of making users wait forever). Queued requests also feed batch formation for continuous batching.

Framing: "Queuing + scheduling is how you enforce SLAs under load: separate queues for tiers (free vs paid), priority ordering, and admission control so the system degrades gracefully instead of collapsing. A subtle point — the queue is also where you assemble batches for continuous batching, so scheduling and throughput are linked. Set a max wait so you fail fast rather than blowing latency budgets."

Question 19

How do you handle cold-start latency for serverless AI deployments?

Accepted Answer

The cold-start problem: scaling from zero requires loading a multi-GB model into GPU memory, which takes seconds to minutes, so the first request after scale-up is very slow. Mitigations: keep a warm pool / min replicas (costs money, avoids the stall); speed up loading (memory-mapping, streaming weights, fast local NVMe over network storage); node-level model caching; snapshot/restore of GPU state; smaller/quantized models (load faster); and predictive pre-warming.

Core trade-off: "GPU cold starts are far worse than CPU serverless because the model itself is huge — loading tens of GB dominates. So it's fundamentally cost (keep warm) vs latency (scale to zero): for spiky, latency-tolerant workloads you scale to zero; for user-facing ones you pay for a warm floor. Faster storage and quantized models shrink the cold start when you can't keep everything warm."

Question 20

How do you implement model caching to reduce redundant computations?

Accepted Answer

Cache at two levels. Prefix/prompt caching: cache the KV state for shared prompt prefixes (system prompts, few-shot examples, RAG context) so the expensive prefill isn't recomputed for every request — a huge win when many requests share a prefix (vLLM automatic prefix caching, Anthropic prompt caching). Response/semantic caching: cache full responses for identical or semantically similar queries (embedding-match, e.g. GPTCache), skipping inference entirely.

Cache types:
- Prefix/prompt (KV): Saves = prefill recompute; Best for = shared system prompts, RAG context.
- Response (exact/semantic): Saves = whole inference; Best for = repeated/FAQ queries.

Which to reach for: "Prefix caching is the highest-value LLM-specific optimization — long shared prefixes (a big system prompt, retrieved context) are common, and caching their KV skips the compute-heavy prefill. Response/semantic caching skips inference entirely but risks stale or falsely-matched answers, so use it where queries genuinely repeat. Both trade memory and freshness for latency/cost."

AI Infrastructure and Scalability

AI Infrastructure and Scalability