LLM Operations.

In LangSmith Hub, re-point the active alias to the prior semantic version tag (e.g., "v1.2.3"). No code deploy needed — the inference service loads by alias and picks up the rollback on the next request. Log the rollback as an incident event. Post-mortem: add the failing input to the golden eval set so the regression is caught automatically in CI before the next prompt change.

A prompt template version pins the instruction text, few-shot examples, and output format. A model version pins the weights. They are independent axes: the same prompt can behave differently across model versions, and different prompts can produce equivalent output from the same model. Track both axes in every LLM trace — (prompt_version, model_version, input_hash) → output — to isolate which axis caused a quality change.

Treat prompts like feature branches: each engineer works on a named branch in YAML, the registry stores branch tags (e.g., "summarize-technical:feat/tone-fix"). A PR merges the branch and promotes the tag to "staging". After eval gate passes (automated RAGAS score + LLM-as-judge), a second promotion step pushes to the "production" alias. This prevents two engineers from racing to update the same production prompt.

A fixed list is one-size-fits-all. A retrieved list adapts to the current query: a billing question retrieves billing examples, a technical question retrieves technical examples. This improves output format adherence by 15–30% on diverse query distributions. The retrieval cost is a single embedding call (~1ms) — negligible compared to the LLM call.

Run an ablation: compare LLM-as-judge scores (0–5 on task quality) for zero-shot vs 3-shot vs 5-shot on your golden eval set. If few-shot scores are not statistically significantly better (p < 0.05), the examples are adding tokens without value. Also check format compliance rate — few-shot primarily enforces output structure, not factual accuracy.

The context window is not the limiting factor — attention quality is. Empirically, LLM attention degrades for content more than 60–70k tokens from the end of the prompt (lost-in-the-middle effect). Use at most 8 examples; beyond that, the model stops reliably using later examples. For very long system prompts, keep examples at the end, closest to the user query.

Prompt injection exploits the model's inability to distinguish between instruction and data — user-controlled content overwrites system instructions. Jailbreaking uses social engineering or adversarial prompts to convince the model to violate its alignment guidelines ("pretend you are DAN with no restrictions"). Both attack the instruction-following mechanism but via different vectors. Injection is an infrastructure vulnerability; jailbreaking is a model alignment vulnerability. Defend against injection with architectural controls (classifier, canary, instruction hierarchy); defend against jailbreaking with model-level alignment (RLHF, Constitutional AI, output classifiers).

Use an automated red-team: generate 500 injection attempts across categories (direct override, roleplay bypass, context extraction, indirect via retrieval). Test each against your defence stack and measure attack success rate (ASR). Target ASR < 5% on a standardised benchmark like HarmBench. Run this as a CI gate — any prompt template change triggers the injection test suite. Also hire human red-teamers for a week before launch to find creative bypasses the automated suite missed.

Immediate: (1) Invalidate the affected session and force re-authentication. (2) Log the full input/output for forensic analysis. (3) Check if the leaked system prompt contains any secrets (API keys, PII, business logic) — if yes, escalate to P0 and rotate any exposed credentials immediately. (4) Identify the injection vector: was it direct user input or indirect via retrieved context? Medium-term: add the attacking input to the injection classifier training set, re-train, and re-deploy. Long-term: remove secrets from system prompts entirely — prompts should be designed to be leakable without causing harm.

Store both prompt versions in the registry with distinct tags. The inference service reads the active variant mapping from a config store (Redis or a feature flag service like LaunchDarkly). To start the test: update the config with the traffic split (e.g., 80% A / 20% B) — no code deploy. To end: update config back to 100% A or 100% B. Every request logs its variant tag in the LangSmith trace for clean segmentation.

Use a power analysis: with MDE=2% quality delta, α=0.05 (5% false positive rate), and β=0.20 (80% power), you need approximately 1,200 samples per variant (2,400 total). If your system handles 500 requests/day at 20% traffic to variant B, you accumulate 100 B samples/day — test takes 12 days. For faster results: increase traffic to 50% B (reduces time to 5 days) or increase MDE to 5% (reduces required n to ~200/variant).

Trust human raters over the LLM judge — they represent ground truth. This signals judge miscalibration: the judge prompt or model is rewarding a quality dimension that humans do not value (e.g., verbosity, hedging language, or sycophancy). Calibrate the judge: run the same 100 examples through both judge and human raters, compute Cohen's κ — if κ < 0.70, the judge is unreliable. Revise the judge prompt or switch to a stronger judge model. Then re-evaluate the prompt variants with the calibrated judge before making a promotion decision.

Use a tiered memory strategy: (1) Keep the last 8 turns in the context window directly (buffer memory). (2) Summarise older turns with the LLM into a running summary (200 tokens max) and include that summary at the top of the history. (3) For factual details mentioned earlier (names, preferences, decisions), use entity extraction and store them in a structured entity memory (Redis hash). This gives 3-layer coverage: immediate context + summarised history + structured facts.

Context window (or context limit) is the maximum total tokens the model can process in one call — a fixed architectural property. Context length refers to the actual number of tokens in a specific request. A 128k context window model can handle requests up to 128k tokens; a typical chat request might use 2–10k tokens. Confusingly, "context window" is sometimes used to mean the context length of a specific request — always clarify which meaning is intended in design discussions.

Three reasons: (1) Cost — 128k input tokens at $2.50/1M = $0.32/call; at 1,000 calls/day, that is $320/day versus ~$0.01/day with RAG retrieving 5 relevant chunks. (2) Quality — the lost-in-the-middle effect means models attend poorly to content in the middle of very long contexts; retrieval puts relevant content at the optimal position. (3) Latency — processing 128k tokens takes 10–30 seconds TTFT; RAG with 2k context takes under 1 second. Use full-context only for document analysis tasks where all content is genuinely needed.

Hybrid entity + summary. Entity memory persists facts the support agent needs (account ID, issue type, steps already tried, escalation status) with zero context cost. Summary memory retains the conversation narrative (tone, context, resolution path) in a compact 200-token block. Buffer memory for the last 4 turns ensures the model has verbatim access to the immediate exchange. Avoid vector-store memory for support — the user needs all context from this session, not sparse retrieval from past sessions.

Use session_id as the memory key. Store memory state in Redis with TTL (24 hours for support, 30 days for research assistants). On each request: (1) Load memory state from Redis by session_id. (2) Add to memory. (3) Save back to Redis. For entity and vector-store memory, namespace all keys by session_id to prevent cross-user contamination. Never store memory in-process — process restarts would lose all session context.

This is the fundamental limitation of lossy memory. Mitigation: (1) Increase entity extraction coverage — add domain-specific entity types to the extraction prompt so more facts are captured. (2) Keep a full transcript in cold storage (S3) and retrieve on explicit "remind me what we discussed" requests. (3) Use vector-store memory as a fallback for long sessions. (4) Design the UX to set expectations: "I may not remember details from earlier in our conversation — please re-mention them if needed." Perfect recall requires full-context retrieval, which has cost and latency tradeoffs.

LLMLingua uses a small LM (e.g., LLaMA-2-7B) as a proxy for the target LLM. It computes the conditional perplexity of each token given the surrounding context — tokens with low perplexity (easily predictable from context) carry less information and can be dropped without losing meaning. The method preserves high-perplexity tokens (domain-specific terms, numbers, named entities) and drops filler words, repeated phrases, and transitional language. LLMLingua-2 improves on this with a BERT-based extractive approach that is faster and more multilingual-friendly.

On standard RAG benchmarks (NQ, TriviaQA), quality drops sharply below a 3× compression ratio (target_ratio < 0.33). At 5× compression, answer quality typically drops 8–15% relative, which is noticeable in production. The safe operating range is 2–3× compression (target_ratio 0.33–0.50) for most technical content. For highly structured content like tables or code, do not compress at all — structural tokens are high information density and LLMLingua cannot preserve tabular relationships.

Run compression asynchronously in parallel with the embedding/retrieval step. Use a fast compression model (LLMLingua-2-BERT is 10–50× faster than LLaMA-based) that adds < 100ms on CPU. Cache compressed versions of frequently retrieved chunks: hash the chunk content and store compressed_chunk in Redis with TTL=24h. On cache hit, skip compression entirely. This reduces the average compression overhead to < 10ms for well-trafficked chunks.

LLMs are trained with a causal attention mask that creates recency bias — each token attends more strongly to nearby tokens. The beginning of the context also benefits from being the anchor for all subsequent attention computations. Tokens in the middle of the context lack both primacy (attention anchor) and recency (nearby tokens) effects, resulting in relatively weaker attention weights. The shape is U-shaped: high attention at position 0, decaying toward the middle, then rising again at the end. This effect is stronger for longer contexts and models not specifically trained for long-context retrieval (e.g., Gemini 1.5 Pro has better middle-context attention than GPT-4 at 32k).

No. Models specifically trained for long-context retrieval (Gemini 1.5 Pro with 1M context, Claude 3.5 Sonnet) show significantly weaker U-shaped bias. Models trained primarily on short contexts and extended with RoPE scaling or ALiBi show stronger degradation. Test your specific model: retrieve 10 chunks, put the answer in positions 0, 5, and 9 of equal-quality context, and measure answer extraction accuracy across positions. If accuracy drops > 15% in the middle position, the model has strong lost-in-the-middle bias and position reranking is essential.

Ablation study on your golden eval set (200+ QA pairs): run three conditions — (1) random chunk ordering, (2) relevance-score ordering (most relevant first), (3) U-shape interleaved ordering. Measure recall@k (does the correct answer appear in the top-k chunks?) and answer extraction rate (does the model extract the correct answer when the relevant chunk is present?). If conditions 2 and 3 improve extraction rate vs condition 1 by > 5%, position reranking is worth the complexity.

Static allocation reserves max_seq_len × layers × 2 × num_heads × head_dim bytes per request at creation — for a 70B model with max_seq_len=4096, that is ~2GB per request, limiting the server to ~20 concurrent sequences on an 80GB A100. PagedAttention allocates 16-token pages on demand: a 100-token response uses 7 pages (~112MB) instead of 2GB. The freed memory supports 10–15× more concurrent sequences, enabling continuous batching that keeps GPU utilisation at 85–95% vs 30–50% for static allocation.

Traditional batching waits for all requests in a batch to complete before starting new ones — if one request generates 2,000 tokens and others generate 50, the GPU idles for 95% of the long request's duration. Continuous batching (also called iteration-level scheduling) adds new requests to the batch at each token-generation step. Short requests complete and leave; new requests join. GPU utilisation stays high regardless of output length variance. vLLM implements this natively; naive HuggingFace generate() does not — this is the primary source of vLLM's 2–24× throughput advantage.

Work backward: at 100 RPS with 500ms TTFT budget, the prefill phase must complete in < 500ms. Measure your model's prefill throughput (tokens/second) on your target GPU. For a 70B model on A100: ~10,000 prefill tokens/second. At 1,000 input tokens/request: prefill takes 100ms — well within budget at low concurrency. At high concurrency, prefill requests queue — add replicas when P99 TTFT exceeds 300ms (give yourself 200ms headroom). Rule of thumb: 1 A100 80GB handles ~50 RPS at 1k input / 256 output tokens for a 70B INT4-quantized model.

AWQ: identifies salient weight channels via activation statistics and quantizes non-salient channels to INT4 — highest quality, fastest GEMM kernels, best for production GPU serving. GPTQ: uses second-order (Hessian) information for layer-wise quantization — slightly lower quality than AWQ, slower calibration (1–3 hours for 70B), but widely supported. bitsandbytes NF4: 4-bit Normal Float quantization with no custom GEMM kernels — designed for QLoRA training flexibility, not inference throughput. Use AWQ for serving, GPTQ as fallback, bitsandbytes only for fine-tuning.

INT4 quantization gives approximately 4× VRAM reduction vs FP32 and 2× vs FP16/BF16. Practical impact: LLaMA 3 70B in BF16 requires ~140GB VRAM (2× A100 80GB). In AWQ INT4, it fits on a single A100 80GB (~35GB), enabling single-GPU serving. LLaMA 3 8B BF16 requires ~16GB (one A10G or A100 40GB); in INT4, it fits on consumer GPUs with 10GB VRAM. This unlocks self-hosted serving on much cheaper hardware — a single A100 80GB at $2.50/hour vs two A100s at $5/hour for the same 70B model.

Step 1: Measure perplexity on a held-out eval set — if it jumped > 2 points, the quantization is the cause. Step 2: Check calibration data alignment — recalibrate with domain-specific samples. Step 3: Increase q_group_size from 128 to 64 (finer-grained quantization, slightly higher quality at cost of 10% more VRAM). Step 4: Identify which layers are most sensitive (usually attention projection layers in early and last layers) and exclude them from quantization with --excluded-layers. Step 5: Fall back to INT8 (AWQ w_bit=8) — lower VRAM savings but much smaller quality delta.

Speculative decoding is provably equivalent to standard decoding in output distribution under the rejection sampling scheme. When the verifier rejects a draft token, it samples a corrected token from the verifier's distribution (not just rejects and restarts). This ensures the final output sequence has exactly the same probability distribution as if the verifier had generated every token autoregressively. The speed improvement comes purely from parallelising verification — no quality tradeoff.

Target acceptance rate > 70% for positive throughput gains. Below 60%, speculative decoding is net negative. Improve acceptance rate: (1) Use a draft model from the same fine-tuning lineage as the verifier — instruction-tuned draft for instruction-tuned verifier. (2) Reduce K: fewer proposed tokens → less compound probability drop. K=3 often outperforms K=8 on diverse workloads. (3) Use prompt-lookup decoding (a speculative method using the prompt itself as the draft) for RAG tasks where the answer heavily quotes the retrieved context — acceptance rate reaches 85–95%.

TTFT and throughput are in tension — optimising for throughput (large batches) increases TTFT (queuing). Solution: split the workload. (1) Interactive tier (TTFT < 200ms): dedicated replicas with small batch size (max_num_seqs=16), no speculative decoding, priority scheduling for short requests. (2) Batch tier (throughput-optimised): speculative decoding with K=5, large batch size (max_num_seqs=128), accepts longer TTFT (< 5s). Route by latency budget at the API gateway: requests with user_facing=true go to interactive tier, background jobs go to batch tier. Each tier can be independently scaled.

Use TP when GPUs are connected by NVLink (intra-node, low All-Reduce latency < 1ms). Use PP when GPUs are across nodes connected by InfiniBand or Ethernet (inter-node, All-Reduce latency 5–50ms). For interactive serving (TTFT < 500ms), pure TP on 4 intra-node GPUs minimises latency. For batch serving on a large cluster, PP across nodes reduces cross-node communication to pipeline boundaries only (much less frequent than layer-wise All-Reduce). Empirical rule: TP=4 on one 4×A100 node before adding PP across nodes.

Use a rolling deployment with readiness probes. Set maxSurge=1 and maxUnavailable=0 in the Deployment strategy. The new replica starts and downloads/loads the model (5–15 minutes for 70B). The readiness probe (HTTP GET /health → 200) passes only after the model is fully loaded. Once the new replica is ready, K8s routes traffic to it and terminates the old replica. Result: no requests hit a replica with a partially loaded model. For TP=4 deployments, all 4 GPU pods must be ready before any traffic is routed — use pod readiness gates or a custom controller.

405B in BF16 = 810GB — needs 8× A100 80GB minimum (just for weights). Plan: TP=8 on one 8-GPU DGX node connected by NVLink 3.0 (bidirectional 900GB/s — All-Reduce latency ~2ms per layer). INT4 AWQ quantization reduces to ~202GB — TP=4 on one node, leaving 40GB/GPU for KV cache. Deploy 3 TP=4 replicas behind a load balancer. KV cache budget per replica: 40GB × 4 GPUs = 160GB for ~100 concurrent sequences at 4k tokens. For P99 TTFT < 1s: limit prefill queue depth to 20 requests, enable prefix caching for shared system prompts, use speculative decoding with a 1B draft model (same family). Estimated cost: 3 × 4 A100s at $3/GPU/hour = $36/hour.

Set a daily spend alert at 80% of (monthly_budget / 30). Log cost_usd to BigQuery or CloudWatch. Create a scheduled query that sums daily cost_usd at midnight and publishes the result to a CloudWatch metric. Set a CloudWatch alarm at the threshold — notify via PagerDuty (P2) or Slack. For real-time alerting, maintain a Redis counter incremented by cost_usd on each call and check against threshold per request. This gives sub-minute alerting on runaway cost events.

Immediate: apply a per-user token rate limit (token bucket, 10k tokens/min default, 100k for enterprise). Investigate: check what that user is doing — are they running automated scripts, looping on a long document, or just a power user with legitimate needs? Options: (1) Implement progressive rate limiting with graceful UX messaging. (2) Move the user to a dedicated endpoint with appropriate billing. (3) Cache their repeated queries (semantic cache). (4) Route their use case to a cheaper model if quality is acceptable. Never throttle silently — communicate the limit and the reason.

Build a unit economics model: (1) LLM cost per user per month = avg_calls × avg_tokens × cost_per_token. (2) Revenue attribution: run an A/B test comparing conversion/retention with vs without the feature — measure revenue delta per user. (3) ROI = revenue_delta / llm_cost — target > 3×. (4) Sensitivity analysis: what happens to ROI if token prices drop 50% (likely as open models improve)? What if usage grows 10×? Features with ROI < 3× at current scale but > 10× at 10× scale are worth keeping if the growth trajectory is credible. Features with ROI < 2× at any scale should be re-architected or removed.

Replace the Redis key-scan approach with a vector database (Redis Vector Search, Qdrant, or Pinecone) that supports ANN search natively. At 10k concurrent users, the cache index will contain millions of entries — O(n) scan is infeasible. Qdrant's HNSW index returns the nearest neighbour in < 5ms at 10M vectors. Architecture: (1) Embed query (1ms). (2) ANN search in Qdrant (5ms). (3) If similarity > threshold, return cached response. (4) Total overhead: < 10ms added latency vs 500–2000ms LLM call. Use horizontal scaling for both the embedding service and the vector database.

Target > 30% for FAQ-style workloads (support bots, product Q&A). For diverse generation tasks (code, creative writing), realistic target is < 5% — semantic caching has less value here. Measure: log cache_hit: true/false on every request. Daily cache_hit_rate = sum(cache_hits) / total_requests. If hit rate is < 10% on an FAQ workload, investigate: threshold may be too high, TTL may be too short, or the cache may not be warm (cold start problem — pre-warm with top 1,000 historical queries at startup).

Use TTL as the primary invalidation mechanism — short TTLs (1–4h) for frequently updated content, long TTLs (24h) for stable content. For immediate invalidation on a specific update (e.g., a policy change): maintain a namespace prefix per knowledge-base version (cache:v3:...). On knowledge base update, bump the version — all queries now miss the old cache and build a new one under the new prefix. Old prefix entries expire naturally via TTL. Avoid active deletion (complex, race conditions); version prefix + TTL is simpler and equally effective.

Fine-tune a DistilBERT or DeBERTa-small model on 2,000–5,000 labeled production queries. Labels: simple (FAQ, yes/no, short extraction) vs complex (reasoning, code, multi-step, long generation). Use active learning: start with rule-based labeling (query length < 50 tokens → simple), run inference, sample uncertain predictions (confidence 0.4–0.6) for human labeling. Target: 95% accuracy, < 5ms inference latency on CPU (batch size 1). Re-train monthly. Cost of classifier: ~$0 GPU inference at 5ms/call; cost savings from routing: 10–50× reduction on simple queries.

Assume 1,000 queries/day, average 500 input + 200 output tokens. All GPT-4o: 1000 × (500 × $2.50 + 200 × $10.00) / 1M = $3.25/day. Tiered routing (70% cheap, 30% expensive): 700 × Gemini Flash + 300 × GPT-4o = 700 × (500 × $0.075 + 200 × $0.30) / 1M + 300 × (500 × $2.50 + 200 × $10.00) / 1M = $0.068 + $0.975 = $1.04/day. Savings: $2.21/day = $806/year. At 100k queries/day, savings are $80,600/year — a meaningful engineering investment.

Three safeguards: (1) Confidence threshold: only route to the cheap model if classifier confidence > 90% — below 90%, default to the expensive model. (2) Output quality check: sample 1% of cheap-model responses and score with an LLM-as-judge (strong model evaluates quality). Alert if cheap-model quality drops > 5% relative to expensive model baseline. (3) User feedback loop: track explicit thumbs-down signals per model — if the cheap model's negative feedback rate is 2× the expensive model's rate for a query category, that category is mis-routed.

At 1,000 RAG calls/day with average 3,000 input tokens per call: Daily input cost = 1000 × 3000 × $2.50/1M = $7.50/day. With 3× compression (1,000 tokens after compression): Daily input cost = 1000 × 1000 × $2.50/1M = $2.50/day. Savings = $5.00/day = $1,825/year. At 10,000 calls/day, savings are $18,250/year. These numbers justify a dedicated compression service and regular calibration effort. For output-heavy workloads, compression has less impact — output pricing is 4× input pricing, and compression only affects input.

On standard benchmarks (NQ, TriviaQA): 2× compression (50% tokens retained) → < 1% quality drop. 3× compression (33% retained) → 2–4% quality drop. 5× compression (20% retained) → 8–15% quality drop. These numbers apply to narrative text. For structured data (tables, code, numbered lists): quality drops sharply even at 2× compression. Measure on your specific domain — a legal RAG system may see 10% quality drop at 2× compression on contract clauses. Always validate before deploying, and build the quality gate into CI.

Add a step in CI that runs whenever retrieved context size changes > 10% (triggered by chunking config or retrieval parameter changes): (1) Load golden Q&A dataset (200 items). (2) Run RAG pipeline with current configuration — measure RAGAS faithfulness baseline. (3) Run RAG pipeline with compression applied. (4) Compute faithfulness delta. (5) Fail CI if faithfulness drops > 3% relative. This prevents compression regressions from shipping silently. Run nightly (not on every PR) to keep CI fast — compression eval takes 10–20 minutes for 200 items.

Fixed-size is the right default: predictable size makes VRAM budgeting deterministic, overlap prevents boundary fragmentation, and it outperforms semantic chunking on most standard benchmarks when overlap is set correctly (20–25% of chunk size). Use semantic chunking when: (1) your corpus has strong paragraph-level semantic boundaries (academic papers, legal sections), (2) chunk size variance is acceptable in your retrieval pipeline, (3) you have measured a recall@5 improvement > 10% on your golden eval set. Semantic chunking adds spaCy dependency and slower indexing — only adopt when it provably improves retrieval quality on your domain.

Parent-child retrieval indexes small child chunks (128 tokens) for semantic matching precision, but returns the parent chunk (512 tokens) as the generation context. The small child chunk improves retrieval precision (higher signal-to-noise in the embedding), while the larger parent provides sufficient context for generation. Outperforms fixed chunking when answers require context beyond a 128-token window but retrieval precision is being hurt by 512-token chunk noise. Typical improvement: precision@1 increases 8–15% on question-answering benchmarks. Overhead: 2× the number of indexed vectors (one per child chunk) — acceptable at < 10M chunks.

Run a controlled experiment on your golden Q&A eval set (200+ items): (1) Recall@k — does the relevant chunk appear in the top-k retrieved? Measures retrieval quality independently of generation. (2) Answer extraction rate — given the relevant chunk is retrieved, does the LLM correctly answer the question? Measures generation quality. (3) End-to-end answer quality — RAGAS faithfulness and answer correctness. If recall@5 is low (< 70%), chunking or retrieval is the bottleneck — fix before tuning generation. If recall@5 is high but end-to-end quality is low, the generation model or prompt is the bottleneck.

Monitor cosine centroid drift: weekly, embed 100 random production queries with both the current query model and the model used to build the index. If the average cosine similarity between the two embedding spaces drops below 0.95, the models are diverging (or were never aligned). Add a startup assertion: the serving code reads the embedding_model_version tag from the index metadata and asserts it matches the configured query model. Any mismatch prevents the service from starting — forces explicit migration before serving.

At 512 tokens/document with text-embedding-3-small ($0.02/1M tokens): 10M × 512 tokens / 1M × $0.02 = $102.40. API rate limit: 10M tokens/minute → 10M × 512 / (10M) = 512 minutes = 8.5 hours at max rate. Practical timeline with batching and backoff: 12–18 hours. Optimise: cache embeddings for documents that have not changed (hash document content — if hash unchanged, reuse existing embedding). For a corpus with 30% document churn, only re-embed 3M documents → $30.72 and 3–5 hours.

Use a collection naming convention that encodes both tenant and model version: tenant_{id}_docs_{model_version}. The tenant config stores their active collection name. When a new embedding model releases: (1) Create new collections for opted-in tenants in parallel. (2) Validate recall@5 per tenant. (3) Switch opted-in tenants to the new collection. (4) Keep old collections for 30 days as rollback. Tenants on the old model see no disruption — they continue querying their old collection until they opt in. This prevents a single re-index event from affecting all tenants simultaneously.

BM25 outperforms dense retrieval for: (1) Exact keyword queries — product codes (SKU-4892), version numbers (v2.3.1), legal clause references (section 7.4(b)). Dense embeddings smear these specifics across the semantic space and miss exact matches. (2) Rare domain terms — medical jargon, proprietary acronyms — that appear infrequently in the embedding model's training data. (3) Code search — variable names, function signatures, error codes. (4) Queries where the user is trying to find a specific document they know exists (known-item retrieval). Pure dense retrieval fails on these; hybrid search with BM25 captures them.

Run a grid search over k ∈ {10, 20, 30, 60, 100} on your golden eval set (200 QA pairs). For each k: compute RRF scores, take top-5, measure recall@5. Choose the k that maximises recall@5. For typical production RAG with k=20 candidates per retriever, k_rrf=30 often outperforms the default k=60 because it provides better rank differentiation. If you retrieve different numbers from BM25 and dense (e.g., k_bm25=50, k_dense=20 for better lexical coverage), use weighted RRF: score = α/BM25_rank + (1-α)/dense_rank where α is tuned on the eval set.

Use Qdrant's native sparse-dense hybrid search (available in Qdrant v1.7+): store both the dense embedding and a sparse BM25 vector (using a sparse encoder like SPLADE or BM42) in the same collection. Qdrant handles the fusion natively in one query. Alternatively, use Weaviate with its built-in BM25+dense hybrid (alpha parameter). This eliminates the need for Elasticsearch as a separate BM25 index and reduces operational complexity. The tradeoff: BM25 via Elasticsearch is more mature and tunable; native sparse-dense is simpler to operate but less configurable.

At 100k docs/day with 512 tokens/doc: embedding cost = 100k × 512 × $0.02/1M = $1.02/day. Processing architecture: (1) Ingest queue (Kafka or SQS) buffers incoming documents. (2) Embedding workers (auto-scaled based on queue depth) embed in batches of 100. (3) Qdrant upsert in batches of 1,000. (4) Async stale embedding checker runs every 6 hours and queues re-embeddings for updated documents. (5) Full rebuild on Sunday night if delete count exceeds 20% since last rebuild. Monitor: embedding queue depth (alert if > 10min backlog), upsert latency, stale document count.

50M × 1536-dim float32 = 50M × 1536 × 4 bytes = 307GB for vectors alone. HNSW graph adds ~50% overhead = 460GB total. This requires a Qdrant cluster with at least 512GB RAM (use 6 × 96GB nodes). For cost reduction: use int8 quantization (reduces to 50M × 1536 × 1 byte = 77GB vectors + graph = ~115GB) with < 1% recall degradation. Memmap storage: keep cold vectors on NVMe SSD, hot vectors in RAM — Qdrant's mmap mode enables this. Practical hardware: 2 × 64GB RAM nodes with 2TB NVMe SSD each handle 50M vectors with mmap at < 20ms P99 search latency.

Three-gate validation before traffic cutover: (1) Completeness check — count of documents in new index matches source document count (within 0.1%). (2) Recall@5 comparison — run 200-item golden Q&A set against both old and new indexes; new index recall must be ≥ old index recall − 2%. (3) Latency check — P99 search latency on new index must be ≤ old index P99 × 1.2. Only after all three gates pass does the alias flip. Automate these gates as part of the rebuild pipeline — a human should never be manually validating index quality for a routine rebuild.

Low context precision (< 0.70): retrieval is returning many irrelevant chunks alongside relevant ones. Fix: tighten the retrieval query (better keyword extraction), increase the similarity threshold, or use reranking to filter irrelevant chunks before passing to the LLM. Low context recall (< 0.75): relevant information is not being retrieved at all — the answer facts are not in the top-k chunks. Fix: increase k, improve chunking (boundary fragmentation), or tune the embedding model. They diagnose different problems: precision is about noise in retrieval, recall is about coverage.

CI (every relevant code change): run RAGAS on the full 200-item golden set. Trigger: changes to prompt templates, chunking config, embedding model, retrieval parameters, or reranking logic. This gate blocks bad changes before they reach production. Production (continuous sampling): run RAGAS on 1% of production traffic using LLM-as-judge approximations of faithfulness and relevancy — full RAGAS on production traffic is too expensive ($10/200 items × 1% of 100k calls/day = $5,000/day). Weekly spot-check: 50 human-reviewed production examples for ground truth quality measurement.

Bootstrap with LLM-generated synthetic ground truth (acceptable for initial setup, not final validation): (1) Take 500 representative documents from your corpus. (2) Use GPT-4o to generate 2–3 questions per document and the corresponding ground-truth answers. (3) Human review: filter out bad questions (ambiguous, unanswerable, too easy). (4) Result: 200–500 high-quality QA pairs. For ongoing maintenance: sample 20 real production queries per week, have a human annotate ground-truth answers, and add them to the golden set. Refresh quarterly to stay representative of current query distribution.

Five elements: (1) Explicit rubric — each score level (1–5) is defined with concrete criteria, not vague adjectives. "5=every claim supported" is better than "5=excellent". (2) One dimension per prompt — evaluating groundedness and helpfulness in the same prompt leads to conflated scores. Use separate prompts for each dimension. (3) Chain-of-thought reasoning — require the judge to explain before scoring: "first identify unsupported claims, then assign a score". Reasoning reduces variance and catches edge cases. (4) JSON output format — structured output prevents parsing failures in automated pipelines. (5) Examples — include 2–3 scored examples in the prompt to anchor the rubric; the judge's calibration improves significantly with concrete references.

Disagreement signals judge miscalibration. Protocol: (1) Compute Cohen's κ on 100 items with human labels — if κ < 0.70, the judge needs fixing. (2) Inspect disagreements: sample 20 items where judge and human differ by > 1 point. (3) Identify patterns: does the judge over-rate verbose answers? Under-rate answers with hedging language? (4) Revise the judge prompt to address the specific failure pattern — add rubric examples for the edge cases causing disagreement. (5) Re-calibrate on 100 new items — target κ > 0.75 before using judge scores for production decisions.

Sample 1% of production requests (or a stratified sample by query type). For each sample: run the judge on (query, retrieved_context, response) and log the groundedness score. Compute rolling 7-day average groundedness — alert if the average drops below 0.80. Track hallucination rate (groundedness < 3) — alert if it exceeds 2%. This costs approximately $10/day at 1% of 10,000 calls/day × $0.01/judge call. Plot judge scores on a time series dashboard aligned with code deployment events — quality regressions from bad prompt or retrieval changes appear as inflection points on the score trend.

Refresh quarterly using the same clustering process on recent production queries. Before adding new items, run the existing golden set against the current system to establish a stable baseline. Add 20% new items per quarter (100 new items for a 500-item set) — focus on new query types that emerged since the last refresh. Retire items that no longer represent current production traffic (query type became obsolete). Version the golden dataset in DVC or Git with a semantic version tag — eval results are always tied to a specific dataset version for reproducibility.

Inter-annotator agreement (IAA) measures how consistently two annotators label the same item independently. Cohen's κ > 0.70 indicates substantial agreement — annotations are reliable enough to use as ground truth. To achieve this: (1) Provide clear, specific annotation guidelines (10+ pages with examples for each answer quality level). (2) Run a calibration session: annotators jointly label 20 training items and resolve disagreements with discussion. (3) Use a structured annotation schema (binary relevance per retrieved chunk + correctness rating for the answer) rather than holistic quality scores. (4) Adjudicate disagreements (κ < 0.5 on an item) with a senior annotator as tie-breaker.

Use synthetic data generation: (1) Sample 200 representative documents from your corpus. (2) Use GPT-4o to generate 3–5 diverse questions per document with answers grounded in the document text. Prompt: "Generate 3 questions about this document that require reading the document to answer correctly. For each question, provide the answer as a direct quote or paraphrase from the document." (3) Human review: filter out questions that are too easy (keyword match only), too hard (multi-hop across documents), or ambiguous. (4) Add hard negatives: for each QA pair, manually identify 2 plausible-but-wrong documents. Target: 200 synthetic items pre-launch, transition to real production queries within 4 weeks of launch.

Four strategies: (1) Parallelize: split the 200-item golden set into 4 batches of 50, run simultaneously with 4 workers — 6.7min becomes 1.7min. (2) Cache LLM responses: hash (prompt, model, temperature) and cache in Redis — for unchanged prompts, items that already have responses skip the LLM call. (3) Incremental eval: only re-run items that could be affected by the specific change (a chunking change re-runs retrieval-dependent items; a prompt change re-runs generation-dependent items). (4) Cheap judge for fast eval: use GPT-4o-mini as the judge for PR evals, GPT-4o only for nightly full evaluation. Target: < 3 minutes for PR eval gate.

Sources of variance: (1) LLM non-determinism (temperature > 0) — fix by setting temperature=0 for all generation and judge calls. (2) ANN search non-determinism (different HNSW entry points) — fix by setting a random seed in Qdrant search or using exact search for the eval index. (3) RAGAS LLM-based metrics variance — fix by running each item 3× and using the median score. (4) Floating-point variance in embeddings — negligible after the above fixes. With temperature=0 and exact search, variance should be < 0.005 across runs. If variance is still > 0.01, investigate which specific metric is varying and apply targeted fixes.

Escalation path: (1) Review the specific failing metric and the failing items — is it a meaningful regression or a known flaky item? (2) If the regression is in a query type not affected by the change (e.g., a prompt change causing a retrieval metric failure — unlikely but check), document and skip. (3) If the regression is real, fix it before deploying — a CI gate exists for exactly this scenario. (4) If deploy is truly urgent (security patch, P0 bug), override with an incident record, deploy anyway, and create a follow-up ticket to fix the regression immediately post-deploy. Never permanently disable the CI gate — the next regression will ship silently.

A run is LangSmith's term for a single traced operation — equivalent to a span in OpenTelemetry. A trace is the full tree of runs for one user request (root run + all child runs). A span (OpenTelemetry term) maps to a run. The root run has no parent_run_id; child runs have parent_run_id pointing to their parent. A chain run (root) contains retriever runs + LLM runs + tool runs as children. Each run has a run_id (UUID) used for linking — store the root run_id in your application logs and database for cross-system correlation.

Propagate the LangSmith root run_id as a correlation ID through your entire request pipeline. (1) At request entry, capture the run_id from LangSmith's run context. (2) Add it to your structured application log as X-Trace-Id. (3) Store it in the database row created by this request (e.g., the chat message table: message.trace_id = run_id). (4) When a user complains about a specific message, query the database for that message's trace_id, then look up the full trace in LangSmith. This gives you the complete picture: user query → retrieval → LLM call → response, with all intermediate inputs and outputs.

Process: (1) Filter LangSmith traces to the period before and after the model upgrade (filter by start_time and model_name). (2) For post-upgrade traces with user_feedback=-1 (negative) or low LLM-as-judge scores, inspect the full span tree. (3) Look for patterns: are the failing traces concentrated on specific input types (long queries, queries with code, multi-turn conversations)? (4) Compare the LLM input (retrieved context + prompt) between passing and failing traces — is the issue in retrieval or generation? (5) Reproduce a failing trace locally by replaying the exact input from the trace. (6) File a regression report with the specific input patterns and the new model's failure mode.

The most valuable queries in practice: (1) Find all failed requests in the last hour: SELECT * FROM llm_logs WHERE finish_reason='content_filter' AND timestamp > NOW()-1h ORDER BY timestamp DESC. (2) Cost by feature: SELECT feature, SUM(cost_usd) FROM llm_logs WHERE DATE(timestamp)=CURRENT_DATE GROUP BY feature. (3) Latency percentiles by model: SELECT model_version, PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY latency_ms) FROM llm_logs GROUP BY model_version. (4) Trace a specific session: SELECT * FROM llm_logs WHERE session_id='abc123' ORDER BY timestamp. Store logs in Athena (S3+Parquet) for cost-efficient querying at scale — 1TB/month of logs costs ~$5 to query with Athena vs ~$50 in BigQuery.

Do not delete logs — make them unattributable. Maintain a user_id → session_id mapping in a mutable store (PostgreSQL). On erasure request: delete the mapping rows for the user's session IDs. The logs in S3 still contain session_ids but the mapping linking them to the user is gone — logs become effectively anonymised from that user's perspective. This approach satisfies GDPR erasure requirements without the operational complexity of deleting immutable S3 objects (which would require Object Lock bypass procedures). Retain the audit trail of the erasure request itself in a compliance log.

Balanced approach: 90 days in hot storage (S3 Standard + Athena queryable) for incident debugging and monitoring. 2 years in cold storage (S3 Glacier Instant Retrieval) for compliance audits and model training data. 7 years in archive (S3 Glacier Deep Archive) for regulated industries (finance, healthcare) with SOC 2 or HIPAA requirements. Implement S3 Lifecycle rules to automate tier transitions. Compress logs with Parquet + Snappy — typical compression ratio 5–10× vs JSON, reducing storage cost proportionally. At 100k requests/day × 1KB/log: 100MB/day → 3GB/month → 36GB/year → $0.83/month in Glacier.

Use your historical data to set dynamic thresholds. Segment by query type: FAQ responses are typically 50–150 tokens; code generation responses are 200–2000 tokens. A single global threshold will produce false positives for one type and miss anomalies for another. Compute per-query-type rolling mean and standard deviation. Set alert threshold at mean ± 3σ per type. For a new system without history, start with ±200% of the configured max_tokens as the outer bounds, then tighten to ±3σ after 2 weeks of production data.

Run anomaly checks asynchronously: (1) Return the response to the user immediately. (2) Publish the response to a queue (Redis Pub/Sub or Kafka) for async anomaly analysis. (3) Anomaly workers consume from the queue and run length, language, and toxicity checks. (4) Alert if anomalies found — no user-facing latency impact. Exception: safety-critical applications (children's content, regulated industries) must run toxicity synchronously before returning the response (< 50ms with Detoxify on GPU). For general applications, async anomaly detection is the right tradeoff between safety and latency.

Immediate response: (1) Identify the affected session_ids and user_ids from the toxicity alert. (2) Pull the full trace for the highest-scoring responses from LangSmith. (3) Determine the cause: was it a specific input pattern (jailbreak attempt), a model change, or a prompt change? (4) If jailbreak-driven: your guardrails are being bypassed — activate the incident response playbook, add the attacking patterns to the injection classifier training set, and consider rate-limiting the affected user IDs. (5) If model-change-driven: roll back the model version immediately and open a P1 incident. (6) If prompt-change-driven: revert the prompt version in the registry — no code deploy needed.

Each implicit signal needs a context window to be valid: copy-to-clipboard is positive only if it occurs after reading (dwell time > 10s on the response); follow-up question is positive only if it is a related follow-up (cosine similarity > 0.7 with the previous query), not an unrelated new topic; session abandon is negative only if abandon occurs within 5s of the response (user left immediately) — long sessions that end naturally are not abandons. Build a signal quality filter that applies these context rules before logging feedback. Validate: correlate implicit signals with explicit thumbs-up/down on a held-out set and measure Pearson correlation — target > 0.6 per signal type.

For RLHF preference data: minimum 5,000 high-quality preference pairs (human-reviewed) for a meaningful reward model. For DPO: 3,000–10,000 preference pairs. For supervised fine-tuning on corrections: 1,000 high-quality (query, preferred_response) pairs. At 1% annotation rate with 5% thumbs-down rate: 100k daily requests generates 1k feedback events/day, 50 annotated examples/day → reaching 1,000 annotated pairs takes 20 days. Trigger fine-tuning monthly (collecting ~1,500 annotated pairs) or when any eval metric drops > 5% from the base model baseline.

Six-step loop: (1) User signals collected via explicit feedback UI and implicit event tracking → Postgres feedback table. (2) Annotation queue: sample low-confidence + thumbs-down items. (3) Human annotation: label (preferred response, rejected response) for each item. (4) Dataset prep: format as DPO preference pairs or SFT correction pairs. (5) Fine-tuning: monthly DPO run using PEFT/LoRA on the accumulated pairs. (6) Eval gate: fine-tuned model must pass the RAGAS golden eval and MT-Bench regression (< 5% MT-Bench drop vs base) before promotion to production. Total loop latency: ~30 days from user signal to deployed improvement — acceptable for non-critical quality regressions, too slow for safety issues which need immediate attention.

Two strategies: (1) Set temperature=0 for all generation and judge calls — makes outputs deterministic for the same input. This is the right default for eval pipelines. (2) If you must evaluate at non-zero temperature (to test creative writing quality): run each item 3× and report mean ± std. Gate on mean score, not a single run. In CI, set temperature=0 always — non-determinism in CI causes flaky tests. Reserve temperature variation testing for nightly or pre-release evaluations.

After every merge to main: run the full eval suite and store the resulting metrics as data/baseline_metrics.json in the repo (or in S3 as an artifact keyed by the main branch commit hash). PRs read this file to get the current main-branch baseline. Never update baseline_metrics.json in a PR — only update it in the post-merge CI job. If you store baselines in S3: key by {branch}/{commit_sha}/metrics.json. The PR eval downloads {main_branch}/latest/metrics.json as its baseline.

Actionable eval means knowing which items failed and why. Generate a per-item report: for each of the 200 golden items, show (query, retrieved chunks, answer, faithfulness score, judge reasoning). Sort by worst faithfulness score. The top 10 worst items reveal the failure mode: are they all queries about a specific topic? All using a specific retrieval pattern? All from a specific document type? This per-item analysis converts a 0.78 faithfulness score (abstract number) into "the model is not faithfully answering questions about pricing when retrieved context has conflicting pricing in different chunks" (actionable engineering task).

Post-hoc tests validate what the current implementation does — they codify the existing behaviour, not the desired behaviour. EDD writes the eval against the requirement before any implementation, so the eval defines what "done" means. A post-hoc test for a prompt that adds unnecessary disclaimers would pass because it tests "does the prompt add disclaimers" (yes, it does). An EDD eval would fail because it tests "does the response answer directly without disclaimers" (no, it does not). EDD also forces engineers to articulate the quality requirement precisely before touching the prompt — which often reveals that the requirement was underspecified.

Conflicting eval items are a signal that the prompt is trying to serve two incompatible user intents from the same endpoint. Resolution options: (1) Separate endpoints: one for quick answers (concise), one for deep dives (detailed). Different prompts, different evals, different routing. (2) Explicit user intent signal: add a query parameter or system instruction that specifies the desired response length — the model honours it, and evals are parameterised accordingly. (3) If the conflict is inherent (some queries genuinely need brevity, others need depth): use a length classifier and dynamically inject length instructions into the prompt, then eval both paths separately.

Track three metrics: (1) Mean time to detect (MTTD) quality regressions: before EDD, quality regressions are found via user complaints (days); with EDD, they are found at CI time (minutes). (2) PR revert rate: PRs that regress quality and must be reverted post-merge. EDD should drive this to near-zero. (3) Prompt iteration time: how many prompt changes are needed to achieve a target quality metric? EDD provides a clear pass/fail signal per change, reducing the random walk of prompt exploration. Measure these quarterly — teams that adopt EDD typically see MTTD drop from days to hours within 6 weeks.

Compute Cohen's κ on the overlapping annotations (configure Label Studio with overlap=2 to assign each item to 2 annotators). Interpretation: κ > 0.8 = near-perfect agreement, 0.6–0.8 = substantial (acceptable for most tasks), 0.4–0.6 = moderate (guidelines need revision), < 0.4 = poor (fundamental disagreement about the task). To improve κ: (1) Hold a calibration session where annotators jointly review 20 items and discuss disagreements. (2) Add concrete examples to the guidelines for each decision boundary. (3) Simplify the task: replace 5-point Likert with 3-point (good/ok/bad) or binary (acceptable/not acceptable). (4) For persistent low κ on specific item types, consider splitting them into a separate annotation task with more specific guidelines.

Present annotators with (query, response_A=old_prompt, response_B=new_prompt) in random order (blind to which is A/B). Collect 200 pairwise preferences for each tested prompt pair. Preference rate for B: P(prefer B) = count(B preferred) / total. Bradley-Terry model converts pairwise preferences to quality scores: score = log(win_rate / (1 - win_rate)). Statistical significance: binomial test, p < 0.05 requires approximately 180 pairs for a 60% win rate to be statistically significant. If P(prefer B) > 0.58 with p < 0.05, promote variant B. Pairwise preference is more sensitive than per-response ratings for detecting small quality differences.

Use a quality pyramid: (1) Automated eval (RAGAS + LLM-as-judge) on 100% of items — $0.001/item. (2) Spot-check human eval on 1% of items (100 items/week from 10k) — $0.10/item from crowd = $10/week. (3) Expert human eval on the worst 20 items from the crowd eval (identified by low LLM-as-judge scores or crowd disagreement) — $2/item = $40/week. Total: $50/week for 10k items with 3-tier quality validation. The automated layer catches 80% of regressions; the crowd layer catches subtle quality issues; the expert layer catches domain-specific failures.

Balance cost and statistical power: to detect a 5% faithfulness regression with 80% power (β=0.20) and α=0.05, you need ~400 judged samples. At 10k requests/day, 4% sampling rate gives 400 samples/day — adequate for daily regression detection. At 1% sampling, you get 100 samples/day — adequate for weekly trend detection but too slow for same-day incident detection. Rule: if you need to detect regressions within 24 hours, sample at 4%+. If weekly trend monitoring is sufficient, 1% is cost-effective. Monitor the judge cost daily and adjust the rate if it exceeds 10% of the feature's total LLM cost.

Use control charts: compute the rolling 7-day mean and standard deviation for each monitored metric. Raise an alert only when: (1) A single day's metric falls more than 3σ below the 7-day mean (unusual single-day event), OR (2) 5 consecutive days trend in the same negative direction (systematic drift). This avoids false alerts from normal statistical variation while catching genuine regressions early. Also use event markers: annotate the monitoring chart with deployment events (model upgrades, prompt changes, index rebuilds) so regressions are correlated with their cause within minutes of appearing.

Use a three-layer dashboard: (1) Executive summary (one number): monthly quality score (0–100, normalised from RAGAS faithfulness + user feedback rate). Green/yellow/red RAG status. (2) Operational view (7 metrics): daily trend charts for faithfulness, answer relevancy, refusal rate, latency P99, cost/request, thumbs-up rate, hallucination rate. Each with a target line and alert threshold. (3) Engineering view (full detail): per-query-type breakdowns, model version comparison, prompt version comparison, per-item failing examples. Stakeholders see the summary layer; on-call engineers see the operational layer; debugging engineers dive into the engineering layer.

Use ChatML for fine-tuning models you will deploy via OpenAI-compatible APIs (vLLM, LiteLLM, Ollama) — the format matches exactly how the model will receive inference requests, minimising format mismatch degradation. Use ShareGPT for multi-turn conversational fine-tuning where you have real conversation logs to learn from. Use Alpaca only for simple single-turn instruction tasks where you have clean (instruction, output) pairs — it is the simplest format but lacks system prompt and multi-turn support. When in doubt, use ChatML: it is the most expressive and maps directly to production.

Rule of thumb: (1) Task adaptation (add a new capability to an existing base model): 1,000–5,000 high-quality examples. (2) Behavioural fine-tuning (change tone, format, style): 500–2,000 examples. (3) Domain adaptation (make a general model perform well on a specific domain): 10,000–50,000 examples. (4) General instruction tuning (improve overall instruction-following): 500k+ examples. Quality beats quantity: 1,000 carefully curated examples consistently outperform 10,000 noisy examples. Measure: plot validation loss vs training steps — if validation loss diverges from training loss before 3 epochs, you have insufficient data for the task complexity.

Three strategies: (1) Diversity capping (preferred): embed examples, cluster by topic, cap at equal count per cluster — preserves natural variation within each class. (2) Oversampling: duplicate minority class examples — risks overfitting to those specific examples, especially with a small minority class. (3) Weighted loss: assign higher loss weight to minority class tokens — requires modifying the training loop. Evaluate: after fine-tuning, check per-class accuracy on the test set. If any class's accuracy is > 20% below the average, the dataset imbalance is still causing issues and more diversity work is needed.

Full fine-tuning updates all model weights — highest capability gain but requires 2–4× the model's VRAM for weights + gradients + optimizer states, and risks catastrophic forgetting of pre-training knowledge. LoRA updates only small adapter matrices (0.1–1% of total parameters) — much lower VRAM, trains faster, much lower forgetting risk. Use LoRA for: task-specific fine-tuning (code, customer support, domain Q&A), format adaptation, tone/style changes. Use full fine-tuning for: continued pre-training on a large new corpus, fundamental capability changes (adding a new language), cases where LoRA's capacity ceiling is measurably limiting task performance.

Use QLoRA when GPU VRAM is the constraint: QLoRA enables 7B on 24GB GPU, 13B on 40GB GPU. Quality difference vs LoRA in BF16 is < 1% on most tasks — negligible. Use LoRA in BF16 (without NF4 base quantization) when VRAM is sufficient and training speed matters: BF16 LoRA trains 20–30% faster than QLoRA because NF4 dequantization adds overhead per forward pass. For production pipelines with many fine-tuning runs: use QLoRA on smaller (24GB) GPUs for experiments, then do one final BF16 LoRA run on larger (80GB) GPUs for the production model.

Four strategies: (1) Use LoRA instead of full fine-tuning — adapter layers touch a tiny fraction of weights, dramatically reducing forgetting. (2) Mix a small fraction (5–10%) of pre-training or general instruction data into the fine-tuning dataset — keeps general capabilities active. (3) Lower learning rate (1e-4 instead of 2e-4) and more epochs — gentler weight updates. (4) EWC (Elastic Weight Consolidation) for full fine-tuning: compute Fisher information matrix on a general capability eval set, add a regularisation term that penalises large deviations from the original weights for high-importance parameters. Monitor MT-Bench score on the fine-tuned model — if it drops > 5% vs the base model, forgetting is occurring.

Decision framework: (1) Measure GPT-4o few-shot accuracy on your task golden set — if it is already > 90%, fine-tuning will likely improve to 92–95% (marginal gain may not justify cost). (2) Measure GPT-4o-mini few-shot accuracy — if it is < 70%, fine-tuning a 7B model can close most of that gap while being 10× cheaper per call. (3) Consider volume: at 100k calls/day, GPT-4o costs $2.50/1M tokens × avg 1k tokens = $250/day = $91k/year vs. self-hosted fine-tuned LLaMA 3 8B at $2,000/month GPU cost. Break-even is roughly 40 days of API usage. (4) Consider latency: self-hosted fine-tuned models have predictable latency; API calls have variable latency under load.

Generalization test: create a test set with paraphrased versions of training examples (same facts, different phrasing). If the fine-tuned model performs well on original phrasing (0.89) but drops > 15% on paraphrased versions (< 0.75), it is memorising rather than generalising. Also test on queries that were not in the training distribution but are adjacent: fine-tuning on Python Q&A should generalise to new Python questions, not just ones similar to training examples. If the model fails on clearly in-distribution but unseen examples, the dataset is too small or too homogeneous — add more diversity.

An eval card documents: (1) Model ID and base model version. (2) Training dataset: source, size, format, quality filters applied. (3) Task-specific metrics: accuracy, ROUGE-L, human preference rate vs baseline — on the held-out test set only. (4) General capability regression: MT-Bench score vs base model (delta). (5) Comparison: fine-tuned vs GPT-4o vs GPT-4o-mini on the task eval. (6) Known limitations: query types where the model underperforms (from per-cluster analysis). (7) Intended use and out-of-scope uses. (8) Training details: hyperparameters, compute used. Store the eval card in the model registry alongside the model artifact — anyone deploying the model can see exactly what was evaluated and how.

Use merge_and_unload() for: (1) Production deployment where only one task is served — the merged model loads faster and has no PEFT overhead. (2) When exporting to non-PEFT serving frameworks (vLLM, TorchScript). (3) When you want to quantize the fine-tuned model with AWQ — quantize the merged model. Keep adapters separate when: (1) Serving multiple task-specific adapters from one base model (multi-adapter serving). (2) Continual fine-tuning — start from the adapter, not the merged model. (3) A/B testing multiple adapter versions — swap adapters without reloading the base model.

Use MLflow Model Registry for adapter versioning: log the adapter as an artifact with tags (base_model_version, task, dataset_version, eval_metrics). Promote to "Staging" after passing the eval gate; promote to "Production" after canary validation. Store adapter weights in S3 with the path following the naming convention: s3://ml-models/adapters/{base_model}/{task}/{dataset_version}/{step}/. The registry entry contains the S3 path, all tags, and a link to the training run that produced the adapter. This gives full traceability: which training run, which data version, which base model, and what eval metrics were achieved.

Architecture: (1) Load the base model once in BF16 or INT4 on GPU. (2) Load all adapters in memory using PEFT's load_adapter() — adapters are small (100MB–1GB for r=16), so loading 5–10 adapters adds minimal memory. (3) An intent classifier (DistilBERT, < 5ms) routes each request to the appropriate adapter name. (4) model.set_adapter(adapter_name) takes < 1ms — negligible latency. (5) If adapters have very different optimal temperatures or sampling settings, store these alongside the adapter in the registry. This gives you 5–10 task-specific models from one base model instance — dramatically more cost-efficient than running separate fine-tuned models for each task.

RLAIF (Reinforcement Learning from AI Feedback) uses a capable AI (Claude, GPT-4o) instead of humans to label preference pairs. RLAIF is better when: (1) Scale is needed quickly (RLAIF generates 10,000 pairs/day vs 500/day for human annotators). (2) The task requires consistent application of objective criteria (factual accuracy, code correctness) that AI can assess reliably. (3) Cost is a constraint ($0.01/pair AI vs $0.10–$5/pair human). RLAIF is worse when: (1) Preferences are deeply subjective (humor, cultural context, user personality fit). (2) The task requires real-world domain expertise (medical diagnosis, legal judgment). (3) You need annotation that is maximally different from the model's own biases — using GPT-4o to annotate GPT-4o's outputs has sycophancy risks.

10,000 pairs is the practical minimum for a task-specific reward model — below this, the reward model generalises poorly to out-of-distribution prompts. 50,000 pairs gives a robust reward model for diverse task coverage. For general-purpose alignment (Anthropic/OpenAI scale), millions of pairs are needed. Quality thresholds: filter out pairs with annotator confidence < 60% (ambiguous) and pairs where both responses are poor quality (neither should win — the model learns nothing useful from these). After filtering, a clean 10k-pair dataset consistently outperforms a noisy 50k-pair dataset.

Signs of gaming: (1) Annotator completes tasks 3× faster than the group median — likely not reading responses. (2) One annotator's label agreement with others is < 50% on calibration items (which have known correct labels). (3) All annotations from one annotator in a session lean 90%+ to "A" (position bias). Mitigations: (1) Randomise A/B position — "preferred" should not correlate with position. (2) Include calibration items with known correct answers (agreed by 3+ humans) — fail annotators who get < 80% of calibration items right. (3) Set a minimum time per annotation (enforce via UI) — responses < 10 seconds are flagged for review. (4) Track per-annotator IAA continuously — remove annotators who fall below κ=0.50.

Monitor three signals: (1) Reward score distribution: if the mean reward score increases over PPO training steps but human preference ratings plateau or decline, reward hacking is occurring. (2) Response length: reward-hacking policies often become more verbose (length correlates with reward in poorly calibrated RMs). Alert if mean response length increases > 30% over fine-tuning steps. (3) Specific reward-hacking patterns: hedging language ("I think", "probably", "you might want to consider") often gets rewarded as "cautious" but is actually less helpful. Check if these phrases increase in frequency over PPO steps.

Minimum: 68% accuracy on held-out preference pairs (18% above chance). Below this, the RM is providing noisy gradients that may harm the policy. Target: 72–78% accuracy. Above 80% is often overfitting to the specific annotator population rather than capturing genuine preferences. At 72% accuracy, the RM will make mistakes on roughly 28% of comparisons — this noise in the training signal is acceptable because PPO training uses many RM evaluations per policy update, and the error averages out. Monitor RM accuracy at the start of PPO training; if it drops, the RM needs retraining.

RLHF (PPO-based): trains a reward model, then uses PPO to update the policy to maximise the RM score. Advantages: can handle complex reward functions, more flexible. Disadvantages: requires training 4 models simultaneously (SFT policy, reference policy, reward model, value model), high VRAM, training instability, reward hacking. DPO: directly optimises on preference data without a reward model — equivalent to implicit reward modelling with the policy itself. Advantages: trains only 2 models (policy + frozen reference), more stable, lower VRAM, no reward hacking. Disadvantages: requires complete preference pairs, cannot easily incorporate non-preference reward signals. Current best practice: use DPO for most alignment tasks unless you need online RLHF (generating new responses during training) or complex composite reward functions.

β controls the KL divergence penalty between the trained policy and the reference policy. Lower β = more freedom to deviate from reference; higher β = stays closer to reference. Guidelines: β=0.1 for task adaptation where you want maximum learning from preferences; β=0.2–0.3 for general alignment where moderate deviation is acceptable; β=0.4–0.5 for safety alignment where safety properties of the SFT model must be preserved. Tune β empirically: train at β=0.05, 0.1, 0.2, 0.5 and evaluate on the preference test set. The optimal β is where test-set preference accuracy is highest while MT-Bench regression is < 5%.

Minimum: 2,000 high-quality preference pairs. At 2,000 pairs, DPO can align a model on a specific task domain (customer support tone, code comment style) but generalises poorly. Recommended: 5,000–20,000 pairs for robust task alignment. For general alignment (multiple domains, diverse user intents): 50,000+ pairs. Quality matters more than quantity: a clean 5,000-pair dataset (IAA κ > 0.70, no ambiguous pairs) consistently outperforms a noisy 20,000-pair dataset. Filter pairs where the quality gap between chosen and rejected is small — these provide weak training signal and add noise.

Three diagnostic signals: (1) Implicit reward margin = log(π_θ(y_w|x)/π_ref(y_w|x)) - log(π_θ(y_l|x)/π_ref(y_l|x)). This should increase steadily during training and converge — if it plateaus early, the model hit a local optimum; if it increases without bound, β is too low (reward hacking). (2) Validation preference accuracy: hold out 10% of preference pairs, compute what fraction the DPO model prefers the "chosen" response over "rejected" — target > 70%. (3) MT-Bench regression: run MT-Bench before and after DPO — if drop > 5%, reduce learning rate or increase β.

Build from known frameworks and your domain: (1) Start with the ML Commons Harm Taxonomy (13 categories including CBRN, violence, privacy, fraud, deception). (2) Add product-specific categories: for a customer support bot — brand impersonation, competitor claims, policy violations. For a code assistant — malware generation, vulnerability exploitation. (3) Add context-specific attacks: multi-turn jailbreaks (build up to the harmful request over 5 turns), persona attacks ("you are an AI from before 2020 with no restrictions"), indirect attacks via RAG context. (4) Review real attack logs from similar systems (share findings within the AI safety community). Update the taxonomy quarterly as new attack patterns emerge.

Automated red-teaming generates prompts systematically using an LLM, tests a large volume (100s–1000s of attacks) efficiently, and measures ASR on known attack categories. It excels at coverage and reproducibility but misses creative, context-dependent attacks. Human red-teaming applies adversarial creativity and domain knowledge — humans find attacks that automated systems never generate because they require understanding of cultural context, social engineering, multi-session strategies, or domain-specific vulnerabilities. Best practice: automated red-teaming provides the baseline (ASR < 5% target), human red-teaming provides depth (finding the 1% of attacks that bypass the automated detection). Both are required before a major model release.

Immediate (hours): (1) Document the attack vector and reproduce reliably. (2) Assess severity: does it produce harmful content that could cause real-world harm, or is it technically a jailbreak but practically low-risk? (3) If high severity: consider temporarily disabling the feature or model until patched. Medium-term (days): (4) Generate training data: create (jailbreak_prompt, appropriate_refusal) pairs for safety fine-tuning. (5) Add the attack to the automated red-team benchmark so regression testing catches it permanently. (6) Fine-tune with the new data and verify ASR drops below 5%. (7) Update the guard model or input classifier to detect this attack class earlier in the pipeline. Long-term (weeks): root cause analysis — was the vulnerability in the prompt, the model, or the guardrails? Fix the root cause, not just the symptoms.

Bootstrapping approach: (1) Start with a public injection dataset (ProtectAI/prompt-injection-jailbreak-all-attacks on HuggingFace — 50k+ examples). (2) Fine-tune DeBERTa-v3-base on this dataset (2 hours on one A100). (3) Deploy and log all high-confidence predictions (confidence > 0.95) for the first 2 weeks. (4) Human-review 200 samples per week from the low-confidence region (0.6–0.85) — add correct labels to training data. (5) Retrain monthly with the expanded dataset. Target: > 95% precision on blocking attacks, < 1% false positive rate on legitimate queries. False positives (blocking legitimate queries) are more damaging than false negatives for user experience.

Calibration process: (1) Embed 500 in-scope production queries and 200 clearly out-of-scope queries. (2) Compute the distance distribution from the in-scope centroid for each group. (3) Find the threshold where: (a) 95% of in-scope queries are below the threshold (< 5% false rejection rate), and (b) the maximum fraction of out-of-scope queries you are willing to serve is above the threshold. (4) Typical result: threshold around the 97th percentile of in-scope query distances. Plot the ROC curve and choose the operating point based on your false positive vs false negative cost tradeoff — over-rejection harms user experience, under-rejection allows off-topic abuse.

Common failure modes: (1) Masker incorrectly identifies a medical drug name as a PERSON entity. (2) Masker masks a product code that the LLM needs to look up. (3) Masker breaks a structured query (SQL, code) by replacing a variable name. Mitigation: (1) Run the masker in "analyze" mode first — log which entities are detected and their confidence scores. (2) Add a domain allowlist: entity patterns that match your product's identifiers (SKU-[0-9]+, USER-[A-Z]+) are excluded from masking. (3) For structured inputs (code, SQL), disable PII masking and use a different risk control (access control at the LLM level, not at the input). Always test the masker on a sample of real production queries before enabling it.

Use Llama Guard for: 14-category harm classification that generalises well across diverse harmful content types — it is the fastest to deploy and most battle-tested. Use NeMo Guardrails for: complex programmable safety logic (topical rails, fact-checking against a specific knowledge base, conversation flow control) — it requires more engineering but enables nuanced business-specific rules. Use custom classifiers for: domain-specific harmful content that general models miss (e.g., financial misinformation in a fintech app, medical misinformation in a health app). Best practice: Llama Guard as the base layer (low engineering cost, broad coverage), custom classifier for your top 2–3 domain-specific failure modes.

Measurement: sample 1,000 production responses that passed to users (no guardrail trigger) and 1,000 that were blocked. Human-review the blocked responses — false positives are legitimate responses that were incorrectly blocked. Target: false positive rate < 2% (< 20 of 1,000 blocked responses are legitimate). If false positive rate is high: (1) Raise the classifier threshold (0.7 → 0.8) — fewer blocks, higher precision. (2) Add a whitelist of safe phrases that should never trigger blocks. (3) Fine-tune the classifier with the false positive examples as negative training samples. Monitor false positive rate monthly — user complaints about "my question was refused" are the leading indicator that false positive rate is rising.

Stratified approach: (1) Fast path (< 50ms): run a lightweight toxicity classifier (DistilBERT-based, 20ms) on every response. This catches the highest-severity content with minimal latency impact. (2) Standard path (< 150ms): run Llama Guard on 100% of responses for the full 14-category classification — run asynchronously and only block if the response is genuinely harmful (Llama Guard accuracy is high, false positive rate is low). (3) Deep path (event-triggered): NeMo fact-checking only for responses about sensitive topics (medical, legal, financial) identified by a topic classifier. This ensures 95% of responses see only the 20ms fast path while high-risk responses get comprehensive checking.

Sources for jailbreak prompts: (1) Public datasets: JailbreakBench (1,500+ prompts), HarmBench (3,000+ red-team prompts), PromptBench, ChatGPT jailbreaks from Reddit and Discord communities. (2) Your own red-team sessions — add every successful attack immediately. (3) Failed injection attempts from production logs — if the injection classifier blocks a prompt with high confidence, embed it and add it to the store. Size: start with 5,000 diverse prompts (covering all major attack categories), grow organically. Use HNSW index in Qdrant for < 5ms lookup at 50,000 entries. Deduplicate: cosine similarity > 0.98 between two stored prompts → keep only one.

Defense in depth means no single failure point can compromise the whole system. Concrete implementation: Layer 1 (input, fast): Jailbreak embedding similarity check (3ms, blocks known patterns). Layer 2 (input, slow): Injection classifier (20ms, catches novel patterns). Layer 3 (system prompt): Instruction hierarchy hardening (zero latency — already in the prompt). Layer 4 (model weights): Adversarial fine-tuning (the model refuses at the weight level). Layer 5 (output): Llama Guard classification (80ms, catches successful jailbreaks). An attacker must simultaneously bypass all 5 layers. Layers 1, 2, 3, and 4 are input-side; layer 5 is output-side — a full stack attack requires different techniques for each layer.

Incident response for active jailbreak campaign: (1) Detection: jailbreak attempt rate spikes from 0.2% to 3% of traffic. (2) Immediate containment: identify the attacking pattern from logs, add it to the jailbreak embedding store, and bump the similarity threshold to 0.85 temporarily. (3) Rate limiting: apply aggressive rate limiting (100 requests/hour) to accounts with > 5 jailbreak attempts — legitimate users rarely hit this limit. (4) Communication: post a security advisory if user data was at risk, inform trust-and-safety team. (5) Post-incident: fine-tune the model on refusals for the specific attack pattern, add it to the automated red-team benchmark permanently. (6) Metrics: track ASR before and after each remediation step.

High-risk AI systems under EU AI Act Annex III include: AI in critical infrastructure, AI for employment decisions (hiring, performance evaluation), AI for education (access, assessment), AI for essential private services (credit scoring, insurance), AI for law enforcement, AI for border/migration control, AI for judicial decisions. If your LLM product influences decisions in these areas, it is high-risk and must: (1) Have a conformity assessment (internal or third-party). (2) Maintain technical documentation (training data description, model architecture, eval results, known limitations). (3) Implement human oversight mechanisms. (4) Register in the EU AI system database before deployment. (5) Have a quality management system. Non-compliance: fines up to €30M or 6% of global annual turnover.

The GDPR-SOC2 tension: GDPR requires erasure of personal data on request; SOC 2 requires immutable audit logs. Resolution: (1) PII masking before logging — the log never contains personal data in the first place (GDPR-compliant without needing erasure). (2) User-session mapping in a mutable PostgreSQL table (not in the immutable log) — erasure deletes the mapping, making logs unattributable. (3) Log only hashes of identifiers (SHA-256 of user_id) — the hash is in the log, the plaintext is in the mapping table. (4) GDPR Data Protection Impact Assessment (DPIA) for the logging system — document the legal basis for retaining session-level data for 7 years (legitimate interest: security/audit, documented in privacy policy). This architecture has been successfully audited under both frameworks at several EU-based AI companies.

Minimum viable audit log per request: request_id (UUID), session_id, user_id (hashed), feature, model_version, prompt_hash (SHA-256, not raw text), output_hash, input_tokens, output_tokens, latency_ms, safety_score, guardrail_decision (safe/blocked + reason), timestamp (UTC ISO8601). Why each field: request_id for correlation across systems, session_id for GDPR erasure mapping, prompt_hash for deduplication and incident investigation without PII exposure, guardrail_decision for safety audit, timestamp for time-based queries. Minimum retention: 90 days for operational debugging, 2 years for security incident investigation, 7 years if SOC 2 or in regulated industries. Storage: S3 + Athena (cost-efficient, queryable). Total cost: ~$10/month for 1M requests/day at 1KB/log.

LiteLLM supports live config reloads without restart: (1) Store the config in a Redis-backed config store (LiteLLM Pro feature) or an S3 object. (2) Post a PATCH request to LiteLLM's admin API to update routing rules or add a new provider. (3) LiteLLM applies the change within 30 seconds without dropping in-flight requests. For self-managed LiteLLM: use Kubernetes rolling update with maxSurge=1 and a readiness probe — the new pod loads the updated config and replaces the old pod without downtime. Always test config changes in staging with production-like traffic before applying to production.

Create model aliases that group providers by capability tier: (1) "llm-long-context" → GPT-4o (128k), Claude 3.5 Sonnet (200k) — for document analysis. (2) "llm-standard" → GPT-4o-mini (128k), Claude Haiku (200k), Gemini Flash (1M) — for standard Q&A. (3) "llm-local" → LLaMA 3.1 8B (128k, local). Route requests to the appropriate alias based on the estimated input token count (measure before the API call with tiktoken). If the primary provider for "llm-long-context" fails, only fall back to providers that also support the required context length — never fall back to a provider with insufficient context window for the request.

Rotate via secrets manager: (1) Generate a new key at the provider. (2) Add it to AWS Secrets Manager or HashiCorp Vault as a new version. (3) Update LiteLLM to read from Secrets Manager at runtime (LiteLLM supports os.environ/KEY_NAME for env-var injection, which is populated by secrets manager at pod startup). (4) Deploy a new LiteLLM pod (it picks up the new key from Secrets Manager on startup). (5) Drain old pod. The old key remains valid for 24 hours (during which both pods can use either key). (6) Revoke the old key at the provider after 24 hours. Automate this process: schedule monthly key rotation via a CI job that triggers the above steps, notifies the on-call engineer, and verifies the new key works before completing the rotation.

Tiered limits by user type: free tier (10k TPM), standard (50k TPM), pro (200k TPM), enterprise (1M TPM, negotiated). Implement soft limits (degrade quality: route to cheaper, faster model) before hard limits (return 429). When a user approaches their limit: (1) First 80% of limit: full service. (2) 80–95%: route to cheaper model (same capability for most queries). (3) 95–100%: add 500ms artificial delay (signals constraint without full refusal). (4) > 100%: return 429 with retry-after header. This degrades gracefully: most users never notice, power users see degraded service before a hard stop.

LiteLLM tracks each provider's rate limit usage automatically when using its router. Configure TPM and RPM limits per model in the router_settings: LiteLLM will stop routing to a provider when its limit is approached and shift to the next provider in the fallback chain. For fine-grained control: set rpm_policy and tpm_policy per model in the model_list config. Monitor the LiteLLM admin dashboard (/admin/model_info) for real-time rate limit usage per provider. Alert when any provider is at > 80% of its rate limit — time to provision a higher-tier key or add a second API key from the same provider (LiteLLM supports multiple keys per provider for limit stacking).

Four strategies: (1) Stream the response: streaming distributes token consumption across time, reducing burst TPM impact. (2) Break into sub-requests: split a long document processing task into chunks and queue them with delays between requests. (3) Route to a provider with higher limits: if the standard OpenAI key has a 30M TPM limit, an enterprise key may have 300M TPM. (4) Batch API: OpenAI and Anthropic offer async batch APIs with higher limits (and lower cost) for non-real-time requests. The batch API is ideal for offline document processing — no streaming, 24-hour completion window, but 50% lower cost and no rate limit constraints.

Tiered degraded responses by query type: (1) FAQ queries (60% of traffic): serve pre-computed answers cached during normal operation (refresh every 30 minutes). (2) Account/billing queries (15%): route to human support with context about the AI outage. (3) Complex queries (25%): acknowledge the AI is unavailable, provide a support link. Pre-compute the FAQ cache by running the top 100 production queries through the LLM daily and storing responses in Redis with 24-hour TTL — these become the degraded mode responses. Monitor cache staleness: if the primary provider has been down > 24 hours, the cached responses may be outdated — add a freshness warning to the user message.

The 100ms budget: (1) Outage detection happens before the request, not during it — the provider state is stored in Redis (1ms read). (2) Provider state is checked at request entry (< 1ms). (3) If primary is unhealthy, route directly to secondary — no retry of primary needed. (4) The LiteLLM router does all of this within 2–3ms overhead. The key insight: failover is fast because the routing decision is based on cached health state, not on waiting for a request to fail. Testing: use Toxiproxy or Chaos Mesh to inject 100% failure on the primary provider. Measure: (a) Time until the first failed request is detected (< 15s with 3-failure threshold at 5s timeout). (b) Time until subsequent requests route to secondary (< 1ms — health state cached in Redis). (c) Tail latency for requests during failover (< 20ms added overhead for re-routing).

Communication protocol: (1) Immediate (< 5 minutes): activate degraded mode in the UI — show a banner "Our AI assistant is temporarily unavailable. We're working to restore service." (2) Customer success notification (< 15 minutes): Slack #customer-success with outage scope (feature, user impact %, start time, fallback active). (3) Status page update (< 20 minutes): update status.yourdomain.com to reflect the outage — enterprise customers check this before escalating. (4) SLA breach notification (at 30 minutes): email enterprise customers with SLA implications and mitigation steps. (5) Resolution: update status page, send resolution email with incident timeline and RCA summary within 48 hours. Maintain customer trust by being transparent, fast, and accountable — a well-communicated outage damages trust less than a silent one.

Minimum viable: (1) PagerDuty integration with Prometheus alert for LLM error_rate > 5% (P1) and P99 latency > 5s (P2). (2) LiteLLM with 2 providers configured (OpenAI primary, Anthropic secondary) — failover is automatic. (3) A Slack #incidents channel where PagerDuty alerts post. (4) A one-page runbook: Detect (check Grafana) → Isolate (query structured logs: which provider/feature?) → Failover (update LiteLLM config or restart with secondary) → Communicate (Slack message). (5) Monthly rotation of on-call responsibility. This costs < 2 hours to set up and prevents the most common failure mode (provider outage = product outage).

LLM post-mortem template: (1) Incident summary: what failed, when, how long, user impact (N users affected, revenue impact if applicable). (2) Timeline: minute-by-minute from first alert to resolution — include detection lag, response time, and recovery time. (3) Root cause analysis (5-Whys): start from the user-visible symptom and ask why 5 times. (4) LLM-specific contributing factors: Was it a prompt change? Model upgrade? Data drift? Provider issue? Retrieval failure? (5) Detection gaps: why did it take N minutes to detect? What monitoring would have caught it sooner? (6) Action items: each item has an owner, a due date, and a clear definition of done. (7) Metrics: MTTR, MTTD, SLA impact. Publish within 48 hours — share with the broader team for learning.

Zero-downtime rotation procedure: (1) Generate a new API key at the provider. (2) Add the new key to AWS Secrets Manager as a new secret version (the old version remains active). (3) LiteLLM reads keys from env vars injected by the Secrets Manager agent — update the env var to point to the new version. (4) Rolling restart: K8s rolls out a new LiteLLM pod that picks up the new key. During the rollout, old pods use the old key and new pods use the new key — both work simultaneously. (5) After all pods are updated (5 minutes), revoke the old key at the provider. (6) Verify: test an API call with the new key from a monitoring pod. Automate this with a GitHub Actions workflow on a monthly schedule: generate key → store in Secrets Manager → trigger K8s rollout → revoke old key → verify.