Module 8 · Ship It
Ship It
Everything so far ran for you, on your machine, on data you generated. This module puts one real thing in front of real people: a problem someone actually has, built with the pieces from Modules 1 to 7, deployed, measured where it runs, and written up honestly, including what broke. The through-line is that every claim you make about the project should be one a tool made and a check can verify.
This is the most open-ended module, so the lesson gives you structure and checks rather than a fixed task. The code here (a small web service, its tests, a secrets scanner, a production-metrics report, README and demo-script checkers, and a stranger test) was run and tested; none of it was deployed to a real host, so your cold-start, memory, latency and cost numbers are yours to measure. Each lab says exactly what was and wasn't verified.
Concepts
Shipping is part of the skill
A project that only runs on your laptop proves you can follow a tutorial. A project a stranger can open, run, and check proves a harder set of things: that you can scope a problem, make it reproducible, keep secrets secret, measure what it does in the real world, and be honest about where it fails. Those are the skills the earlier modules built toward, and this module is where they become visible to anyone but you.
The module's requirements say it plainly: a real problem, your own loop before a framework, both a RAG app and an agent deployed, a README whose numbers are measured, a 60-second demo, and a write-up of what broke. Each of those has a mechanical check in this lesson, because "I think the README is accurate" is exactly the kind of claim this course has trained you to distrust.
A real problem, and what "real" means
A tutorial problem is one nobody has. A real problem has a named person who has it now and would notice if it went away. The test is to ask that person about the last time it happened (what they did, how long it took, what it cost them) rather than whether they'd like your idea. Rob Fitzpatrick's The Mom Test (2013) is built around exactly this: people will politely endorse any idea, but they can't fake a specific recent memory.
Write the pitch before any code, with six fields: the user, the problem, why an LLM helps, how it could be done without an LLM, a success number, and what you are not building. The no-LLM field is Module 7 applied: if you can't explain why a lookup table or a rule isn't enough, you may not need the model. The not-building field is what keeps a six-week project from becoming a six-month one.
README first: the spec you can't skip
Writing the README before the code forces the decisions that bite later: who this is for, what the architecture is, what success looks like, and how a stranger will run it. A README written last records whatever you happened to build; one written first is the spec, and the code is checked against it.
The most important rule is about the numbers. A results table typed by hand drifts from reality the first time you change a prompt. So this lesson's pattern is that tools write the numbers: the eval suite writes its scores to a file, the production log produces latency and cost, a script fills the README table with each value's source file and date, and a check in CI fails if the table is stale, edited by hand, or contains a metric nobody measured. The rule "every number is measured, not estimated" becomes something a build can enforce.
What "deployed" requires
Hosting a script is the easy part. A service other people can hit needs a handful of properties, each of which prevents a specific bad day. Configuration from the environment: keys and limits are read from environment variables, never from files in the repo (the twelve-factor app's rule on config, from Adam Wiggins's 2011 The Twelve-Factor App), and a missing key stops the process at boot so you find out at deploy time, not at 2 a.m. A health check and a readiness check: the host asks "is the process alive?" and your app can answer "yes, but still loading the index", so traffic arrives only when you can serve it. Rate limiting, because an open endpoint that calls a paid model is an invitation to a surprise bill. Input limits on size and shape. And clean failures: a model outage returns a short error, never a stack trace.
Lab 3 builds all of these into one small service in front of your RAG app and your agent, with tests. Two honest caveats ride along. An in-memory rate limiter is per process, so a service with several instances needs a shared store. And the agent endpoint here runs the whole job before it answers, which is fine for a demo and wrong for real traffic, where you start a background job and let the client poll.
Measure it where it runs
Numbers measured on your laptop describe your laptop. In production you need three: latency (report the median and the 95th percentile, because users feel the slow tail), cost per request (from the token counts the API already returns, and with cache hits counted separately, since a hit costs nothing), and the error rate. The service in Lab 3 writes one structured log line per request, with the token counts and a hash of the question instead of the question itself, and a small script turns those lines into the numbers.
Two traps. Cold starts: some hosting tiers stop idle services, so the first request after a pause waits for the process to start and your index to load. Measure cold and warm separately and say which your README quotes. And memory: the reranker from Module 3 is a neural model, and hosts limit memory by plan. Check what your service uses idle and under load before you pick a plan; if it won't fit, replace the reranker or call a hosted one, and re-run your evals to see what that costs in quality.
Before it is public
Four checks, all cheap now and expensive later. Secrets: scan every tracked file for key-shaped strings before every push. A key that reaches a public repo should be treated as leaked, even if you delete the commit, because history and mirrors keep it; rotate it. Logs: don't log user text you don't need, and redact what you do (Module 4's redaction applies here). Exposure: the deployed agent should have only read-only tools, since a public endpoint can't pop up an approval prompt the way your terminal did in Module 5; anything that acts needs an explicit approval flow or must be left out. Licenses: check every dependency before you publish. Module 2 flagged that pymupdf4llm is licensed AGPL-3.0 (or commercially from Artifex), and AGPL has a network clause, so putting it behind a public service generally means publishing your source under AGPL or buying a commercial license. That's not legal advice, only a reason to look before you launch.
Product judgment: autonomy, trust and recoverability
Everything above is a skill you can get right by following a checklist. This is judgment, and it's the harder half: how much should the system be allowed to do on its own before a person has to say yes? Match the answer to the risk and reversibility of the specific action, not to how confident the model's text sounds, because confident wording and a correct action are two different things, measured two different ways.
Three levels cover most of what you'll build. Suggest: the system proposes, a person decides. Act with confirmation: it executes once a person approves that specific action. Act and report: it executes on its own and tells you afterward. Module 4's decision models give you a principled way to pick between these per action, since a calibrated probability is a reasonable input to "ask or just act"; an uncalibrated one is a confident-sounding guess wearing the same clothes.
Three more habits round it out. Trust calibration: show your work and your sources, say plainly when you're unsure, and watch for both over-trust (a user who stops checking) and under-trust (a user who re-verifies everything and gets none of the speed back). Recoverability: confirm before anything irreversible, offer an undo where one is possible, and keep an activity log written in plain language, not log-level jargon, since the person reading it afterward is not you. Failure UX: when the system can't finish, hand off to a human or a manual path instead of a dead end.
The blameless postmortem
Things will break in the first week. The professional response is a written, blameless postmortem, the practice described in the Postmortem Culture chapter of Google's Site Reliability Engineering (O'Reilly, 2016): describe what happened and how the system allowed it, never who did it. Blame teaches people to hide failures; a blameless review teaches the system to catch them.
The structure is short: a timeline, a root cause, what you changed, how you know it's fixed, and follow-ups. The root cause uses the same seven classes as Module 5's harness table (missing context, wrong tool, bad output, repeated loop, unsafe action, lost decision, unknown), and the fix is held to the same standard: a change to the system, plus a case added to the golden set so the failure cannot quietly return. "We rewrote the prompt" isn't a fix; it is a patch.
Telling the story
A 60-second demo has to earn its seconds: the problem in 8, the real app doing a real task in 30, how it's built in 12, the numbers in 8, and where to try it in 2. Write the script first and time it, because people read aloud at about 150 words a minute and a draft that feels short almost never is. Show the app working on the problem, not slides about it.
For interviews, the same material compresses into four sentences: the problem and who had it, one decision you defended with numbers, one failure you diagnosed and fixed, and what you would do next. If you can say those four things with real numbers, the project has done its job. Each of them comes from work you already did in Modules 3 to 7.
Hands-on Labs
- Find a person before you pick a project. Someone you can actually talk to has a recurring, annoying, text-heavy task: a sibling who reads contracts, a colleague who answers the same questions, a club that fields the same emails. Ask them about the last time it happened: what they did, how long it took, what went wrong, what they've already tried. Don't describe your idea yet.
- Write
pitch.mdwith these six fields. Be specific enough that the checker below can tell.**User:** Dana, a support lead on a team of six who answers warranty questions from PDF manuals all day. **Problem:** Finding one number takes about four minutes, and twice last month she quoted an outdated manual. **Why an LLM:** Questions arrive in free-form wording ("how long am I covered?") that keyword search misses. **Without an LLM:** A lookup table keyed by product would answer the ten most common questions. **Success looks like:** At least 90% of 50 real questions answered correctly with a citation, in under 3 seconds. **Not building:** Logins, editing manuals, or answering anything outside the manuals. - Run the checker. It fails a pitch with a missing or thin field, a success line with no number in it, or a user described as "everyone".
"""A problem pitch has to name a person, a problem, a reason for an LLM, a non-LLM alternative, a measurable success number, and what you will NOT build. Usage: python checks/check_pitch.py pitch.md""" import re, sys from pathlib import Path FIELDS = ["User", "Problem", "Why an LLM", "Without an LLM", "Success looks like", "Not building"] text = Path(sys.argv[1]).read_text() problems = [] for f in FIELDS: m = re.search(rf"^\*\*{re.escape(f)}:?\*\*:?\s*(.+?)(?=^\*\*|\Z)", text, re.S | re.M) if not m or len(m.group(1).strip()) < 15: problems.append(f"missing or too short: **{f}**") continue body = m.group(1) if f == "Success looks like" and not re.search(r"\d", body): problems.append("**Success looks like** needs a number (a score, a time, a count), not an adjective") if f == "User" and re.search(r"\b(everyone|anyone|users|people)\b", body, re.I) and len(body.split()) < 12: problems.append("**User** is too vague: name a specific person or role you can actually ask") print("\n".join(problems) if problems else "pitch looks specific enough to get feedback on") sys.exit(1 if problems else 0)python checks/check_pitch.py pitch.md - Get feedback before building. Send the pitch to the person from step 1 and one skeptic, and ask two questions only: "Would this have helped last time?" and "What would make you not trust the answers?" Write down what changes in the pitch.
- Take the no-LLM field seriously. If a table, a rule or a search box would do 80% of it, build that first and measure it; the LLM has to earn the rest.
- Write the README before any new code, using this template. Fill in every section except the numbers, and leave the two metrics markers exactly as they are: a script will write between them.
# Project name One sentence a stranger understands. ## What it does Two or three sentences. What goes in, what comes out. ## Who it is for The specific person from your pitch, and the problem you solved for them. ## Architecture A diagram or a short list: where a request goes, and which model does what. Name the decision you are proudest of. ## Quick start Clone the repo, create and activate a virtual environment, then: ```bash pip install -r requirements.txt cp .env.example .env # then add your keys python rag/generate_corpus.py 12 python rag/pipeline.py ``` ## Results How the numbers below were measured, in one sentence. <!-- metrics:start --> <!-- metrics:end --> ## Limits and known failures - A real failure you found and have not fixed. - Something it is not built to do. ## Live demo https://your-app.example.com - Create
checks/make_metrics.py. It reads the files your tools produce (the CI eval'sevals/scores.jsonfrom Module 4 and the production metrics from Lab 3), and writes a table between the markers with each value, the file it came from, and the file's date. A metric whose file doesn't exist yet shows asNOT MEASURED YET."""Write the README's results table from the files your tools produced. Nobody types these numbers. Usage: python checks/make_metrics.py [README.md] (rewrites the block between the metrics markers)""" import datetime, json, sys from pathlib import Path START, END = "<!-- metrics:start -->", "<!-- metrics:end -->" def get(path, *keys): p = Path(path) if not p.exists(): return None, None v = json.loads(p.read_text()) for k in keys: v = v.get(k) if isinstance(v, dict) else None return v, datetime.date.fromtimestamp(p.stat().st_mtime).isoformat() ROWS = [ # label, file, keys, format ("Answerable questions correct", "evals/scores.json", ["answerable_pass"], "{:.0%}"), ("Unanswerable questions declined", "evals/scores.json", ["unanswerable_pass"], "{:.0%}"), ("Faithfulness (judged slice)", "evals/scores.json", ["faithfulness"], "{:.2f}"), ("Median latency, /ask", "service/prod_metrics.json", ["ask", "p50_ms"], "{:,.0f} ms"), ("95th percentile latency, /ask", "service/prod_metrics.json", ["ask", "p95_ms"], "{:,.0f} ms"), ("Cost per uncached request, /ask", "service/prod_metrics.json", ["ask", "cost_per_miss_usd"], "${:.4f}"), ("Cache hit rate, /ask", "service/prod_metrics.json", ["ask", "cache_hit_rate"], "{:.0%}"), ("Server errors (5xx)", "service/prod_metrics.json", ["error_rate_5xx"], "{:.1%}"), ] def table(): lines = ["| Metric | Value | Measured by | On |", "|---|---|---|---|"] for label, path, keys, fmt in ROWS: v, when = get(path, *keys) lines.append(f"| {label} | {fmt.format(v) if v is not None else 'NOT MEASURED YET'} | `{path}` | {when or '-'} |") return "\n".join(lines) def block(): return f"{START}\n{table()}\n{END}" if __name__ == "__main__": path = Path(sys.argv[1] if len(sys.argv) > 1 else "README.md") text = path.read_text() if START not in text or END not in text: sys.exit(f"{path} has no {START} ... {END} markers") head, rest = text.split(START, 1) path.write_text(head + block() + rest.split(END, 1)[1]) print(table()) - Create
checks/check_readme.py. It fails if a section is missing, the metrics table differs from what the script would write now (stale, or edited by hand), any metric is unmeasured, the Results prose uses an estimate word ("about", "roughly", "~", "TBD"), the Quick start is missing or longer than about ten commands, the live demo has no link, or the limits section is thin."""Fail the build when the README is incomplete, estimated, or out of date. Usage: python checks/check_readme.py [README.md]""" import re, sys from pathlib import Path sys.path.insert(0, "checks") from make_metrics import block, START, END REQUIRED = ["What it does", "Who it is for", "Architecture", "Quick start", "Results", "Limits and known failures", "Live demo"] ESTIMATE_WORDS = re.compile(r"\b(tbd|todo|approximately|roughly|about|around|estimated)\b|~\s*\d", re.I) def sections(text): out, cur = {}, None for line in text.splitlines(): m = re.match(r"##\s+(.*)", line) if m: cur = m.group(1).strip(); out[cur] = [] elif cur: out[cur].append(line) return {k: "\n".join(v) for k, v in out.items()} def check(path): text = Path(path).read_text() s, problems = sections(text), [] for name in REQUIRED: if not s.get(name, "").strip(): problems.append(f"missing or empty section: ## {name}") if START not in text or END not in text: problems.append("Results has no metrics markers") else: current = text[text.index(START): text.index(END) + len(END)] if current != block(): problems.append("the metrics table is out of date or was edited by hand: run python checks/make_metrics.py") if "NOT MEASURED YET" in current: problems.append("some metrics are NOT MEASURED YET: measure them before you ship") prose = text.replace(current, "") results_prose = sections(prose).get("Results", "") if ESTIMATE_WORDS.search(results_prose): problems.append("Results contains an estimate word (about, roughly, ~, TBD): replace it with a measured number or delete the claim") qs = s.get("Quick start", "") cmds = [l for blk in re.findall(r"```(?:bash|sh|shell)?\n(.*?)```", qs, re.S) for l in blk.splitlines() if l.strip() and not l.strip().startswith("#")] if not cmds: problems.append("Quick start has no fenced code block of commands") elif len(cmds) > 10: problems.append(f"Quick start is {len(cmds)} commands long: a stranger should need about ten at most") if not re.search(r"https://\S+", s.get("Live demo", "")): problems.append("Live demo needs an https:// link") if len(s.get("Limits and known failures", "").strip()) < 40: problems.append("Limits and known failures is too thin: list at least one real failure you found") return problems if __name__ == "__main__": problems = check(sys.argv[1] if len(sys.argv) > 1 else "README.md") print("\n".join(problems) if problems else "README looks complete and every number is measured") sys.exit(1 if problems else 0)python checks/make_metrics.py python checks/check_readme.py - Break it on purpose. Edit one number in the table by hand and re-run the check. Add the word "roughly" to the Results text. Delete a metrics file. Each should fail with a message that tells you what to do.
- Make it automatic. Add these two lines as steps in Module 4's workflow, after the eval, so the README is checked on every pull request:
- run: python checks/check_readme.py - run: python checks/check_secrets.py
NOT MEASURED YET until all eight metrics exist (the cost row stays unmeasured until you fill in prices in Lab 3, which is the point). With every metric present it passes. Adding "roughly" fails, editing a number by hand fails, and removing a source file fails. Two limits worth knowing: the check can't tell whether the numbers are good, only that they're measured and current; and it only governs the table, so prose claims elsewhere ("fast", "accurate") are still on you.- Add the dependencies and create the service. It wraps whatever
ask()your RAG project exposes (here, Module 3's) and your agent'sresearch()(Module 5's). It reads its keys and limits from the environment and refuses to start without them; answers/healthimmediately and/readyonly once the index has loaded; rate limits each client; validates input size; turns model failures into a clean 502; and writes one JSON log line per request with token counts and a hash of the question, never the question itself.pip install fastapi uvicorn httpx pytest"""One small web service in front of two of your projects: /ask (the RAG app) and /research (the agent). Run locally: uvicorn service.app:app --reload On a host: uvicorn service.app:app --host 0.0.0.0 --port $PORT""" import hashlib, json, logging, os, sys, threading, time, uuid from contextlib import asynccontextmanager from collections import defaultdict, deque from fastapi import FastAPI, HTTPException, Request from fastapi.responses import JSONResponse from pydantic import BaseModel, Field # 1. Configuration comes from the environment, and a missing key stops the process at boot, not at 2 a.m. REQUIRED = ["OPENAI_API_KEY", "ANTHROPIC_API_KEY"] missing = [k for k in REQUIRED if not os.environ.get(k)] if missing: raise RuntimeError(f"missing environment variables: {', '.join(missing)}") RATE_LIMIT_PER_MIN = int(os.environ.get("RATE_LIMIT_PER_MIN", "10")) TRUST_PROXY = os.environ.get("TRUST_PROXY") == "1" # only true behind a proxy you control MAX_QUESTION_CHARS = 500 logging.basicConfig(stream=sys.stdout, level=logging.INFO, format="%(message)s") log = logging.getLogger("service") def load_projects(): """Build the RAG index and import the agent in the background, so the process answers /health at once.""" sys.path.insert(0, "rag"); sys.path.insert(0, "agent") import pipeline, research app.state.ask = pipeline.ask app.state.research = research.research app.state.ready = True @asynccontextmanager async def lifespan(app): if os.environ.get("SKIP_LOAD") != "1": # tests set this and inject stand-ins threading.Thread(target=load_projects, daemon=True).start() yield app = FastAPI(title="AI Beyond Fundamentals service", lifespan=lifespan) app.state.ready = False app.state.ask = None app.state.research = None # 2. A sliding-window rate limiter per client. In memory, so it is per process: with several instances use a shared store. _hits = defaultdict(deque) _lock = threading.Lock() def client_id(request: Request): forwarded = request.headers.get("x-forwarded-for", "") return forwarded.split(",")[0].strip() if TRUST_PROXY and forwarded else request.client.host def allowed(cid, now=None): now = now or time.time() with _lock: q = _hits[cid] while q and now - q[0] > 60: q.popleft() if len(q) >= RATE_LIMIT_PER_MIN: return False q.append(now) return True # 3. One structured log line per request: enough to compute latency, cost and errors, with no question text in it. @app.middleware("http") async def observe(request: Request, call_next): started = time.perf_counter() request.state.usage = {} if request.url.path not in ("/health", "/ready") and not allowed(client_id(request)): response = JSONResponse({"error": "rate limit exceeded, try again in a minute"}, status_code=429, headers={"Retry-After": "60"}) else: response = await call_next(request) ms = (time.perf_counter() - started) * 1000 log.info(json.dumps({"ts": round(time.time(), 2), "path": request.url.path, "status": response.status_code, "ms": round(ms, 1), **request.state.usage})) return response class Question(BaseModel): question: str = Field(min_length=3, max_length=MAX_QUESTION_CHARS) @app.get("/health") def health(): """Liveness: the process is up. Hosts poll this to decide whether to restart you.""" return {"status": "ok"} @app.get("/ready") def ready(): """Readiness: the index is loaded and the app can answer. Return 503 until it can.""" if not app.state.ready: raise HTTPException(503, "warming up") return {"status": "ready"} @app.post("/ask") def ask(body: Question, request: Request): if not app.state.ready: raise HTTPException(503, "warming up") try: r = app.state.ask(body.question) except Exception: log.exception("ask failed") raise HTTPException(502, "the model call failed; try again") # never leak a stack trace to a client request.state.usage = {"kind": "ask", "q_len": len(body.question), "q_hash": hashlib.sha256(body.question.encode()).hexdigest()[:10], "cached": r["cached"], "input_tokens": r["input_tokens"], "output_tokens": r["output_tokens"]} return {"answer": r["answer"], "sources": r["sources"], "cached": r["cached"], "latency_ms": round(r["latency_ms"])} @app.post("/research") def research_endpoint(body: Question, request: Request): """Runs the whole agent before answering, which can take a long while. Fine for a demo; for real traffic start a background job and let the client poll for the result.""" if not app.state.ready: raise HTTPException(503, "warming up") run_id = uuid.uuid4().hex[:8] try: report, receipt = app.state.research(run_id, body.question) except Exception: log.exception("research failed") raise HTTPException(502, "the agent run failed; try again") request.state.usage = {"kind": "research", "q_len": len(body.question), "input_tokens": receipt.get("tokens", 0), "output_tokens": 0} return {"run_id": run_id, "report": report, "citations_ok": receipt["checks"]["passed"]} - Test it before you host it. These tests inject stand-ins for the model, so they cost nothing: health is always up, readiness is a 503 until loaded, input limits reject bad requests, the rate limiter returns 429 after the cap but never blocks health checks, a model failure is a clean 502 with no stack trace, and a missing key stops the process at boot.
import os os.environ.update(OPENAI_API_KEY="x", ANTHROPIC_API_KEY="x", SKIP_LOAD="1", RATE_LIMIT_PER_MIN="5") import pytest from fastapi.testclient import TestClient from service import app as appmod def fake_ask(q): return {"answer": f"stub answer to {q} [a.pdf p.1]", "sources": [{"doc": "a.pdf", "page": 1}], "cached": False, "latency_ms": 12.0, "input_tokens": 300, "output_tokens": 20} @pytest.fixture def client(): appmod._hits.clear() appmod.app.state.ready = True appmod.app.state.ask = fake_ask appmod.app.state.research = lambda run_id, q: ("report", {"checks": {"passed": True}, "tokens": 900}) return TestClient(appmod.app) def test_health_is_always_ok(client): appmod.app.state.ready = False assert client.get("/health").json() == {"status": "ok"} def test_ready_is_503_until_loaded(client): appmod.app.state.ready = False assert client.get("/ready").status_code == 503 appmod.app.state.ready = True assert client.get("/ready").status_code == 200 def test_ask_returns_answer_and_sources(client): r = client.post("/ask", json={"question": "warranty on the Delta Sensor?"}) assert r.status_code == 200 and r.json()["sources"][0]["doc"] == "a.pdf" def test_rejects_empty_and_oversized_questions(client): assert client.post("/ask", json={"question": "hi"}).status_code == 422 assert client.post("/ask", json={"question": "x" * 501}).status_code == 422 def test_rate_limit_returns_429_after_the_cap(client): codes = [client.post("/ask", json={"question": "what is the warranty?"}).status_code for _ in range(7)] assert codes[:5] == [200] * 5 and codes[5:] == [429, 429] def test_health_is_not_rate_limited(client): assert all(client.get("/health").status_code == 200 for _ in range(20)) def test_model_failure_is_a_clean_502_without_a_stack_trace(client): appmod.app.state.ask = lambda q: (_ for _ in ()).throw(RuntimeError("secret internal detail")) r = client.post("/ask", json={"question": "what is the warranty?"}) assert r.status_code == 502 and "secret" not in r.text def test_research_endpoint(client): r = client.post("/research", json={"question": "compare two products"}) assert r.status_code == 200 and r.json()["citations_ok"] is True def test_missing_keys_stop_the_process_at_boot(): import importlib, subprocess, sys out = subprocess.run([sys.executable, "-c", "import service.app"], capture_output=True, text=True, env={"PATH": os.environ["PATH"], "PYTHONPATH": "."}) assert "missing environment variables" in out.stderrpython -m pytest service/test_app.py -q - Fill in your model's prices, so cost per request can be computed from the logged token counts.
"""Dollars per 1,000,000 tokens (input, output) for the model your app calls. Fill in from the provider's pricing page.""" PRICES = (0.0, 0.0) def cost_usd(input_tokens, output_tokens): return (input_tokens * PRICES[0] + output_tokens * PRICES[1]) / 1_000_000 - Scan for secrets before the first push, and put the scan in CI (Lab 2). It checks tracked files for key-shaped strings and for files that must never be committed, such as
.env."""Refuse to ship a secret. Scans tracked files (or every file, outside a git repo) for key-shaped strings and for files that should never be committed. Run before every push; add it to CI.""" import re, subprocess, sys from pathlib import Path PATTERNS = { "OpenAI-style key": re.compile(r"sk-(?:proj-)?[A-Za-z0-9_-]{20,}"), "Anthropic-style key": re.compile(r"sk-ant-[A-Za-z0-9_-]{20,}"), "Google API key": re.compile(r"AIza[0-9A-Za-z_-]{30,}"), "GitHub token": re.compile(r"gh[pousr]_[A-Za-z0-9]{30,}"), "AWS access key id": re.compile(r"AKIA[0-9A-Z]{16}"), "private key block": re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH )?PRIVATE KEY-----"), "key assigned in code": re.compile(r"""(?i)(?:api[_-]?key|secret|token|password)\s*[:=]\s*["'][A-Za-z0-9_\-]{16,}["']"""), } FORBIDDEN_NAMES = {".env", "id_rsa", "id_ed25519"} def tracked_files(root="."): try: out = subprocess.run(["git", "ls-files"], capture_output=True, text=True, cwd=root, check=True).stdout.split("\n") return [Path(root) / f for f in out if f] except Exception: return [p for p in Path(root).rglob("*") if p.is_file() and ".git" not in p.parts and ".venv" not in p.parts] def scan(root="."): problems = [] for path in tracked_files(root): if path.name in FORBIDDEN_NAMES: problems.append(f"{path}: this file should never be committed") continue try: text = path.read_text(errors="ignore") except Exception: continue for label, pat in PATTERNS.items(): for m in pat.finditer(text): line = text.count("\n", 0, m.start()) + 1 problems.append(f"{path}:{line}: looks like a {label}: {m.group()[:8]}...") return problems if __name__ == "__main__": problems = scan(sys.argv[1] if len(sys.argv) > 1 else ".") print("\n".join(problems) if problems else "no secrets found") if problems: print("\nIf this is real: rotate the key now (a pushed key is a leaked key, even if you delete the commit), " "then remove it from the file and read it from the environment.") sys.exit(1 if problems else 0)python checks/check_secrets.py - Deploy. Render, Fly.io, Railway or another host will build the repo from
requirements.txtand run one start command. Set these in the host's environment variables screen, not in a file:OPENAI_API_KEY,ANTHROPIC_API_KEY, and optionallyRATE_LIMIT_PER_MIN. If the host puts a proxy in front of your service, also setTRUST_PROXY=1so the limiter sees real client addresses, but only then, since the header can be forged otherwise. Point the host's health check at/health. The build must also create the corpus your RAG app indexes. Exact setting names change, so follow your host's current documentation.# build command pip install -r requirements.txt && python rag/generate_corpus.py 12 # start command uvicorn service.app:app --host 0.0.0.0 --port $PORT - Smoke-test the live URL. It waits for readiness (that wait is your cold start, including loading the index), times warm requests, then fires a burst to prove the rate limiter works.
"""Smoke test a running service. Usage: python service/smoke.py https://your-app.example.com Measures cold-start time, warm latency, and proves the rate limiter works.""" import statistics, sys, time import httpx base = sys.argv[1].rstrip("/") QUESTION = "How long is the warranty on the Larkfield Delta Sensor?" c = httpx.Client(timeout=60) print("health :", c.get(f"{base}/health").json()) t0 = time.time() while True: # the first request on a sleeping free-tier host can take a minute r = c.get(f"{base}/ready") if r.status_code == 200: break if time.time() - t0 > 180: sys.exit("never became ready within 3 minutes") time.sleep(2) print(f"ready : after {time.time() - t0:.1f}s (cold start, including loading the index)") lat = [] for _ in range(4): # stay under the rate limit while measuring warm latency t = time.perf_counter() r = c.post(f"{base}/ask", json={"question": QUESTION}) lat.append((time.perf_counter() - t) * 1000) assert r.status_code == 200, r.text print(f"warm : median {statistics.median(lat):.0f} ms over {len(lat)} requests, answer: {r.json()['answer'][:70]!r}") codes = [c.post(f"{base}/ask", json={"question": QUESTION}).status_code for _ in range(25)] print("burst :", "rate limiter returned 429 after", codes.index(429) + 1, "more requests" if 429 in codes else "NEVER FIRED") print("bad body:", c.post(f"{base}/ask", json={"question": "x"}).status_code, "(expect 422, or 429 if you are still rate limited)") sys.exit(0 if 429 in codes else 1)python service/smoke.py https://your-app.example.com - Generate honest traffic: send your golden-set questions through the live URL, staying under the rate limit. Then export the host's logs to a file and turn them into the numbers for your README.
"""Turn the service's request log into the numbers a README needs: latency percentiles, cost, errors. Usage: python service/prod_report.py requests.log (pipe your host's log export into a file first)""" import json, statistics, sys from pathlib import Path sys.path.insert(0, "service") from costs import PRICES, cost_usd rows = [] for line in open(sys.argv[1]): try: e = json.loads(line) except ValueError: continue # hosts mix their own lines into the log; skip them if "path" in e and e["path"] not in ("/health", "/ready"): rows.append(e) def pct(xs, p): xs = sorted(xs) return xs[min(len(xs) - 1, int(round(p / 100 * (len(xs) - 1))))] summary = {} for kind in ("ask", "research"): rs = [r for r in rows if r.get("kind") == kind and r["status"] == 200] if not rs: continue ms = [r["ms"] for r in rs] hits = sum(bool(r.get("cached")) for r in rs) miss = [r for r in rs if not r.get("cached")] cost = [cost_usd(r["input_tokens"], r["output_tokens"]) for r in miss] print(f"/{kind}: {len(rs)} ok requests, p50 {pct(ms, 50):.0f} ms, p95 {pct(ms, 95):.0f} ms, " f"cache hits {hits}/{len(rs)}") summary[kind] = {"requests": len(rs), "p50_ms": round(pct(ms, 50)), "p95_ms": round(pct(ms, 95)), "cache_hit_rate": round(hits / len(rs), 3)} if PRICES == (0.0, 0.0): print(" cost: fill in service/costs.py") else: summary[kind]["cost_per_miss_usd"] = round(statistics.mean(cost), 6) print(f" mean cost per cache-miss request: ${statistics.mean(cost):.5f}") errors = [r for r in rows if r["status"] >= 500] limited = [r for r in rows if r["status"] == 429] summary["error_rate_5xx"] = round(len(errors) / len(rows), 4) Path("service/prod_metrics.json").write_text(json.dumps(summary, indent=1)) print(f"errors (5xx): {len(errors)}/{len(rows)} rate limited (429): {len(limited)}/{len(rows)}")python service/prod_report.py requests.log python checks/make_metrics.py
uvicorn service.app:app) and confirm /ready turns 200 before you deploy anything. We also could not deploy to a real host, so your cold-start time, your memory use and your real latency are yours to measure, and you should expect cold start to be much longer than a warm request on hosts that stop idle services. If the build fails or the instance runs out of memory, the reranker model is the first suspect.- Write the demo script first, in the 60-second shape: problem 8 seconds, demo 30, how it's built 12, numbers 8, where to try it 2. Anything in square brackets is a screen cue and isn't spoken.
# Demo script ## Problem (8s) Support staff dig through forty PDF manuals to find one warranty number. [on screen: a messy folder of PDFs] ## Demo (30s) I ask for the Delta Sensor warranty. A few seconds later I get the answer, with a link that opens the exact page. Now a question the manuals cannot answer, and it says so instead of guessing. [on screen: the real app, one take] ## How it is built (12s) Hybrid search, a reranker, and one model call, with citations checked in code before any answer goes out. [on screen: the architecture diagram] ## Numbers (8s) Ninety-two percent right on fifty questions, about two seconds typical, under a cent a question. [on screen: the results table] ## Try it (2s) The link is below. - Time it. The script counts spoken words per beat against a speaking rate of 150 words a minute and tells you how many words to cut. A script that fits on paper usually doesn't fit on the clock.
"""Time a spoken demo script before you record it. People read aloud at about 150 words a minute (2.5 a second). Each '## Beat (Ns)' heading gives that beat's time budget. Usage: python checks/script_timer.py demo_script.md""" import re, sys from pathlib import Path WORDS_PER_SECOND = 2.5 beats, total_budget, total_est = [], 0, 0.0 current = None for line in Path(sys.argv[1]).read_text().splitlines(): m = re.match(r"##\s+(.*?)\s*\((\d+)\s*s\)\s*$", line) if m: current = {"name": m.group(1), "budget": int(m.group(2)), "words": 0} beats.append(current) elif current and line.strip() and not line.startswith("#"): current["words"] += len(re.findall(r"\b[\w'-]+\b", re.sub(r"\[.*?\]", "", line))) # [on screen: ...] cues aren't spoken if not beats: sys.exit("no '## Name (Ns)' headings found") ok = True for b in beats: est = b["words"] / WORDS_PER_SECOND flag = "" if est <= b["budget"] else " TOO LONG: cut " + str(int(b["words"] - b["budget"] * WORDS_PER_SECOND + 0.99)) + " words" ok &= est <= b["budget"] total_budget += b["budget"]; total_est += est print(f"{b['name']:16s} {est:5.1f}s of {b['budget']:2d}s ({b['words']} words){flag}") print(f"{'total':16s} {total_est:5.1f}s of {total_budget}s") sys.exit(0 if ok and total_est <= 60 else 1)python checks/script_timer.py demo_script.md - Record the real app, one take per beat, doing a real task: don't narrate slides. Use any screen recorder you like. Show one question it answers well and one it correctly declines.
- Pick three things that broke while you built or deployed it, and write the postmortem for the most expensive one. Use the template below, keep it blameless (describe the system), and finish with a change to the system and a case added to the golden set.
# Postmortem: what broke, in plain words Date: YYYY-MM-DD Duration: how long it was broken Impact: who noticed, and what it cost ## Timeline - 14:02 The first sign, and where you saw it (a log line, a user message) - 14:20 What you tried, and what you learned - 15:05 What fixed it ## Root cause Which class of failure? (missing context, wrong tool, bad output, repeated loop, unsafe action, lost decision, unknown) Then one paragraph on the specific cause. Describe the system, not a person. ## What we changed in the system - A guide, a test, a tool description, a validator, a permission or a trace. Not "we rewrote the prompt". ## How we know it is fixed - The failing case, now in the golden set (id and question), passing. ## What went well, and what was luck - One of each. ## Follow-ups - Each with an owner (a role) and a date. - Write your four-sentence interview summary from the same material.
**The problem:** one sentence, naming the person who had it. **A decision I defended with numbers:** "I chose X over Y because on 50 questions X scored A against B, at C a request." **A failure I diagnosed and fixed:** "It broke when ..., because ..., so I changed ..., and this test would catch it next time." **What I would do next:** one thing, and why it is next.
- Create
service/activity.py. The three levels match this module's Concepts section exactly:suggestproposes and stops,confirmwould wait for a second, explicit approval before acting (simulated here with a flag, since this service has no UI to click "approve" in), andactexecutes immediately and logs it afterward. The log is written as plain sentences, not JSON keys, because the person reading it back is the point, not a grep target."""An autonomy gate and a human-readable activity log for the agent endpoint. The three levels match this module's Concepts section: suggest, act with confirmation, act and report.""" import json, time from pathlib import Path ACTIVITY_LOG = Path("service/activity.jsonl") def log_activity(user, level, action, detail, executed): with open(ACTIVITY_LOG, "a") as f: f.write(json.dumps({"ts": time.time(), "user": user, "autonomy": level, "action": action, "detail": detail, "executed": executed}) + "\n") def gate(level, action, detail, do_action, confirmed=False, user="demo"): """suggest: proposes only, never executes. confirm: executes only if the caller already passed confirmed=True (a real UI would ask first, then resend the request). act: executes immediately and the log is how you find out about it afterward.""" if level == "suggest" or (level == "confirm" and not confirmed): log_activity(user, level, action, detail, executed=False) return {"proposed": detail, "executed": False, "note": "awaiting your confirmation"} result = do_action() log_activity(user, level, action, detail, executed=True) return {"result": result, "executed": True} def read_activity(n=20): """Plain-language lines, not raw JSON, for a non-engineer reading what the system did.""" if not ACTIVITY_LOG.exists(): return ["(nothing logged yet)"] lines = [json.loads(l) for l in ACTIVITY_LOG.read_text().splitlines()[-n:]] out = [] for e in lines: when = time.strftime("%H:%M:%S", time.localtime(e["ts"])) did = "did" if e["executed"] else "proposed, but did not do" out.append(f"[{when}] at autonomy level '{e['autonomy']}', the system {did}: {e['detail']}") return out - Wire it into
/researchfrom Lab 3. Reuse Module 5 Lab 12'sfile_findingas the side-effecting action worth gating: once the agent has an answer worth keeping, route filing it throughgate()instead of calling it directly, with the autonomy level read from a query parameter.from activity import gate, read_activity import sys; sys.path.insert(0, "agent") from idempotent_tools import file_finding @app.get("/research") def research_endpoint(q: str, autonomy: str = "confirm", confirmed: bool = False): answer = research(q) # Module 5's agent, already wrapped by this service outcome = gate(autonomy, "file_finding", answer[:200], lambda: file_finding(key=str(hash(q)), finding=answer), confirmed=confirmed) return {"answer": answer, **outcome} @app.get("/activity") def activity_endpoint(n: int = 20): return {"activity": read_activity(n)} - Call
/researchthree times with the same question, once per level (?autonomy=suggest,?autonomy=confirm,?autonomy=confirm&confirmed=true,?autonomy=act), then read/activityand confirm the plain-language log matches what each call actually did. - Run a short usability test with two or three people. Show them only the
/activityoutput, not the code, and ask two questions per entry: "did you expect this to have already happened?" and "would you have wanted to approve it first?" Write down every entry where their answer disagreed with the level you picked.
suggest and unconfirmed confirm calls return "executed": false and leave no new line in agent/report.jsonl; the confirmed call and the act call both execute and file a finding. We verified the gate's branching logic directly; your usability answers are the actual result of this lab, not the code. A disagreement where someone expected act-level autonomy to have asked first is under-trust in the interface (make the wording louder); a disagreement where someone assumed a suggest-level proposal had already happened is the more dangerous one, over-trust, and it means "executed": false needs to be said in the log's own sentence, not left to a field they never read.Project
Ship every project from this course, not only the best one. Each gets a README a stranger can follow, numbers its tools measured, and a check that keeps it honest.
- Give every project folder (and the repo root) its own README from Lab 2's template, with a Quick start of about ten commands at most. Where a project has no eval or live numbers, delete those rows from its table instead of leaving them unmeasured; a missing claim is better than an invented one.
- Run the stranger test on each project. This script clones the repo into an empty folder, creates a fresh virtual environment, runs your Quick start commands exactly as written with none of your API keys or settings available, and times the whole thing. It fails if any command fails or if the total passes ten minutes.
"""The stranger test: clone the repo into an empty folder, make a new virtual environment, run the README's Quick start commands exactly as written, and time it. Usage: python checks/fresh_clone_test.py <repo path or URL>""" import os, re, subprocess, sys, tempfile, time from pathlib import Path LIMIT_SECONDS = 600 repo = sys.argv[1] work = Path(tempfile.mkdtemp(prefix="fresh_clone_")) t_all = time.time() subprocess.run(["git", "clone", "--quiet", repo, str(work / "repo")], check=True) checkout = work / "repo" subprocess.run([sys.executable, "-m", "venv", str(work / "venv")], check=True) bindir = work / "venv" / ("Scripts" if os.name == "nt" else "bin") env = {**os.environ, "PATH": f"{bindir}{os.pathsep}{os.environ['PATH']}", "VIRTUAL_ENV": str(work / "venv")} env.pop("PYTHONPATH", None) # nothing from your own machine may leak in for k in [k for k in env if k.endswith("_API_KEY")]: env.pop(k) # and none of YOUR keys: a stranger has none readme = (checkout / "README.md").read_text() quick = re.search(r"##\s+Quick start\s*\n(.*?)(?=\n##\s|\Z)", readme, re.S) if not quick: sys.exit("README has no '## Quick start' section") commands = [l.strip() for blk in re.findall(r"```(?:bash|sh|shell)?\n(.*?)```", quick.group(1), re.S) for l in blk.splitlines() if l.strip() and not l.strip().startswith("#")] failed = None for cmd in commands: t = time.time() r = subprocess.run(cmd, shell=True, cwd=checkout, env=env, capture_output=True, text=True) print(f"{'ok ' if r.returncode == 0 else 'FAIL'} {time.time() - t:6.1f}s {cmd}") if r.returncode != 0: failed = (cmd, (r.stderr or r.stdout)[-400:]) break total = time.time() - t_all print(f"\ntotal: {total:.0f}s (limit {LIMIT_SECONDS}s) folder: {work}") if failed: print(f"\nA stranger would be stuck here:\n {failed[0]}\n{failed[1]}") sys.exit(1 if failed or total > LIMIT_SECONDS else 0)A Quick start that needs a key will fail here on purpose. Make that step explicit instead: copypython checks/fresh_clone_test.py https://github.com/you/your-repo.env.exampleto.envand tell the reader which keys to add, and have the project fail with a clear message when they are missing. - Put every check in CI: the Module 4 eval,
check_readme.pyandcheck_secrets.py, so a pull request that breaks the README, leaks a key or lowers the scores goes red. - Choose a license on purpose, after checking your dependencies' licenses (see the AGPL note in the concepts). Add it to the repo.
- Add the live links (your deployed service, the demo video, the write-up) to the README, then open every one from a browser that isn't signed in to anything.
- Last, watch someone else do it. Ask a friend or colleague who hasn't seen the project to follow the README on their own machine while you stay quiet and take notes. Every question they ask is a README bug.
The tooling in this lesson was run and tested: the web service (nine tests plus a live smoke run), the secrets scanner, the production-metrics report, the README checker in all its failing and passing states, the demo-script timer, the pitch checker, and the stranger test against a small sample repository, including a deliberately broken Quick start. The README tooling and the stranger test were not run against your project, and nothing here was deployed to a real host, so cold-start time, memory use, real latency and cost per request are yours to measure. Those are the numbers your README will carry.
Done when:
- A stranger can run each project from its README in under 10 minutes, and the stranger test passes.
- The live links work from a signed-out browser.
- Every number in every README is written by a tool from a file in the repo, with its date, and the README check passes in CI.
- No secrets are tracked, the history has none, and any key that ever leaked has been rotated.
- A blameless postmortem and a 60-second demo exist, and you can tell the four-sentence story without notes.
- At least one deployed app has an autonomy setting and a plain-language activity log, and a one-page product spec covering autonomy, trust and recoverability exists for it, doubling as a portfolio case study.
Sources
- Fitzpatrick, R. (2013). The Mom Test: How to Talk to Customers and Learn If Your Business Is a Good Idea When Everyone Is Lying to You.
- Wiggins, A. (2011). The Twelve-Factor App. 12factor.net.
- Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.) (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly. See the chapter Postmortem Culture: Learning from Failure.
- Module 2 (
pymupdf4llmlicensing), Module 4 (redaction, guardrails, CI evals), Module 5 (the seven failure classes) and Module 7 (the no-LLM option) of this course. - Google PAIR. People + AI Guidebook.
- Microsoft. Guidelines for Human-AI Interaction.