Architecture Decisions

1. Freeze your eval set and clean the corpus. Copy the golden set so later steps can't change it, and make sure the poisoned page from Lab 2 is gone, since a poisoned document would contaminate every system.
   cp evals/golden.jsonl ragvsfinetune/golden_v1.jsonl rm -f rag/corpus/900_delta_sensor_faq.pdf
2. Build the training set. One system prompt is shared by training and evaluation (so they can't drift apart). The builder writes four differently worded questions per fact, plus refusals for things the manuals never say, and then asserts that no training question is identical to a golden question. The model is tested on facts it was trained on, but never on a question it saw verbatim; without that check a fine-tuned model can look brilliant by memorizing the exam.
   """The one system prompt shared by training and evaluation, so the two can never drift apart.""" SYSTEM = ("You answer questions about Larkfield product manuals. Reply in the form " "'Answer: <value>. Source: <file> p.<N>', or 'Answer: The manuals do not contain this.'")
   """Build a supervised fine-tuning set from the manuals' facts. Every question is worded differently from the golden set's questions, so the model is tested on facts it was trained on but not on questions it saw.""" import json, re, random from pathlib import Path import pypdf from build_train_system import SYSTEM WARRANTY = ["What warranty period does the {n} carry?", "{n}: how many months of warranty?", "For the Larkfield {n}, how long does the manufacturer's warranty run?", "Tell me the warranty term of the {n}."] PRESSURE = ["What pressure rating does the {n} have?", "{n} max pressure in bar?", "What is the highest pressure the Larkfield {n} can handle?", "Give me the pressure limit for the {n}."] INTERVAL = ["How frequently does the {n} need recalibrating?", "{n}: recalibration interval?", "After how many months should the Larkfield {n} be recalibrated?", "When is the {n} due for recalibration?"] CODE = ["What is the part number of the {n}?", "{n}: part number?"] ABSENT = ["What color is the {n}?", "How much does the {n} weigh?", "What is the battery life of the {n}?"] golden = set() # every eval question that exists, old and new for path in ["evals/golden.jsonl", "ragvsfinetune/golden_v1.jsonl", "ragvsfinetune/golden_new.jsonl"]: if Path(path).exists(): golden |= {json.loads(l)["question"] for l in open(path)} qs = json.load(open("rag/questions.json")) facts = {} for q in qs: f = facts.setdefault(q["doc"], {}) f["name"] = re.sub(r"^\d+_|_scan|\.pdf", "", q["doc"]).replace("_", " ").title() t = q["q"].lower() if "warranty" in t: f["warranty"] = q["answer"] elif "pressure" in t: f["pressure"] = q["answer"] elif "recalibrate" in t: f["interval"] = q["answer"] for doc, f in facts.items(): # part numbers come from page 1 of the PDF itself m = re.search(r"LK-\d{4}-[A-H]", pypdf.PdfReader(f"rag/corpus/{doc}").pages[0].extract_text() + "") if m: f["code"] = m.group() def ex(question, answer): return {"messages": [{"role": "system", "content": SYSTEM}, {"role": "user", "content": question}, {"role": "assistant", "content": answer}]} rows = [] for doc, f in facts.items(): n = f["name"] for key, page, templates in [("warranty", 1, WARRANTY), ("pressure", 2, PRESSURE), ("interval", 3, INTERVAL)]: for t in templates: rows.append(ex(t.format(n=n), f"Answer: {f[key]}. Source: {doc} p.{page}")) if "code" in f: for t in CODE: rows.append(ex(t.format(n=n), f"Answer: {f['code']}. Source: {doc} p.1")) for t in ABSENT[:2]: # teach it to decline; the golden set uses different asks rows.append(ex(t.format(n=n), "Answer: The manuals do not contain this.")) random.seed(5); random.shuffle(rows) leaks = [r for r in rows if r["messages"][1]["content"] in golden] assert not leaks, f"{len(leaks)} training questions are identical to golden questions: {leaks[0]}" with open("ragvsfinetune/train.jsonl", "w") as f: for r in rows: f.write(json.dumps(r) + "\n") tokens = sum(len(m["content"]) for r in rows for m in r["messages"]) // 4 print(f"{len(rows)} examples, about {tokens:,} tokens per epoch. No question matches a golden question.")
   python ragvsfinetune/build_train.py
   Refusals use the sentence "The manuals do not contain this", chosen so Module 4's refusal grader recognizes it without any change.
3. Run the fine-tune. The script uploads the file, starts a hosted job, polls until it finishes, saves the model's name, and prints the wall-clock time and trained-token count. Those two numbers are your effort and cost to update. Fill in ragvsfinetune/prices.py first. Check your provider's current documentation for which models can be fine-tuned and their exact ids; this changes often.
   """Upload the training set, start a hosted fine-tuning job, wait, and save the new model's name. Check your provider's current docs first: which models can be fine-tuned, and the exact model id, change.""" import os, time from dotenv import load_dotenv load_dotenv() from openai import OpenAI BASE = "gpt-4.1-mini-2025-04-14" # confirm this id is fine-tunable for your account before running client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) uploaded = client.files.create(file=open("ragvsfinetune/train.jsonl", "rb"), purpose="fine-tune") job = client.fine_tuning.jobs.create(training_file=uploaded.id, model=BASE) print("job", job.id, "started at", time.strftime("%H:%M:%S")) started = time.time() while True: job = client.fine_tuning.jobs.retrieve(job.id) print(f" {time.time() - started:5.0f}s {job.status}") if job.status in ("succeeded", "failed", "cancelled"): break time.sleep(30) if job.status != "succeeded": raise SystemExit(f"job {job.status}: {job.error}") open("ragvsfinetune/ft_model.txt", "w").write(job.fine_tuned_model) print("fine-tuned model:", job.fine_tuned_model) print(f"wall-clock time to a usable model: {(time.time() - started) / 60:.1f} minutes") print("trained tokens (for your cost line):", job.trained_tokens)
   """Dollars per 1,000,000 tokens: (input, output). Fill in from your provider's pricing page on the day you run this; a fine-tuned model usually costs more per token than its base. Training is priced per trained token.""" PRICES = { "base": (0.0, 0.0), "finetuned": (0.0, 0.0), } TRAINING_PER_MTOK = 0.0
   python ragvsfinetune/finetune.py
4. Evaluate all three systems on the frozen golden set: the base model with no retrieval, your Module 3 RAG pipeline (cache off), and the fine-tuned model with no retrieval. Every answer, token count and latency is saved.
   """Run the same eval set through three systems and save every answer, token count and latency. python ragvsfinetune/eval_systems.py (golden set from Module 4) python ragvsfinetune/eval_systems.py new (questions about a product added after training)""" import json, os, sys, time sys.path.insert(0, "rag"); sys.path.insert(0, "evals") from dotenv import load_dotenv load_dotenv() from openai import OpenAI from grading import grade from build_train_system import SYSTEM import pipeline pipeline.CACHE_THRESHOLD = 2.0 # a cache hit would hide what we're measuring client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) BASE = "gpt-4.1-mini" FT = open("ragvsfinetune/ft_model.txt").read().strip() if os.path.exists("ragvsfinetune/ft_model.txt") else None def closed_book(model): """No retrieval. Whatever the model knows, or was trained to know, is all it has.""" def run(q): t0 = time.perf_counter() r = client.chat.completions.create(model=model, max_tokens=60, messages=[ {"role": "system", "content": SYSTEM}, {"role": "user", "content": q}]) return r.choices[0].message.content, r.usage.prompt_tokens, r.usage.completion_tokens, (time.perf_counter() - t0) * 1000 return run def rag(q): r = pipeline.ask(q) return r["answer"], r["input_tokens"], r["output_tokens"], r["latency_ms"] SYSTEMS = {"base": closed_book(BASE), "rag": rag} if FT: SYSTEMS["finetuned"] = closed_book(FT) if __name__ == "__main__": which = sys.argv[1] if len(sys.argv) > 1 else "golden" path = "ragvsfinetune/golden_new.jsonl" if which == "new" else "ragvsfinetune/golden_v1.jsonl" items = [json.loads(l) for l in open(path)] for name, fn in SYSTEMS.items(): rows = [] for it in items: text, i, o, ms = fn(it["question"]) rows.append({"id": it["id"], "answerable": it["answerable"], "question": it["question"], "answer": text, "input_tokens": i, "output_tokens": o, "latency_ms": ms, "passed": grade(it, text)}) json.dump(rows, open(f"ragvsfinetune/run_{which}_{name}.json", "w"), indent=1) a = [r for r in rows if r["answerable"]]; u = [r for r in rows if not r["answerable"]] print(f"{name:10s} answerable {sum(r['passed'] for r in a)}/{len(a)} " f"unanswerable {sum(r['passed'] for r in u)}/{len(u)}")
   python ragvsfinetune/eval_systems.py
5. Now the update test, which is where the two approaches differ most. Add a thirteenth manual. The generator is seeded, so the first twelve come out identical and only the new product's questions are new. Evaluate everyone on them. RAG needs its index rebuilt (time it); the fine-tuned model hasn't heard of the product, so it needs another training run (time and cost it too, after adding the new facts to the training set).
   """After you add a 13th manual (python rag/generate_corpus.py 13), build eval questions about it. The first 12 manuals come out identical, so only the new product's questions are new.""" import json qs = [q for q in json.load(open("rag/questions.json")) if q["doc"].startswith("012_")] assert qs, "no questions for doc 012_: regenerate the corpus with 13 manuals first" with open("ragvsfinetune/golden_new.jsonl", "w") as f: for i, q in enumerate(qs, 1): f.write(json.dumps({"id": i, "question": q["q"], "answerable": True, "expected": q["answer"], "doc": q["doc"], "pages": q["pages"]}) + "\n") print(f"{len(qs)} questions about the new product")
   python rag/generate_corpus.py 13 python ragvsfinetune/make_new_golden.py python ragvsfinetune/eval_systems.py new
   Record minutes and dollars to add one product, for each system, in ragvsfinetune/effort.json, for example {"rag": {"minutes": 2, "dollars": 0.01}, "finetuned": {"minutes": 25, "dollars": 0.50}} with your own measurements.
6. Build the comparison table: quality on answerable and unanswerable questions, performance on the new product, tokens per request, dollars per 1,000 requests, median latency, and effort to add a product.
   """Turn the run files into one comparison table: quality, cost, latency and effort to update.""" import json, statistics, sys from pathlib import Path from prices import PRICES, TRAINING_PER_MTOK def load(path): p = Path(path) return json.loads(p.read_text()) if p.exists() else None def rate(rows, key): xs = [r for r in rows if r["answerable"] == key] return f"{sum(r['passed'] for r in xs)}/{len(xs)}" if xs else "n/a" def per_1000(rows, system): pin, pout = PRICES["finetuned" if system == "finetuned" else "base"] if (pin, pout) == (0.0, 0.0): return "fill in prices.py" cost = sum(r["input_tokens"] * pin + r["output_tokens"] * pout for r in rows) / len(rows) * 1000 / 1_000_000 return f"${cost:.2f}" effort = load("ragvsfinetune/effort.json") or {} # you fill this in: minutes and dollars to add one product lines = ["| system | answerable | unanswerable | new product | tokens/request | $ per 1,000 requests | p50 latency | to add a product |", "|---|---|---|---|---|---|---|---|"] for system in ["base", "rag", "finetuned"]: rows = load(f"ragvsfinetune/run_golden_{system}.json") if not rows: continue new = load(f"ragvsfinetune/run_new_{system}.json") tokens = statistics.mean(r["input_tokens"] + r["output_tokens"] for r in rows) p50 = statistics.median(r["latency_ms"] for r in rows) e = effort.get(system, {}) lines.append(f"| {system} | {rate(rows, True)} | {rate(rows, False)} | " f"{(sum(r['passed'] for r in new)) if new else 'n/a'}{'/' + str(len(new)) if new else ''} | " f"{tokens:,.0f} | {per_1000(rows, system)} | {p50:,.0f} ms | " f"{e.get('minutes', '?')} min, ${e.get('dollars', '?')} |") Path("ragvsfinetune/results.md").write_text("\n".join(lines) + "\n") print("\n".join(lines))
   python ragvsfinetune/report.py
7. Compute the break-even yourself. If fine-tuning is cheaper per request, at what request volume does that saving repay the training run, and does that volume change if facts change monthly? Then write the recommendation as a decision record from Lab 4, with targets and a revisit if condition, and run the checker on it.

Done when:

• All systems use the same frozen golden set, and the training set is checked to contain no question identical to it.
• A table compares quality, cost, latency and effort to update, with the new-product test included.
• The recommendation is a decision record that passes the checker, and each target in it is backed by a number in your table.
• The record's revisit if line names the measured change that would flip the recommendation.