AI Beyond Fundamentals › Module 2 › Lesson

Module 2 · Prompting & Reliability

Prompting & Reliability

Module 1 taught you what a call costs and how long it takes. This module is about whether you can trust what comes back — getting structured data you can parse, catching the model when it's about to guess, and treating a prompt as something you version and measure rather than something you tweak until it feels right.

Fair warning going in: several of this module's labs were built expecting classic failure modes (sloppy JSON, hallucinated facts on absent topics) that turned out not to reproduce with current mid-tier models. Rather than fake those failures, the labs below report what actually happened and redesign around the real finding. That's it's own lesson — test your assumptions about model behavior instead of inheriting them.

Concepts

System prompts and few-shot examples

A system prompt sets standing behavior before the conversation starts. Few-shot examples go further: instead of just describing the task, you show 2–5 worked examples of input → correct output directly in the prompt, and the model pattern-matches onto them. More examples isn't automatically better — Lab 1 below measures this directly rather than assuming it.

Choosing good examples matters more than choosing more of them: cover the edge cases and boundary confusions your task actually has, not just the easy, obvious cases.

Step-by-step reasoning

Asking a model to reason through a problem before answering ("think step by step," or a model's own built-in extended thinking) can improve accuracy on multi-step problems — arithmetic, logic, multi-hop questions — by giving it room to work instead of forcing an immediate answer. It costs real tokens (and therefore real money and latency) for every step reasoned through, and on simple tasks it can add cost with no accuracy benefit at all.

Advice on when reasoning helps differs by model family and changes as models improve — the only reliable way to know is to test it on your own task, not memorize a rule from last year.

Structured output: schemas, Pydantic, validation, and retry

Asking a model for JSON and getting back something your code can actually parse are two different problems. A Pydantic model defines the exact shape you expect; after the call, you parse the raw text and validate it against that shape, catching both malformed JSON and the right JSON shape with wrong types.

The real failure mode worth planning for isn't sloppy formatting — current mid-tier models are quite disciplined about that. It's truncation: the model's JSON gets cut off mid-object because max_tokens wasn't enough for a longer or more complex response. That failure is deterministic at a fixed budget, so retrying with the identical budget just fails again — the retry has to increase the budget, not just try again. Lab 2 demonstrates this directly.

Call the model with max_tokens Parse JSON Validate Pydantic schema Valid object use it Retry with a bigger budget double max_tokens, up to N tries passes cut off or wrong type
A retry with the same budget fails the same way, so the retry has to change something: here, a bigger max_tokens.

Tool calling: request, result, follow-up

Tool calling is a three-step loop, not a single call. You send the model a list of tools (name, description, a JSON schema for its arguments); the model's response can ask to call one by name with specific arguments instead of answering directly; you run that tool in your own code and send the result back as a new message; the model then either answers using that result or asks for another tool call. A multi-step task (look up a rate, then do arithmetic with it) can chain several of these rounds before producing a final answer — Lab 3 logs every round so the loop is visible instead of hidden inside the SDK.

Your code The model Your tool function 1. question + list of tools 2. "call exchange_rate(EUR, USD)" 3. you run it result: 1.09 4. result sent back as a message 5. final answer in plain text
The model never runs anything. It asks, you run, you report back. Steps 2 to 5 can repeat; that repetition is the agent loop.

Grounding and hallucination — what it actually buys you now

The classic advice is: require the model to quote or cite the source document, and to explicitly say "not found" rather than guess, to stop it from inventing facts about topics the document never covers. Testing that directly in Lab 4 (across plain questions, false-presupposition questions, zero-instruction prompts, and questions about adjacent-but-uncovered topics) found that a current mid-tier model didn't invent a single fact in any condition, grounding rule or not.

What the grounding rule actually changed was something else entirely: whether a correct answer came with a verifiable quote pointing back to its source. Without the rule, 1 answer out of 8 included an exact quote from the document; with it, all 8 did. The value shifted from "stops fabrication" to "makes correct answers auditable" — still a real, valuable property, just not the one the rule was originally sold on.

Chain-of-Verification

A four-step pattern for catching a model's own mistakes before they reach the user: draft an initial answer, generate a short list of fact-check questions against that draft, answer those check questions independently (often more reliable than the original pass, since each question is narrower), then revise the draft using anything the checks turned up. It costs 3–4x the calls of a single pass, so it's worth reserving for answers where being wrong is expensive — not a default for every request.

1. Draft first-pass answer 2. Check questions what must be true? 3. Answer each one at a time 4. Revise fix what failed each check is narrower than the original question, so it is often answered more reliably
Three to four times the calls of a single pass, so save it for answers where being wrong is expensive.

Prompt versioning and A/B tests

Treat a prompt the way you'd treat code: store each version as its own file, run both versions against the identical set of inputs, and compare with real numbers rather than a gut feeling about which "reads better." Lab 5 found something worth internalizing: a prompt change can fix the specific error it targeted while leaving aggregate accuracy completely flat, because it trades that mistake for a different one elsewhere. Look at which errors changed, not just whether the overall score moved.

Hands-on Labs

  1. Create chatbot/tickets.py with 50 hand-labeled support tickets, 10 per category, across 5 categories: Billing, Bug Report, Feature Request, Account Access, How-To Question. Also build two example pools — the first 2 and first 5 tickets of each category — for the few-shot conditions.
    # chatbot/tickets.py (excerpt -- 50 tickets total in the real file) CATEGORIES = ["Billing", "Bug Report", "Feature Request", "Account Access", "How-To Question"] TICKETS = [ ("I was charged twice for my subscription this month, can you refund the duplicate?", "Billing"), ("How do I update the credit card on file for my account?", "Billing"), ("The export button does nothing when I click it, no error, no download.", "Bug Report"), # ... 47 more, 10 per category ] def _examples_per_category(n): out = [] for cat in CATEGORIES: out.extend([t for t in TICKETS if t[1] == cat][:n]) return out FEW_SHOT_2 = _examples_per_category(2) # 10 examples total FEW_SHOT_5 = _examples_per_category(5) # 25 examples total
  2. Create chatbot/zero_shot_few_shot.py. One function builds the system prompt (category list, plus any examples formatted as Ticket/Category pairs); one classifies a single ticket; one runs all 50 and reports accuracy plus the first few wrong answers.
    import os from dotenv import load_dotenv load_dotenv() from openai import OpenAI from tickets import CATEGORIES, TICKETS, FEW_SHOT_2, FEW_SHOT_5 client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) def build_system_prompt(examples): prompt = ("Classify each support ticket into exactly one of these categories: " + ", ".join(CATEGORIES) + ". Reply with only the category name, nothing else.") if examples: prompt += "\n\nExamples:\n" for text, label in examples: prompt += f"Ticket: {text}\nCategory: {label}\n\n" return prompt def classify(ticket_text, system_prompt): resp = client.chat.completions.create( model="gpt-4.1-mini", max_tokens=10, messages=[{"role": "system", "content": system_prompt}, {"role": "user", "content": f"Ticket: {ticket_text}\nCategory:"}], ) return resp.choices[0].message.content.strip() def run(label, examples): system_prompt = build_system_prompt(examples) correct = 0 for text, true_label in TICKETS: if classify(text, system_prompt) == true_label: correct += 1 print(f"{label}: {correct}/{len(TICKETS)} correct ({correct/len(TICKETS):.0%})") return correct / len(TICKETS) if __name__ == "__main__": run("0-shot", []) run("2-shot", FEW_SHOT_2) run("5-shot", FEW_SHOT_5)
  3. Run it. All 150 calls (50 tickets × 3 conditions) take a couple of minutes.
    python chatbot/zero_shot_few_shot.py
Check: our own run came back 0-shot 90%, 2-shot 94%, 5-shot 92%. Few-shot helped over zero-shot, but more examples (5 vs. 2) didn't keep improving it — and at 50 tickets, a 1-ticket swing is well within noise, worth remembering before reading too much into small differences. The one consistent, non-noisy finding across all three conditions: "How do I update the credit card on file?" kept getting classified as How-To Question instead of Billing — a real category-boundary ambiguity (it's phrased as a how-to question about a billing topic), not something more examples fixed. Lab 5 comes back to this exact confusion.
  1. Install pydantic.
    pip install pydantic pip freeze > requirements.txt
  2. Create chatbot/schema_extraction.py with the Pydantic schema and a plain-prompt extraction call — no provider-level schema enforcement, just instructions, so there's something real to validate.
    import os, json from dotenv import load_dotenv load_dotenv() from openai import OpenAI from pydantic import BaseModel from typing import List client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) class LineItem(BaseModel): description: str quantity: int unit_price: float line_total: float class Invoice(BaseModel): invoice_number: str vendor: str line_items: List[LineItem] total: float SCHEMA_PROMPT = ("Extract structured data from this invoice as compact, single-line JSON " "(no indentation) with fields: invoice_number (string), vendor (string), line_items " "(list of objects with description, quantity, unit_price, line_total), and total (number).")
  3. Try it on a 5-line-item invoice at a few different max_tokens budgets. This is the part worth actually seeing fail: it's not the model being careless, it's the response getting cut off mid-object before the final field.
    inv_text = """Invoice #9981 Supplier: Oakridge Hardware items: Hammer (qty 4) $12.50 ea = $50.00 Nails box (qty 10) $3.00 ea = $30.00 Screwdriver set (qty 2) $18.00 ea = $36.00 Paint roller (qty 6) $4.50 ea = $27.00 Sandpaper pack (qty 8) $2.25 ea = $18.00 TOTAL DUE: $161.00""" for max_tok in [250, 150, 80]: resp = client.chat.completions.create( model="gpt-4.1-mini", max_tokens=max_tok, messages=[{"role": "system", "content": SCHEMA_PROMPT}, {"role": "user", "content": inv_text}], ) raw = resp.choices[0].message.content try: Invoice(**json.loads(raw)) print(f"max_tokens={max_tok}: parsed OK") except Exception as e: print(f"max_tokens={max_tok}: FAILED -> {type(e).__name__}") print(f" raw tail: ...{raw[-60:]!r}")
  4. Now write the retry logic the right way. A truncation failure at a fixed max_tokens is deterministic — retrying with the same budget fails again every time. Each retry has to raise the budget instead.
    from pydantic import ValidationError def extract_with_retry(invoice_text, max_attempts=3, start_tokens=80): max_tokens = start_tokens for attempt in range(max_attempts): resp = client.chat.completions.create( model="gpt-4.1-mini", max_tokens=max_tokens, messages=[{"role": "system", "content": SCHEMA_PROMPT}, {"role": "user", "content": invoice_text}], ) raw = resp.choices[0].message.content try: return Invoice(**json.loads(raw)), attempt except (json.JSONDecodeError, ValidationError): max_tokens *= 2 return None, max_attempts
Check: in our own run, the 5-line-item invoice failed at max_tokens=250 and 150 (cut off one token before the closing brace at 150), succeeded at 80 only after this exact invoice got lucky with a shorter response — the honest takeaway is that the failure point depends on both the invoice's complexity and the model's specific phrasing that run, not a fixed number you can hardcode. extract_with_retry starting at start_tokens=60 against this same invoice needed exactly 2 attempts (60 → 120 tokens) to succeed — confirm that on your own machine before trusting it.
  1. Create chatbot/tool_calling.py. Define both tools' schemas and the (fake, fixed-table) functions that actually run them.
    import os from dotenv import load_dotenv load_dotenv() from anthropic import Anthropic client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"]) EXCHANGE_RATES = {"USD_EUR": 0.92, "USD_GBP": 0.79, "EUR_USD": 1.09, "GBP_USD": 1.27} TOOLS = [ {"name": "calculator", "description": "Evaluate a basic arithmetic expression, e.g. '12.5 * 4 + 30'.", "input_schema": {"type": "object", "properties": {"expression": {"type": "string"}}, "required": ["expression"]}}, {"name": "exchange_rate", "description": "Look up the exchange rate between two currencies.", "input_schema": {"type": "object", "properties": { "from_currency": {"type": "string"}, "to_currency": {"type": "string"}}, "required": ["from_currency", "to_currency"]}}, ] def run_tool(name, tool_input): if name == "calculator": return str(eval(tool_input["expression"])) if name == "exchange_rate": key = f"{tool_input['from_currency']}_{tool_input['to_currency']}" return str(EXCHANGE_RATES.get(key, f"No rate available for {key}")) return f"Unknown tool: {name}"
  2. Write the loop: call the model, check whether it asked for a tool, run that tool yourself, send the result back as a new message, and repeat until it answers in plain text instead.
    def chat_with_tools(user_message, max_rounds=5): messages = [{"role": "user", "content": user_message}] print(f"User: {user_message}") for round_num in range(max_rounds): resp = client.messages.create(model="claude-sonnet-5", max_tokens=500, tools=TOOLS, messages=messages) tool_uses = [b for b in resp.content if b.type == "tool_use"] if not tool_uses: final_text = next((b.text for b in resp.content if b.type == "text"), "") print(f"Claude: {final_text}") return final_text messages.append({"role": "assistant", "content": resp.content}) tool_results = [] for tool_use in tool_uses: result = run_tool(tool_use.name, tool_use.input) print(f" [round {round_num}] {tool_use.name}({tool_use.input}) -> {result}") tool_results.append({"type": "tool_result", "tool_use_id": tool_use.id, "content": result}) messages.append({"role": "user", "content": tool_results}) if __name__ == "__main__": chat_with_tools("I'm buying something that costs 80 EUR. How much is that in USD, " "and if I buy 3 of them, what's the total in USD?")
  3. Run it and read the round-by-round trace.
    python chatbot/tool_calling.py
Check: our own run took 2 rounds — round 0 called exchange_rate(EUR, USD) alone; round 1 called calculator twice in the same round (once for the single-item price, once for 3×), which Claude requested in parallel rather than one at a time. The final answer ($87.20 single, $261.60 for 3) matched hand-calculated arithmetic exactly.
  1. Create chatbot/policy.py with a short reference document (a company policy works well — specific, falsifiable facts) and 15 questions, roughly half answerable from it and half not.
    # chatbot/policy.py (excerpt) POLICY_DOC = """Remote Work & Expense Policy -- Acme Corp, effective January 2026 1. Eligibility. Full-time employees who have completed 90 days of employment may work remotely up to 3 days per week with manager approval. ... 3. Internet reimbursement. Acme reimburses up to $50 per month toward home internet costs for approved remote employees, submitted quarterly via the expense portal. ... # 7 sections total, real file has the full text """ QUESTIONS = [ ("How many days per week can an eligible employee work remotely?", True), ("What's the monthly cap on internet reimbursement?", True), # ... 6 more answerable ("What is the pet insurance reimbursement amount for remote employees?", False), ("Does the company provide a signing bonus for remote hires?", False), # ... 5 more unanswerable ]
  2. Create chatbot/grounding.py with two prompts — one plain, one requiring a verbatim quote and an explicit "not found" refusal — and two separate checks: does it invent facts on the unanswerable half, and when it's right, can you verify where the answer came from.
    import os from dotenv import load_dotenv load_dotenv() from openai import OpenAI from policy import POLICY_DOC, QUESTIONS client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) PLAIN_PROMPT = "Answer the question based on this policy document:\n\n" + POLICY_DOC GROUNDED_PROMPT = ("Answer the question using ONLY this policy document. Quote the relevant " "sentence from the document in your answer. If the document does not cover the question, " "respond with exactly: Not found in the document.\n\n" + POLICY_DOC) def ask(system_prompt, question): resp = client.chat.completions.create(model="gpt-4.1-mini", max_tokens=150, messages=[{"role": "system", "content": system_prompt}, {"role": "user", "content": question}]) return resp.choices[0].message.content.strip() def has_verbatim_quote(answer, doc, min_words=8): # a paraphrase won't match the source verbatim; an actual quote will words = answer.replace('"', "").split() for i in range(len(words) - min_words + 1): if " ".join(words[i:i + min_words]) in doc: return True return False
  3. Run the two checks against both prompts: count invented answers on the unanswerable half, and count how many answerable questions got a real quoted citation.
    python chatbot/grounding.py
Check: our own run found 0/7 invented answers in both conditions — the model correctly said "the document doesn't mention X" every time, grounding rule or not, even on questions designed to invite guessing (false premises, zero-instruction prompts, questions about topics adjacent to what's covered). The real difference showed up in citations: the plain prompt included a verbatim quote in 1/8 correct answers; the grounded prompt did in 8/8. If your use case needs an auditable trail back to the source — compliance, legal, anything someone might ask "where does it say that" about — the rule earns its keep on citation alone, independent of whether fabrication was ever really the risk for your model.
  1. Reuse Lab 1's tickets.py. Create chatbot/prompts/v1_terse.txt (the same bare instruction Lab 1 used at 0-shot) and chatbot/prompts/v2_defined_boundaries.txt, which tries to directly fix the Billing/How-To confusion Lab 1 found.
    # chatbot/prompts/v2_defined_boundaries.txt Classify each support ticket into exactly one of these categories. Reply with only the category name, nothing else. Billing: anything about payment, invoices, charges, subscriptions, or pricing -- including when it's phrased as a "how do I" question (e.g. "how do I update my card" is Billing, not How-To Question). Bug Report: something in the product is broken or behaving incorrectly. Feature Request: the customer wants a new capability that doesn't exist yet. Account Access: login, authentication, permissions, or account security issues. How-To Question: asking how to use an existing feature that already works as intended.
  2. Create chatbot/prompt_versions.py, running both files against all 50 tickets and reporting both aggregate accuracy and the specific confusion count, not just the total.
    import os, pathlib from dotenv import load_dotenv load_dotenv() from openai import OpenAI from tickets import TICKETS client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) PROMPTS_DIR = pathlib.Path(__file__).parent / "prompts" def classify(ticket_text, system_prompt): resp = client.chat.completions.create(model="gpt-4.1-mini", max_tokens=10, messages=[{"role": "system", "content": system_prompt}, {"role": "user", "content": f"Ticket: {ticket_text}\nCategory:"}]) return resp.choices[0].message.content.strip() def run_version(prompt_file): system_prompt = (PROMPTS_DIR / prompt_file).read_text() correct, billing_as_howto = 0, 0 for text, true_label in TICKETS: predicted = classify(text, system_prompt) if predicted == true_label: correct += 1 if true_label == "Billing" and predicted == "How-To Question": billing_as_howto += 1 return correct, billing_as_howto if __name__ == "__main__": for prompt_file in ["v1_terse.txt", "v2_defined_boundaries.txt"]: correct, confused = run_version(prompt_file) print(f"{prompt_file}: {correct}/{len(TICKETS)} ({correct/len(TICKETS):.0%}), " f"Billing->How-To mix-ups: {confused}")
  3. Run it.
    python chatbot/prompt_versions.py
Check: our own run: both versions landed at 47/50 (94%) — identical aggregate accuracy, and also different from Lab 1's 0-shot run of the literal same v1 prompt (45/50 that time), a real reminder that outputs vary run to run even with an unchanged prompt. But the targeted number moved exactly as intended: v1 had 1 Billing/How-To mix-up, v2 had 0. v2 didn't get "better" on paper, it traded that specific mistake for a different one elsewhere — which is the actual finding worth reporting, not the flat headline number.

Project

Project · Invoice PDF to clean JSON

Lab 2 built the schema + retry pattern on one invoice. This project runs it for real at 20-invoice scale, against actual PDF files, with a step this module's earlier labs skipped entirely: converting a PDF into something an LLM can reliably read in the first place.

Why PDF-to-Markdown first, not raw text

A naive pypdf-style text dump reads a PDF's text in element position order with no concept of a table — on a real multi-column invoice, that can scramble or interleave columns into unusable text. A converter that targets Markdown instead (we tested pymupdf4llm) detects the table and emits it as an actual Markdown table, which an LLM parses far more reliably. This is standard practice in real document-extraction pipelines (Microsoft's markitdown, Docling, and commercial tools like LlamaParse all target Markdown as the intermediate format specifically for this reason) — not something invented for this course. pymupdf4llm is dual-licensed AGPL-3.0 or a paid commercial license from Artifex; fine for an open project like this one, but check that license before shipping it inside anything closed-source.

  1. Install the PDF libraries.
    pip install reportlab pypdf pymupdf4llm pip freeze > requirements.txt
  2. Create chatbot/generate_invoices.py. It generates 20 varied invoices as real PDFs with genuine side-by-side table columns (not one item per line), tracking the exact correct values as it goes — that's your hand-labeled answer key, generated honestly rather than eyeballed after the fact.
    # chatbot/generate_invoices.py (excerpt -- full file generates 20) import json, random, pathlib from reportlab.pdfgen import canvas random.seed(42) # reproducible across runs VENDORS = ["Summit Office Supply", "Blue Mountain Hardware", # ... 20 total] ITEM_POOL = [("Widget", 5, 50), ("Office Chair", 80, 220), # ... 20 total, (desc, price_lo, price_hi)] def make_invoice(i): vendor = VENDORS[i] items = random.sample(ITEM_POOL, random.randint(1, 6)) line_items, total = [], 0.0 for desc, lo, hi in items: qty = random.randint(1, 10) unit_price = round(random.uniform(lo, hi), 2) line_total = round(qty * unit_price, 2) line_items.append({"description": desc, "quantity": qty, "unit_price": unit_price, "line_total": line_total}) total += line_total ground_truth = {"invoice_number": f"INV-2026-{1000+i:04d}", "vendor": vendor, "line_items": line_items, "total": round(total, 2)} c = canvas.Canvas(str(pathlib.Path("chatbot/invoices") / f"invoice_{i:02d}.pdf")) c.drawString(72, 750, f"INVOICE #{ground_truth['invoice_number']}") c.drawString(72, 730, f"Vendor: {vendor}") xs = [72, 300, 380, 470] # real side-by-side columns # ... draws header row + one row per line item at these x-positions, then Total c.save() return ground_truth
  3. Create chatbot/extract_invoices.py: convert each PDF to Markdown with OCR explicitly off (these are typed PDFs, not scans — OCR is Module 3's territory, and forcing it off here avoids needing a separate Tesseract system install), extract with Lab 2's retry pattern, then score against the ground truth field by field.
    import pymupdf4llm from pydantic import ValidationError def extract_with_retry(markdown_text, prompt, max_attempts=3, start_tokens=400): max_tokens = start_tokens for attempt in range(max_attempts): resp = client.chat.completions.create(model="gpt-4.1-mini", max_tokens=max_tokens, messages=[{"role": "system", "content": prompt}, {"role": "user", "content": markdown_text}]) try: return Invoice(**json.loads(resp.choices[0].message.content)), attempt except (json.JSONDecodeError, ValidationError): max_tokens *= 2 return None, max_attempts def field_accuracy(extracted, truth): checks = [extracted.invoice_number == truth["invoice_number"], extracted.vendor == truth["vendor"], abs(extracted.total - truth["total"]) < 0.01, len(extracted.line_items) == len(truth["line_items"])] for e, t in zip(extracted.line_items, truth["line_items"]): checks += [e.description == t["description"], e.quantity == t["quantity"], abs(e.unit_price - t["unit_price"]) < 0.01, abs(e.line_total - t["line_total"]) < 0.01] return sum(checks), len(checks) def arithmetic_ok(extracted): return abs(sum(i.line_total for i in extracted.line_items) - extracted.total) < 0.01 # in the run loop, per invoice: # md_text = pymupdf4llm.to_markdown(pdf_path, use_ocr=False) # extracted, attempts = extract_with_retry(md_text, prompt)
  4. Run the full comparison across both of Lab 5's prompt files, reused as-is on this different task.
    python chatbot/generate_invoices.py python chatbot/extract_invoices.py
Check: our own run against all 20 invoices: 340/340 (100%) field accuracy for both the terse and the explicit prompt, 0/20 needed a retry, 0/20 arithmetic check failures. A flat, honest result — at this invoice complexity, prompt wording didn't move the needle, which is itself worth knowing before spending effort hand-tuning a prompt that's already working. We separately confirmed the retry mechanic still fires correctly under a tight budget: the invoice with the most line items (6) forced to start_tokens=60 needed exactly 2 attempts to succeed, the same truncate-then-double pattern Lab 2 found.
Want a harder invoice set that actually produces some failures? Hand your agent this:
Here is my generate_invoices.py (paste the file). All 20 invoices currently extract at 100% accuracy with zero retries, which doesn't show the failure modes this project is supposed to surface. Modify it to generate a few harder invoices: at least one with 8+ line items (to stress max_tokens truncation), at least one with an ambiguous or missing vendor name, and at least one where a line item's quantity times unit_price doesn't exactly equal its printed line_total (a real data-entry inconsistency). Keep the ground truth JSON honest -- it should reflect what the PDF actually shows, including the inconsistency, not a "corrected" version.

Done when:

  • Every one of the 20 outputs passes Pydantic validation — no unhandled parse failures.
  • Field accuracy is reported against the hand-labeled (here, honestly-generated) ground truth, not just "looks right."
  • The arithmetic check (line items sum to the printed total) runs and is reported separately from field accuracy — they can diverge, and that's informative when they do.
  • Two prompt versions are compared with real numbers, even if — like ours — the honest result is that they tied.
← Back to the course map