AI Beyond Fundamentals

What you'll learn

• Keeping API keys in environment files, never in code
• Python environments with venv and pip, in a monorepo layout (uv or poetry are fine alternatives)
• A shared client wrapper that logs tokens, cost and latency for every call

Hands-on labs

1. Scaffold the repo (1 to 2 h). One folder per project (chatbot, extraction, RAG, evals, agent, RAG vs fine-tune) plus a shared folder.
2. Three providers, one prompt (30 to 60 min). Call Claude, OpenAI and Gemini with the same prompt. Find the token counts in each raw response.
3. Run a model locally (about 18 to 36 min). Install Ollama, pull a small open model, and call it through its OpenAI-compatible endpoint.
4. Build the logger (about 13 to 26 min). Every call appends model, input tokens, output tokens, milliseconds and dollars to a log file. Every later module reuses it.

Open the full lesson Top

What you'll learn

• Provider APIs: messages, roles, system prompts, and how request shapes differ
• Tokens, context windows, and what happens at the limit
• Sampling settings: temperature, top-p, max tokens, stop sequences
• Streaming: time to first token versus total latency
• Cost per call, and why output tokens usually dominate
• The KV cache inside the model, and prompt caching as providers expose it
• Embeddings and cosine similarity: how software compares meaning
• Semantic caching: return a stored answer when a new question means the same thing
• Running open models locally with Ollama
• Picking a model by trading off quality, speed and cost

Hands-on labs

1. Token intuition (about 15 to 30 min). Guess the token count for English, code, JSON, emoji and Japanese, then check. Discuss why some cost more.
2. Context builds up (about 22 to 44 min). Run the same four-turn conversation wordy and compact, and watch input tokens and cost grow every turn.
3. Temperature sweep (about 3 to 6 min). Run one prompt 10 times at three temperatures and count how many distinct answers you get.
4. Streaming vs blocking (about 2 to 4 min). Time the same long answer both ways and chart the difference.
5. Prompt caching (about 4 to 8 min). Ask five questions about one long document, with and without caching. Compare cost and latency from your log.
6. Semantic cache (about 6 to 12 min). Test 20 paraphrased questions and find the similarity threshold where wrong answers start slipping through.

Open the full lesson Top

What you'll learn

• System prompts and few-shot examples, and how to choose good examples
• Step-by-step reasoning: when it improves accuracy and what it costs
• Structured output with JSON schemas and Pydantic, with validation and retry
• Tool calling: the request, the tool result, and the follow-up call
• Cutting hallucinations: ground answers in supplied text, require citations, allow "not found"
• Chain-of-Verification: draft, write check questions, answer them, revise
• Prompt versioning and A/B tests

Hands-on labs

1. Zero-shot vs few-shot. Sort 50 support tickets into 5 categories with 0, 2 and 5 examples. Compare accuracy.
2. Schema extraction. Define an invoice model, force schema output, and retry on validation failure. Count retries across 20 documents.
3. Tool calling by hand. Give the model an exchange-rate tool and a calculator. Log each round trip so the loop is visible.
4. Grounding. Ask 15 questions about a policy document, half unanswerable. Count invented answers with and without a "quote or say not found" rule.
5. Prompt versions. Store prompts as files, run two versions on the same inputs, and produce a results table.

Open the full lesson Top

Retrieval-augmented generation (RAG) has two halves. Documents are processed once and stored in an index. Every question then runs through search, and the best passages are handed to the model along with the question.

What you'll learn

• Digitizing documents: OCR for scans, text extraction for native PDFs
• Layout parsing that keeps tables, headers and reading order intact
• Chunking strategy per document type, including chunk size and overlap
• Vector indexes: HNSW (fast and accurate, memory-heavy), IVF (clustered, cheaper at scale), PQ (compressed, smaller and less exact)
• Embedding models and vector databases
• Hybrid search: keyword scores (BM25) plus vector scores, merged with reciprocal rank fusion
• Splitting multi-part questions and filtering by metadata such as date or source
• Reranking, then measuring retrieval with recall@k against labeled questions

Hands-on labs

1. Naive baseline. Build the simplest RAG over the course documents. Write 30 questions with the correct source page and record recall@5. Every later lab has to beat it.
2. Parsing bake-off. Extract 5 hard PDFs (one scanned, two table-heavy) with a basic extractor and a layout-aware parser. Note what each one lost.
3. Chunking experiment. Compare three chunking strategies on the same 30 questions.
4. Index types. Build HNSW and IVF-PQ indexes on 100,000 synthetic vectors. Measure build time, memory, speed and recall.
5. Hybrid search. Add keyword search and fuse the results. Find the questions it fixes, usually exact terms like IDs and product codes.
6. Rerank and filter. Retrieve 20, rerank to 5, and add a metadata filter. Record final recall and the added latency.

Open the full lesson Top

Decision models: a cheap, calibrated judge

A decision model answers one narrow question by choosing from answers you define in advance (yes or no, one of N choices, or a score) and returns a probability instead of prose. For calls shaped like that, a decision model runs far faster and cheaper than a general chat model (reported figures put it around 150ms versus 1.6 seconds for a general-purpose model on the same kind of call). The category sits alongside hosted options such as OpenAI's Decisions API and Perplexity's Decider, plus open-weight alternatives, and it shows up three times in this course: here as a safety gate, later in this module as a grader, and in Module 7 as a router. The core rule in every case: the decision model judges meaning, but authority lives in your code. Your harness sets the thresholds and enforces the outcome; the model is never the one deciding what happens next.

What you'll learn

• Golden sets: real user questions with expected answers, kept fresh
• Grading methods: exact match, code checks, an LLM judge with a rubric, human spot checks
• Calibrating the judge against your own labels before trusting it, and checking that calibration with a reliability diagram and a Brier score
• RAG metrics (RAGAS): faithfulness, answer relevance, context precision, context recall
• Tracing every call, retrieval and tool step (Langfuse, LangSmith)
• The lethal trifecta: private data access, exposure to untrusted content, and a way to send data out. Remove at least one leg for any given task
• Least privilege: scoped tools, allowlisted domains, read-only by default, with secrets injected by the harness at execution time and never placed in the model's context
• Input guardrails (personal data, prompt injection, off-topic) and output guardrails (schema, unsupported claims)
• Decision models as a pre-execution gate: score whether a tool call is safe before it runs, and fail closed on a refusal or an API error
• Human approval gates for irreversible or external actions, and treating the agent itself as a potential insider threat with audit logs and anomaly alerts
• Token budgets, cost alerts, and regression tests on every change

Direct vs. indirect injection

A direct injection is a user trying to override your instructions in the chat box. An indirect injection is malicious text hiding inside a web page, email, or document the agent reads on your behalf. It's the bigger real-world risk, because the agent never asked for it. Removing one leg of the lethal trifecta stops both:

Hands-on labs

1. Golden set. Turn the Module 3 questions into a 50-item set, including 10 the documents can't answer.
2. LLM as judge. Hand-grade 20 answers on a 1 to 5 faithfulness rubric, run the judge on the same 20, and report agreement.
3. RAGAS. Score the Module 3 app on all four metrics and find the weakest.
4. Tracing. Instrument the app and find the slowest step across 10 requests.
5. Guardrails. Redact personal data on input and require every claim to cite a passage on output. Plant a poisoned passage in the Module 3 corpus that tries to make the app exfiltrate notes, then try 10 attack prompts and count what gets through.
6. Decision gate. Put a decision model in front of every tool call with allow/ask/block thresholds. Run the same 10 attack prompts from the Guardrails lab through the gate alone, with no other defense, and report how many it catches on its own.
7. Calibration report. Hand-label 20 attack and 20 benign tool calls. Score them with your decision gate, plot a reliability diagram, compute a Brier score, and pick your allow/ask/block thresholds from that data rather than a guess.
8. Budget alert. Add a daily token cap per user and an alert when spend crosses a threshold.

Open the full lesson Top

What you'll learn

• The harness: model + tools + loop + state
• Context engineering: the context window as a budget. Compaction once you cross a threshold, with full tool outputs kept on disk and passed by reference instead of in full
• Patterns: sequential steps, parallel fan-out, and a loop that critiques and retries
• A coordinator that hands work to sub-agents and merges the results
• Long-running agents that write progress files so work survives a crash
• Sandboxing tool execution: an isolated container, no network, bounded CPU and memory, a timeout
• Idempotent tools, retries with backoff, and replay: rebuilding state by re-reading the session log instead of re-running everything
• In-band memory (notes written while working) versus out-of-band "dreaming": a separate job that reviews logs later and proposes memory edits for your approval
• MCP servers that expose your own tools to any compatible client, and the emerging agent-to-agent and agent-commerce protocols standardizing checkout, identity and payments between agents
• Limits (max steps, budget, timeout) and human approval for risky actions

Harness engineering: fix the system, not the prompt

The model is only the reasoning engine. It can suggest the next action, but it cannot create a reliable operating environment by itself. The harness decides what the model can see, what it can touch, what survives between sessions, what counts as evidence, and when the run must stop.

The prompt is one component inside this system, and the model is another. The product is what happens when every surrounding component works together.

"Prompt engineering improves the instruction. Harness engineering improves the conditions under which the instruction is executed."

rari, Harness Engineering

Put a model in a chat box and it answers questions. Put the same model in a repository with terminal access, tests, browser tools, project memory and a review loop, and it can ship software. The weights did not change. The harness did.

The seven jobs of a production harness

1. Turn the request into a contract. Before acting, write down the goal, inputs, expected output, constraints, and what "done" means. Without it, the agent can finish a different job and still declare success.
2. Give the agent a map. A small root guide that points to architecture, testing, product and security notes. A map preserves context, while a giant manual consumes it.
3. Expose the right tools in the right environment. Every tool needs a clear purpose, predictable output, an explicit failure state, and a permission boundary. Reading files can be allowed by default; deploying or deleting data requires approval. Anything that executes code runs inside an isolated sandbox (no network, bounded CPU and memory, a timeout) so a runaway call can't reach anything it shouldn't.
4. Externalize memory into durable state. The conversation is not the system of record. Store decisions, artifacts, failures and open risks in a file, so the next session inherits the work, not a lossy retelling of the chat.
5. Add sensors before adding autonomy. Tests, linters, screenshots, logs and schema checks turn vague quality into evidence. The model creates an artifact, the environment produces evidence about it, and the harness decides whether that is enough to continue.
6. Enforce permissions outside the model. The model suggests, a policy checks, a tool executes. Don't ask one probabilistic system to invent the plan, approve the risk and carry out the side effect.
7. Record traces and recover locally. Every run leaves a readable trail: the request, context selected, tool calls, state changes, checks, retries, cost and a rollback point. Without traces, failure is a mystery. With them, it is input for the next improvement.

Failure should upgrade the system

When an agent fails, the instinct is to rewrite the prompt, switch models, or add context. The failures usually come back. The better move is to name the class of failure and change the harness. The immediate patch fixes one run. The harness change improves every run after it.

Adapted from rari, Harness Engineering.

Instructions should become infrastructure

Rules kept only in prose get read, then eventually ignored. Encode each important rule twice: once as guidance the agent can understand, and once as a mechanical check it cannot bypass.

The guide explains the reason, and the check enforces the boundary. A past failure becomes a permanent fix. The next agent doesn't need to remember the incident, because the harness remembers it.

The loop belongs to the harness

"Keep trying until it works" is not a control system. A useful loop has evidence, a retry cap, a budget and an escalation path: build, verify, record the gap, and try again up to three times before asking a human. The model decides how to repair the gap. The harness decides whether another attempt is allowed.

Give every run a change receipt

Keep more than the final output. Save a compact receipt of how it was produced: context sources, model route, tools used, test results, human corrections, retries, cost, the accepted artifact and a rollback point. Receipts make model upgrades comparable, regressions attributable and audits possible.

Memory and dreaming: improving between runs

A receipt explains one run. Memory is what carries forward into the next one. There are two ways to write it: in-band, where the agent jots notes while it works, and out-of-band, a separate job that reviews logs later and proposes edits on its own schedule. The out-of-band kind is what gets called "dreaming": consolidation that happens after the work, not during it.

Keep memory in small topic files rather than one growing document. A few high-signal files beat a giant one nobody fully reads, including the agent. The real risk is that consolidation can turn a single bad call into permanent policy, silently, since nothing forced a second opinion on it. Version memory in git and gate every change behind human review before it's committed, the same way you'd review a pull request.

Start with the smallest harness that closes the loop

Move up a level only when the task earns it. A short, low-risk task may need one prompt and one review. A six-hour run that edits files, reaches the network and opens a pull request needs a real harness.

Before you trust an agent with real work

• Is success defined before the run begins?
• Can the agent find the right project knowledge without loading everything?
• Does every tool have a clear contract and failure state?
• Is execution isolated from production systems?
• Are important decisions stored outside the conversation?
• Does every risky step have evidence behind it?
• Are irreversible actions protected by approval?
• Does every loop have a retry cap and a budget?
• Can the run resume after an interruption?
• Can you explain every tool call and state change?
• Does each failure update a guide, test, tool or policy?
• Can the final result be rolled back?

If several answers are no, a stronger model won't make the system reliable. It will only make the failure more expensive. Checklist and levels adapted from rari, Harness Engineering.

Case study: an agent-first codebase

OpenAI describes a team that shipped an internal product over five months with no hand-written code: about a million lines and roughly 1,500 merged pull requests, starting with three engineers directing Codex. The engineers' job shifted to designing the environment, stating intent, and building feedback loops.

• Ask what capability is missing. Early progress was slow because the environment was underspecified, not because the model was weak.
• Give the agent a map, not a manual. One giant instructions file crowded out the task and went stale. A short entry file (about 100 lines) pointing into a structured docs folder worked better.
• If the agent can't see it, it doesn't exist. Decisions in chat threads or people's heads are invisible. Put them in versioned files in the repo.
• Make the app legible. Each change could boot its own instance, and the agent could drive the UI and query its own logs and metrics.
• Enforce invariants, not implementations. Custom lint rules whose error messages tell the agent how to fix the problem.
• Clean up continuously. Agents copy existing patterns, bad ones included. Background agents opened small refactoring pull requests on a schedule.

The authors caution that these results depend on heavy investment in that specific repository and shouldn't be assumed to generalize.

From managing agents to enabling them

In a conversation with Greg Isenberg, Allie K. Miller describes running 34 agents led by an AI chief of staff with a layer of directors beneath it. Her framing is that the job shifts from managing agents to enabling them, and that she has moved "from managing to like waiting for escalations." Four ideas from that conversation are worth building on:

Hands-on labs

1. Raw loop. Write a 60-line agent: call the model, run any requested tool, append the result, repeat until done. Set a budget limit from the start.
2. Three patterns. Summarize five articles three ways (sequential, parallel, loop) and compare time, cost and quality.
3. Sub-agents. A coordinator splits a research question into three parts, gives each to a sub-agent with its own context, and merges cited results.
4. Progress file. Have the agent update a progress file after each step. Stop it mid-run and resume from the file.
5. MCP server. Wrap the Module 3 search as an MCP server and call it from an MCP client.
6. Limits and approval. Add max steps, a dollar budget and a timeout, plus a confirm step before any risky tool. Show each one trigger.
7. Failure clinic. Debug a broken agent with seven seeded failures, one per row of the harness table, using only its traces and history log. Fix each and rerun to prove it's gone.
8. Make a repo agent-legible. Add a short map file and a docs folder (architecture, decisions, active plan) to your RAG repo, plus one lint rule whose error explains the fix. Give a coding agent the same feature request before and after, and count your corrections.
9. Agent chief of staff. Build a coordinator that reads a daily diary file and each sub-agent's progress file, flags duplicate or conflicting work, and escalates to you only when a judgment call is needed.
10. Sandboxed tool. Add a run_python tool that executes inside a Docker container with the network disabled, a 30 second timeout, and CPU and memory limits. Confirm a deliberate attempt to reach the internet from inside it fails.
11. Durable log and replay. Write every event (user message, model response, tool call, tool result) to an append-only SQLite or JSONL log before anything else happens. Add a resume command that rebuilds the conversation from the log, kill the process mid-task, and resume without repeating completed tool calls.
12. Idempotent tools. Give one side-effecting tool an idempotency key, call it twice with the same key, and confirm the side effect happens once. Wrap a flaky tool call in retries with exponential backoff.
13. Memory and dreaming. Write a nightly job that reads a day's session logs, proposes memory edits as a diff, and waits for your approval before committing them to a git-tracked memory folder the agent loads on startup. Run a 10 to 20 task eval set before and after a week of approved edits and compare the results.

Open the full lesson Top

What you'll learn

• Why a tool result is paid for on every later turn, and why to count tokens read instead of calls
• Letting the model write one script that calls your tools as Python functions, with only a summary coming back
• A sandbox that has the network on: what protects you when "no network" is not an option
• Measuring both modes on the same task, and when the loop is still the right choice
• Tools that fail loudly, using a real API that changed while this lesson was being written

Hands-on labs

1. Six tools as plain functions. Wrap the Metropolitan Museum of Art's open collection API with a disk cache, readable errors and schemas generated from type hints.
2. Loop mode. Give a model the tools and a task with checkable requirements, and log the tokens it reads on every turn.
3. The sandbox with the network on. Build the locked-down container and probe what it allows and blocks.
4. Code mode. Let the model write one script, run it in the sandbox, and compare three runs of each mode.
5. Read a failure. Hit a retired endpoint on purpose and see why loud errors beat quiet ones.

Open the full lesson Top

Hands-on labs

1. Router. Compare three ways to route requests between a small and large model: fixed rules, an LLM call that labels the request easy or hard, and a decision model that returns a probability. Score all three on cost, latency and accuracy against always using the large model.
2. Injection red team. Plant hidden instructions in a document in your RAG collection and try to make the app leak its instructions or misuse a tool. Add defenses and retest.
3. Replace the LLM. Find one model step from an earlier project that a rule could handle. Measure both.
4. Design doc. Write a one-page decision record with latency, cost and quality targets.

Open the full lesson Top

Requirements

• A real problem someone actually has, not a tutorial
• Raw API and your own loop first; a framework only if the README explains what it replaced
• A RAG app and an agent app, both deployed
• A README with the architecture, eval scores, cost per request and latency numbers
• An autonomy setting and a human-readable activity log on at least one deployed app
• A 60-second demo video
• A short write-up of what broke and how you fixed it

Product judgment: autonomy, trust and recoverability

Everything else in this course is a skill you can practice until it's reliable. This is judgment, and it's the biggest differentiator as a leader, not only as an engineer. Every feature needs an explicit stance on how much the system is allowed to do on its own before a human has to weigh in.

Match the level to the risk and reversibility of the action, not to how confident the system sounds. The decision models from Module 4 and Module 7 can drive that match directly: a calibrated probability is a reasonable input to "ask or act," while an uncalibrated one is just a confident-sounding guess.

• Trust calibration. Show work and sources, state uncertainty plainly, and avoid both over-trust and under-trust.
• Recoverability. Confirm before irreversible actions, offer undo, and keep an activity log a user can actually read.
• Failure UX. A graceful handoff to a human or a manual path when the system can't finish.
• Feedback loops. Thumbs up and down, corrections, and failures should flow back into your eval set, not disappear.

Hands-on labs

1. Problem pitch. One paragraph: the user, the problem, why an LLM helps, and how you'll measure success. Get feedback before building.
2. README first. Fill in the architecture, eval, cost and latency sections before writing code.
3. Deploy. Render, Fly.io, Railway or a cloud of your choice, with secrets in environment variables, a health check, and rate limiting.
4. Autonomy and activity log. Add an autonomy setting (suggest / confirm / act-and-report) and a human-readable activity log to one deployed app. Run a short usability test with two or three people and write down where they trusted the system when they shouldn't have, or didn't trust it when they should have.
5. Demo and postmortem. Record the video and write a blameless what-broke review.

Open the full lesson Top

An agent that finds the best way to book a trip on points. Points programs are confusing, transfer partners and award charts change, and people regularly overspend points or miss better options. The project touches every layer of the course, and its multi-agent step is honest: searching several loyalty programs at once is real parallel work. About 6 hours per part, around 40 in total; each bonus part unlocks after its matching module.

Hands-on labs

1. B1 · Give it tools. Four tools: search award space, list transfer options, compute cents per point, and explain an itinerary. Back availability with a snapshot dataset. Write the loop yourself, log every round trip, and set max steps and a dollar budget on day one.
2. B2 · Ground it in program rules. Save public award charts, transfer partners, fees and stopover rules as files with the date captured. Record a recall@5 baseline on 25 rule questions, then add keyword search for airport codes, fare classes and program names. Every answer cites its source and says "as of" that date.
3. B3 · Prove it works. Collect 40 real trip questions from friends and family, including 8 where the right answer is "pay cash." Check the math with code, grade explanations with a calibrated LLM judge, and run the set on every pull request. A deliberately broken prompt should fail the build.
4. B4 · Split it into a graph. Prototype three subagents in Claude Code (planner, program searcher, value checker), each with only the tools it needs. Fan out one searcher per loyalty program. Use a hook so the value checker always runs before an answer goes out. Then compare single agent vs graph on the golden set and keep the graph only if the numbers justify the extra tokens.
5. B5 · Make it production-shaped. Deploy behind your front end. Route simple lookups to a small model and trip planning to a large one. The agent never books and never handles loyalty account passwords. Plant an instruction in a rules page and show the defense holding. Track cost and p50/p95 latency per query. Stretch: swap your hand-rolled checkpointing from Module 5 for DBOS, a Postgres-backed durable workflow library, and confirm a killed process still resumes from its last completed step.
6. B6 · Give it a voice. Build a voice front end, either a pipeline (speech-to-text, your agent, text-to-speech) or a realtime speech-to-speech API. Support interruption, speak progress while a search runs, and use a decision model for two sub-second judgments inside each turn: has the user finished speaking, and does this turn need a tool call. Let a user upload a screenshot of a booking page and have the same decision model classify it (award search, confirmation, error page) before extraction.
7. B7 · Ship and tell the story. A 60-second demo, a README with the architecture and eval table, a blameless postmortem on three things that broke, and an interview summary: the problem, one decision defended with numbers, and one failure you diagnosed and fixed.

Decode the hype

When a new term starts trending, map it to a row. If you can't, bookmark it and check back in a month.

Fan-out multiplies tokens: Anthropic reported its multi-agent research system used roughly 15 times the tokens of a chat turn. Set the budget before B4, not after.

Top