LLM Fundamentals

1. Install tiktoken — it's OpenAI's own tokenizer library, the only way to count OpenAI-style tokens without making a real API call.
   pip install tiktoken pip freeze > requirements.txt
2. Create chatbot/token_intuition.py (code chatbot/token_intuition.py, or nano). Before running anything, write your own guess for each sample as a comment — the point of this lab is checking your intuition, not just reading numbers off a script.
   # Guess first! Write your estimate for each sample below, then run this # file and see how close you were. # English: ? Code: ? JSON: ? Emoji: ? Japanese: ? import os from dotenv import load_dotenv load_dotenv() SAMPLES = { "English": "The quick brown fox jumps over the lazy dog.", "Code": "def fib(n):\n return n if n < 2 else fib(n-1) + fib(n-2)", "JSON": '{"name": "Ada", "age": 36, "skills": ["math", "engineering"]}', "Emoji": "🚀🔥😂👍💯🎉😍🙌✨🤔", "Japanese": "猫はかわいいです。今日は良い天気ですね。", } def anthropic_tokens(text): from anthropic import Anthropic client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"]) resp = client.messages.count_tokens( model="claude-haiku-5-5", messages=[{"role": "user", "content": text}], ) return resp.input_tokens def openai_tokens(text): import tiktoken enc = tiktoken.encoding_for_model("gpt-5.4-mini") return len(enc.encode(text)) def gemini_tokens(text): from google import genai client = genai.Client(api_key=os.environ["GEMINI_API_KEY"]) resp = client.models.count_tokens(model="gemini-flash-lite-latest", contents=text) return resp.total_tokens if __name__ == "__main__": for label, text in SAMPLES.items(): a, o, g = anthropic_tokens(text), openai_tokens(text), gemini_tokens(text) print(f"{label:10} anthropic={a:3} openai={o:3} gemini={g:3}")
3. Run it and compare the three columns, then compare them to your own guesses.
   python chatbot/token_intuition.py

A chat API has no memory. Every turn, your code sends the entire conversation again: the system prompt, every earlier question, and every earlier answer. This lab makes that visible and shows why clear, compact prompts are a cost and quality skill, not just a style preference.

1. Create chatbot/context_growth.py (code chatbot/context_growth.py, or nano) and paste in the script. It asks the same four gardening questions twice. The verbose run uses a long system prompt and rambling questions. The compact run asks for the same things in a few words, with a short system prompt that asks for short answers. Each turn appends to messages and resends everything, then prints that turn's token usage. Every call is logged through log_call, so it also lands in shared/calls.jsonl. Note thinking={"type": "disabled"} on the call: Claude Haiku 5.5 can "think" before it answers, and thinking tokens count against max_tokens. With the wordy system prompt it will often spend the entire 400 tokens thinking and return no text at all, so next(...) raises StopIteration. Turning thinking off also keeps the token counts about the conversation and not hidden reasoning.
   import os, sys, time, pathlib from dotenv import load_dotenv load_dotenv() from anthropic import Anthropic sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "shared")) from llm_log import log_call client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"]) MODEL = "claude-haiku-5-5" PRICE_IN, PRICE_OUT = 0.10, 0.50 # $ per 1M tokens, from Module 0's models.yaml # The same four questions, asked two ways. The compact version says the same # thing in far fewer words, with a short system prompt that asks for short answers. COMPACT = { "system": "You are a gardening assistant. Answer in at most two sentences.", "turns": [ "I have a sunny 4x8 ft bed. What should I plant in spring?", "Which of those is easiest for a beginner?", "How far apart should I space it?", "And how often do I water it?", ], } VERBOSE = { "system": ( "You are a friendly, thorough, and highly knowledgeable gardening assistant. " "Always greet the user warmly, explain your reasoning step by step, mention any " "relevant caveats, regional differences, seasonal considerations, and common " "mistakes, offer alternative approaches where possible, and finish by inviting " "follow-up questions. Be comprehensive rather than brief." ), "turns": [ "Hi there! So I've been thinking about getting into gardening for a while now, and " "I finally have a bit of free time. I have a raised bed in my backyard, it's about " "4 feet by 8 feet, and it gets a lot of sun, I think maybe six or seven hours on " "a good day. Anyway, I was wondering, what are some things I could plant this spring?", "Thanks, that's really helpful! There's a lot there. Of all of the ones you just " "mentioned, which one do you think would be the easiest for someone who has honestly " "never grown anything before in their life? I'd hate to fail on the first try.", "Great, I think I'll go with that one. Now, I'm not totally sure about the spacing, " "so could you tell me how far apart I should put the plants so they have enough room?", "Okay, that makes sense. One last thing, I'm a little worried about overwatering or " "underwatering, so how often should I actually be watering it, roughly speaking?", ], } def run(label, scenario): messages = [] total_in = total_out = 0 total_cost = 0.0 print(f"--- {label} ---") for turn, user_text in enumerate(scenario["turns"], 1): messages.append({"role": "user", "content": user_text}) start = time.time() resp = client.messages.create( model=MODEL, max_tokens=400, system=scenario["system"], messages=messages, thinking={"type": "disabled"}, # otherwise Haiku can spend the whole token budget thinking and return no text ) elapsed_ms = (time.time() - start) * 1000 reply = next(b.text for b in resp.content if b.type == "text") messages.append({"role": "assistant", "content": reply}) i, o = resp.usage.input_tokens, resp.usage.output_tokens cost = i / 1_000_000 * PRICE_IN + o / 1_000_000 * PRICE_OUT log_call("anthropic", f"{MODEL}-{label}", i, o, elapsed_ms, cost) total_in, total_out, total_cost = total_in + i, total_out + o, total_cost + cost print(f" turn {turn}: input={i:5} output={o:4} cost=${cost:.6f}") print(f" TOTAL : input={total_in:5} output={total_out:4} cost=${total_cost:.6f}\n") return total_in, total_out, total_cost if __name__ == "__main__": v = run("verbose", VERBOSE) c = run("compact", COMPACT) print(f"verbose / compact: input {v[0] / c[0]:.1f}x, output {v[1] / c[1]:.1f}x, cost {v[2] / c[2]:.1f}x")
2. Run it and read the two tables.
   python chatbot/context_growth.py
3. Before you read our numbers, answer three questions from your own output: (a) In the compact run, the user's turn 4 question is only a few words. Why is the turn 4 input count still several times larger than turn 1's? (b) Where did the verbose run's turn 2 jump come from? (c) Which costs more, the system prompt or the conversation history, by turn 4?

1. Create chatbot/temperature_sweep.py and paste in a function that calls the same prompt n times at a given temperature and collects the answers.
   import os from dotenv import load_dotenv load_dotenv() from openai import OpenAI client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) PROMPT = "Name an animal. Reply with just the word." def sweep(temperature, n=10): answers = [] for _ in range(n): resp = client.chat.completions.create( model="gpt-5.4-mini", max_completion_tokens=15, temperature=temperature, messages=[{"role": "user", "content": PROMPT}], ) answers.append(resp.choices[0].message.content.strip().lower()) # lower() so "Elephant" and "elephant" count as one answer return answers if __name__ == "__main__": for temp in [0, 1, 1.8]: answers = sweep(temp) print(f"temp={temp}: {answers} -> {len(set(answers))} distinct")
2. Run it. Prompt choice matters here more than you'd expect: a narrowly-biased prompt like "name one primary color" stays stuck on whichever answer the model considers overwhelmingly likely (almost always "Red") even at the highest temperature, across all 10 tries — that's a real result if it happens to you, not a bug, it just doesn't make for an interesting lab. "Name an animal" has enough plausible alternatives that temperature actually has room to work.
   python chatbot/temperature_sweep.py

Notice "elephant" and "dolphin" still lead even at temp=1.8 — that's not noise, it's the real signal. The model's underlying probability distribution for "name an animal" isn't flat across every species; a handful of prototypical answers that show up constantly in training data (elephant, dolphin, lion, tiger) carry most of the probability mass, with thousands of other real animals each holding a tiny sliver of the rest. Temperature rescales that distribution so lower-probability tokens can win sometimes — which is why you see 4 distinct answers instead of 1 — but it doesn't flatten the distribution to uniform. If the two biggest peaks still hold most of the mass after rescaling, they'll still win most draws. "Higher temperature" means "more exposure to how peaked the real distribution already is," not "more randomness" in any general sense.

1. Create chatbot/streaming_vs_blocking.py and paste in both versions of the same call.
   import os, time from dotenv import load_dotenv load_dotenv() from anthropic import Anthropic client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"]) PROMPT = "Write a 300-word story about a lighthouse keeper." def call_blocking(show=False): start = time.time() client.messages.create( model="claude-haiku-5-5", max_tokens=500, thinking={"type": "disabled"}, messages=[{"role": "user", "content": PROMPT}], ) elapsed = time.time() - start if show: print(next(b.text for b in resp.content if b.type == "text")) return elapsed def call_streaming(show=False): start = time.time() ttft = None with client.messages.stream( model="claude-haiku-5-5", max_tokens=500, thinking={"type": "disabled"}, messages=[{"role": "user", "content": PROMPT}], ) as stream: for chunk in stream.text_stream: if ttft is None: ttft = time.time() - start if show: print(chunk, end="", flush=True) if show: print() return ttft, time.time() - start if __name__ == "__main__": for i in range(3): blocking_total = call_blocking() ttft, streaming_total = call_streaming() print(f"run {i+1}: blocking={blocking_total:.2f}s | " f"streaming ttft={ttft:.2f}s total={streaming_total:.2f}s") print("\n--- Now watch it. Blocking first: nothing appears until the reply is finished. ---") call_blocking(show=True) print("\n--- Streaming: words appear as they are generated. ---") call_streaming(show=True)
2. Run it. The three timed runs come first, then the script calls the model once more for you to watch: the blocking version prints nothing for several seconds and then drops the whole story at once, while the streaming version starts writing within about a second. Then look at the gap between blocking and ttft in the numbers — that gap is exactly how much sooner a streaming UI would show the user something.
   python chatbot/streaming_vs_blocking.py

1. Grab a real document — any article, README, or chapter, at least a few hundred words across multiple paragraphs — and save it as chatbot/long_doc.txt. Caching only kicks in above a minimum prompt length (512 tokens for Claude Haiku 5.5 and Sonnet 5.5, per Anthropic's prompt caching docs; shorter prompts are processed without caching and no error comes back), so a single short paragraph won't show anything; a ~700-word piece of ordinary prose clears that comfortably (English runs roughly 2 tokens per word).
2. Create chatbot/prompt_caching.py. It asks the same five questions about the document twice — once with cache_control set, once without — and logs every call through Module 0's log_call so the comparison lives in shared/calls.jsonl, not just stdout.
   import os, sys, time, pathlib from dotenv import load_dotenv load_dotenv() sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent.parent / "shared")) from llm_log import log_call DOC = pathlib.Path(__file__).parent.joinpath("long_doc.txt").read_text() QUESTIONS = [ "What is the main topic of this document?", "Summarize it in one sentence.", "What is the second paragraph about?", "Does it mention any numbers? Which ones?", "What tone does it use?", ] def ask(cached): from anthropic import Anthropic client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"]) system = [{"type": "text", "text": DOC}] if cached: system[0]["cache_control"] = {"type": "ephemeral"} label = "claude-haiku-5-5-cached" if cached else "claude-haiku-5-5-uncached" for q in QUESTIONS: start = time.time() resp = client.messages.create( model="claude-haiku-5-5", max_tokens=100, thinking={"type": "disabled"}, system=system, messages=[{"role": "user", "content": q}], ) elapsed_ms = (time.time() - start) * 1000 cost = (resp.usage.input_tokens / 1_000_000 * 0.10 + resp.usage.output_tokens / 1_000_000 * 0.50) log_call("anthropic", label, resp.usage.input_tokens, resp.usage.output_tokens, elapsed_ms, cost) print(f" cache_read={resp.usage.cache_read_input_tokens:5} " f"cache_write={resp.usage.cache_creation_input_tokens:5} {elapsed_ms:.0f}ms") if __name__ == "__main__": print("--- without caching ---") ask(cached=False) print("--- with caching ---") ask(cached=True)
3. Run it, then pull the two batches out of the log and compare total cost.
   python chatbot/prompt_caching.py python -c " import json rows = [json.loads(l) for l in open('shared/calls.jsonl')] for label in ['claude-haiku-5-5-uncached', 'claude-haiku-5-5-cached']: batch = [r for r in rows if r['model'] == label] print(label, 'total cost:', sum(r['cost_usd'] for r in batch)) "

1. Create chatbot/semantic_cache.py. It embeds a small seed set of question/answer pairs, then for any new question finds the closest seed by cosine similarity (plain Python — no numpy needed for three vectors) and returns its answer if the similarity clears a threshold.
   import os, math from dotenv import load_dotenv load_dotenv() from openai import OpenAI client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) def embed(texts): resp = client.embeddings.create(model="text-embedding-3-small", input=texts) return [d.embedding for d in resp.data] def cosine_similarity(a, b): dot = sum(x * y for x, y in zip(a, b)) na = math.sqrt(sum(x * x for x in a)) nb = math.sqrt(sum(x * x for x in b)) return dot / (na * nb) SEED = { "What is the capital of France?": "Paris.", "How many legs does a spider have?": "Eight.", "What's the boiling point of water in Celsius?": "100 degrees.", } seed_questions = list(SEED.keys()) seed_vecs = embed(seed_questions) def lookup(question, threshold): q_vec = embed([question])[0] best_score, best_q = 0, None for sq, sv in zip(seed_questions, seed_vecs): score = cosine_similarity(q_vec, sv) if score > best_score: best_score, best_q = score, sq if best_score >= threshold: return SEED[best_q], best_score, best_q return None, best_score, best_q
2. Add a test set deliberately mixing three kinds of question — true paraphrases of a seed, questions on the same topic as a seed but asking something different, and fully unrelated questions — then sweep a few thresholds and print every score.
   TEST_QUESTIONS = [ # paraphrases -- should hit "Which city is the capital of France?", "What's France's capital city?", "How many legs do spiders have?", "What temperature does water boil at, in Celsius?", # same topic, different question -- should NOT hit "What is the population of France?", "How many eyes does a spider have?", "What's the freezing point of water in Celsius?", # unrelated -- should NOT hit "How do I bake a chocolate cake?", "What's the capital of Japan?", ] if __name__ == "__main__": for threshold in [0.95, 0.85, 0.75, 0.6]: print(f"--- threshold={threshold} ---") hits = 0 for q in TEST_QUESTIONS: answer, score, matched = lookup(q, threshold) status = f"HIT (matched {matched!r})" if answer else "miss" print(f" {score:.3f} {q!r} -> {status}") if answer: hits += 1 print(f" {hits}/{len(TEST_QUESTIONS)} hits\n")
3. Run it and read every threshold's printout, not just the hit count — the bug you're hunting for is a same-topic question matching a seed it shouldn't.
   python chatbot/semantic_cache.py