Quickstart

This guide builds a tiny intent classifier: render a prompt, call a model, extract and validate the answer, handle failures, and lock the behaviour in with a snapshot test.

1. Install

   pip install llm-beacon pytest

2. Render the prompt

   app.py

   from beacon import template
   
   PROMPT = """Classify the customer's message.
   Reply with JSON: {{"intent": "cancel" | "refund" | "question", "confidence": 0..1}}
   <message>{message}</message>"""
   
   def build_prompt(message: str) -> str:
       return template(PROMPT, message=message)

   {{ and }} are literal braces. template XML-escapes message, so a user can’t close </message> and write instructions outside the tag. A missing or misspelled variable raises immediately instead of reaching the model.

3. Extract and validate the answer

   app.py

   from beacon import extract_json
   
   INTENT = {
       "type": "object",
       "properties": {
           "intent": {"type": "string", "enum": ["cancel", "refund", "question"]},
           "confidence": {"type": "number", "minimum": 0, "maximum": 1},
       },
       "required": ["intent", "confidence"],
   }
   
   def classify(message: str, llm) -> dict:
       raw = llm(build_prompt(message))          # any client: OpenAI, Anthropic, local…
       return extract_json(raw, schema=INTENT)

   extract_json handles markdown fences, preamble, trailing prose and citation markers like [1]. It accepts strict JSON only: a trailing comma or a truncated object raises MalformedOutputError with the exact position, rather than a silent guess.

4. Handle failures deterministically

   from beacon import MalformedOutputError, SchemaValidationError
   
   def classify_with_feedback(message: str, llm) -> dict:
       prompt = build_prompt(message)
       raw = llm(prompt)
       try:
           return extract_json(raw, schema=INTENT)
       except (MalformedOutputError, SchemaValidationError) as err:
           # err.code and str(err) are stable, so the model always gets the same correction.
           raw = llm(f"{prompt}\n\nYour previous answer was invalid: {err}\nReply again.")
           return extract_json(raw, schema=INTENT)

5. Snapshot-test it

   Save a real model response to tests/fixtures/cancel.txt, then:

   tests/test_classify.py

   from pathlib import Path
   from beacon.snapshots import assert_snapshot
   from app import classify
   
   FIXTURES = Path(__file__).parent / "fixtures"
   
   def test_cancel():
       recorded = (FIXTURES / "cancel.txt").read_text()
       result = classify("I want to cancel", llm=lambda prompt: recorded)
       assert_snapshot("intents/cancel", result, input=recorded)

   pytest                              # first run writes snapshots/intents/cancel.yaml
   pytest                              # later runs compare byte for byte
   python -m beacon.snapshots update   # accept an intended change

Tip

Commit the snapshots/ directory. A change in how your code parses, validates or routes model output then shows up in code review as a diff, and CI fails until someone accepts it.

Where next

Core conceptsThe mental model behind every block.

extract_json in depthThe exact algorithm, selection rules and every error.

Snapshot testingModes, the pytest fixture and CI.

IntegrationsLangChain, LlamaIndex, Pydantic AI, DSPy, OpenAI.