How it works

parry.install() replaces create on the SDK’s resource classes with a thin hook. On every call the hook finds the active guard and runs the request through seven stages. With no active guard, the hook calls the original method and nothing else happens.

Trace a request

Trace a cache missTrace a cache hit

1. 1
   Hook python/parry/_patch.py

   The patched create() finds the active guard. If there is none, it calls the original method directly.

2. 2
   Cache key src/cache.rs · src/tokenize.rs

   Semantic kwargs are canonicalised and hashed with blake3, namespaced by provider, endpoint and base URL.

3. 3
   Cache lookup moka LRU → SQLite (optional)

   A hit is validated back into the SDK type and returned. Nothing else runs.

4. 4
   Budget pre-check src/cost.rs

   Spend so far plus the prompt's estimated cost must stay under max_cost_usd.

5. 5
   Rate limit src/rate_limit.rs (governor)

   Take a slot from the provider:model bucket, or sleep for exactly as long as the limiter says.

6. 6
   Provider call + retry src/retry.rs

   The original SDK method runs. Transient failures back off with jitter and take a fresh rate-limit slot.

7. 7
   Record & store ledger + cache

   Usage is priced into the ledger and the response is cached. Streams do this when they finish.

Pick a scenario to step through it.

Why hook the class instead of the client?

Clients get created everywhere: in your code, inside LangChain and LlamaIndex, and inside helper libraries. Patching the resource class, for example openai.resources.chat.completions.Completions, catches all of them, including clients created before install() ran.

The hooks are inert without a guard. Importing parry changes nothing. Only install(), a with Parry(): block or parry.wrap(client) turns them on.

Hooked methods:

SDK	Classes
openai	Completions, AsyncCompletions, Responses, AsyncResponses
anthropic	Messages, AsyncMessages, beta.Messages, beta.AsyncMessages

install() is idempotent. Hooks are recognised by a __parry_original__ marker, so calling it twice never wraps anything twice, and uninstall() restores the exact original functions.

Rust core, Python shell

• Directorypython/parry/

  • __init__.py public API: install, wrap, bypass, stats
  • _patch.py SDK hooks
  • _core.py the Parry guard and the per-call pipeline
  • _adapters.py per-endpoint knowledge: model, usage, stream assembly
  • _stream.py in-place stream metering
  • _storage.py optional SQLite
  • Directoryintegrations/ LangChain, LlamaIndex

    • …
• Directorysrc/ Rust, compiled to parry._parry_native

  • cache.rs blake3 key + moka LRU/TTL
  • rate_limit.rs governor token buckets
  • cost.rs pricing table + ledger
  • retry.rs backoff + error classification
  • tokenize.rs canonical JSON, tiktoken counts

Rust holds everything that has to be fast, thread-safe and free of Python objects: the cache (values are strings), the token buckets, the spend ledger, the retry policy, request canonicalisation and token counting. Every type is Sync, so one guard is safely shared across threads and asyncio tasks. Hashing and tokenising release the GIL.

Python handles everything that touches SDK objects: swapping methods, serialising pydantic responses, metering streams, raising exceptions, and sleeping. The rate limiter’s try_acquire returns “seconds to wait” instead of blocking, so sync code can call time.sleep and async code can await asyncio.sleep on the same primitive.

Special cases

• .with_raw_response: detected through the X-Stainless-Raw-Response header the SDK adds. On a cache hit parry returns a CachedRawResponse with .parse() and .headers, which is what LangChain uses. Its parse() is a coroutine exactly when the installed SDK’s own is.
• Streams: never served from cache. parry instruments the SDK’s own Stream object in place, so type(stream) is openai.Stream still holds. See Streaming.
• Positional arguments: SDK create() methods are keyword-only. A call with positional arguments is passed straight through.

Not intercepted in v0.1

chat.completions.parse(), responses.parse(), the messages.stream() helper, .with_streaming_response, embeddings, batches, files, images and audio. See Failure modes.