v0.1.0 Rust core · Python API · MIT

Guard every
LLM call with
one line.

Your OpenAI and Anthropic calls, cached

parry puts caching, per-model rate limits, cost tracking, budgets and retries in front of the SDKs you already use. There's nothing to configure and no call site to rewrite.

$ pip install llm-parry
Read the quickstart How it works
Sync & asyncStreamingLangChain & LlamaIndex
app.pypython
import openai
import parry

parry.install()          # the only line you add

client = openai.OpenAI()
answer = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hi!"}],
)
print(parry.stats().summary())
terminal
Works with the stack you already haveOpenAIAnthropicLangChainLlamaIndexsync · async · streaming · raw responses · Responses API
The problem

LLM apps fail in boring,
expensive ways.

None of these are hard problems on their own. Together they turn into the same glue code copied into every project, and it usually gets skipped until something breaks.

01

Paying twice for the same answer

Re-running a notebook, an eval or a retry loop sends the identical prompt again, and you pay again.

Identical requests are answered from cache, for free.
02

429 storms

Batch jobs hit the provider's limit, everything fails at once, and naive retries make it worse.

Token buckets space requests out before the provider says no.
03

Surprise invoices

An agent loops, a prompt grows, and you find out at the end of the month.

Live spend by model, and a hard stop at your budget.
The aim

Make the safe path the default path. No config files, no external services, no new SDK to learn. Add one line, and remove it when you no longer need it.

How it works

Six steps, every call,
in microseconds.

parry.install() hooks the SDK's own create() methods, so every client in the process is covered, including the ones LangChain and LlamaIndex create for you.

Your codecalls create() as usual
Cacheblake3 key, LRU + TTL
Budgetpre-flight cost check
Rate limittoken bucket per model
Provideroriginal SDK call, with retry
Ledgerrecord cost, fill cache
Waiting for a request…
What you get for free

Production guard rails,
no configuration.

Every feature is on by default and has defaults that suit most apps. You can tune any of them when you need to.

Response caching

Identical requests come back from a bounded LRU in about 10 µs. The key covers every field that changes the answer, plus the base URL.

cache_ttl=3600Learn more →

Per-model rate limits

One token bucket per provider and model, with defaults set to the published tier-1 limits. parry waits for a slot instead of collecting 429s.

set_rate_limit("gpt-4o", 60)Learn more →

Cost tracking

Exact token usage priced from a dated table and broken down by model. Add SQLite for daily and lifetime totals.

stats.total_cost_usdLearn more →

Hard budgets

A request that would push spend past the limit raises BudgetExceeded before it leaves your process. Cached answers stay free.

max_cost_usd=10Learn more →

Retries that behave

Exponential backoff with jitter. Honours Retry-After, retries 429, 5xx and timeouts, and never retries a 401.

max_attempts=3Learn more →

Streaming & async

Streams reach you untouched and are metered as they flow. A finished stream fills the cache for the next call.

stream=TrueLearn more →
Before / after

Delete the glue code.

The homemade version covers one client, one model and one code path. parry covers every client, every model and every code path, including async, streaming and framework calls.

before.py25 lines · 1 code path
cache, spend = {}, 0.0
bucket = TokenBucket(rate=500 / 60)

def ask(msgs, model="gpt-4o"):
    global spend
    key = sha256(dumps([model, msgs]))
    if key in cache:
        return cache[key]
    if spend > BUDGET:
        raise RuntimeError("over budget")
    for attempt in range(5):
        time.sleep(bucket.wait_time())
        try:
            r = client.chat.completions.create(
                model=model, messages=msgs)
            break
        except (RateLimitError, APITimeoutError):
            time.sleep(2 ** attempt + random())
    u, p = r.usage, PRICES[model]
    spend += (u.prompt_tokens * p[0]
              + u.completion_tokens * p[1]) / 1e6
    cache[key] = r
    return r

# ...again for async, streaming, Anthropic
after.py2 lines · every code path
import parry

parry.install(max_cost_usd=10)

r = client.chat.completions.create(
    model="gpt-4o", messages=msgs)
Cache with TTL and size boundPer-model token bucketsBackoff, jitter, Retry-AfterSpend per model, hard budgetAsync, streaming, LangChain, LlamaIndex
Benchmarks

Numbers, not adjectives.

The hot path is Rust: hashing, the LRU, the token buckets and the pricing ledger. These numbers are measured with the real OpenAI SDK over an in-memory transport, so only parry's own cost is counted.

10µs

per cache hit, with no HTTP at all

+65µs

added to a cache miss

0files

of configuration required

1line

to install it, and one to remove it

Plain SDK call (baseline)604 µs
With parry, cache miss669 µs
With parry, cache hit10 µs
Linux x86_64, Python 3.13.5, openai 2.54.0, median of 5 batches of 1000 calls.
Set up

Working in under five minutes.

Install

One abi3 wheel per platform covers Python 3.10 to 3.14. It has no required dependencies.

pip install llm-parry

Add one line

Put it at startup. Clients created before or after it are covered.

import parry
parry.install()

Watch it work

Check hits, spend and throttling at any point.

print(parry.stats().summary())
# cache  2 hits / 1 misses (67%)
# spend  $0.000006 this session
parry.install()Every client in the process
with Parry() as g:One block, thread- and task-local
parry.wrap(client)A single client object
with parry.bypass():Turn it off for one call

Ship guard rails,
not glue code.

Start with the quickstart, or see every option in the reference.

Get started Browse the docsStar on GitHub