Introduction

parry docs

Guard rails for every LLM call

Caching, per-model rate limits, cost tracking, hard budgets and retries for the OpenAI and Anthropic SDKs, with one line and no rewrites.

$ pip install llm-parryQuickstart →How it works

import parry
parry.install()   # every OpenAI / Anthropic call is now guarded

parry sits between your code and the SDKs you already use. It doesn’t wrap your client in a new API, ask you for a config file, or need Redis. It hooks the SDK’s own create() methods, so existing code keeps working, including code inside LangChain and LlamaIndex.

The aim

LLM apps tend to fail in the same boring, expensive ways. They pay twice for identical prompts, hit 429 storms during batch jobs, and run up bills nobody noticed. The fixes are well known but tedious, so every project rewrites the same glue code, or skips it.

parry’s aim is to make the safe path the default path:

• Zero config. One line at startup, with sensible defaults for every feature.
• No rewrites. Your call sites, types and frameworks stay exactly as they are.
• No lock-in. Remove the line and you’re back on the plain SDK. There’s no new client to migrate off.
• No infrastructure. Everything runs in-process. Persistence is an optional SQLite file.

What you get

Response caching

Identical requests come back from a bounded LRU in about 10 µs, for $0. Caching →

Per-model rate limits

A token bucket for each provider:model, with tier-1 defaults. Rate limiting →

Cost tracking

Exact usage priced from a dated table, broken down per model and per day. Cost tracking →

Hard budgets

BudgetExceeded is raised before the request that would cross your limit. Budgets →

Retries that behave

Backoff with jitter, Retry-After, and no retries on 4xx. Retries →

Streaming & async

Streams pass through untouched and are metered. A finished stream fills the cache. Streaming →

What it covers

SDK / framework	Covered by install()
openai	chat.completions.create, responses.create, sync and async, stream=True, .with_raw_response
anthropic	messages.create, beta.messages.create, sync and async, stream=True, .with_raw_response
LangChain	ChatOpenAI, ChatAnthropic (they call the SDKs above), plus ParryCallback for anything else
LlamaIndex	OpenAI, Anthropic LLMs, plus ParryHandler for event-level guarding

Is parry right for you?

parry is built for a single process: scripts, notebooks, eval runs, workers and single-replica services. If several processes or hosts need to share a cache and rate limits, put a shared cache and an API gateway in front of the provider instead. See the FAQ.

Next steps

QuickstartFrom pip install to your first cache hit in five minutes.

How it worksWhat happens, step by step, on every call.

ConfigurationEvery option, with its default.

IntegrationsLangChain and LlamaIndex, globally or per chain.