40%
Cutting inference cost on a live legal-AI platform
- The problem
- A production legal assistant serving 80,000+ users was routing every query, trivial lookups included, through a frontier model, with no caching layer and retrieval returning far more context than answers required.
- What we did
- Introduced prompt caching across the highest-volume paths, and built a routing layer that classifies each query and sends the simple majority to lighter models, keeping the frontier model for work that genuinely needs it.
- Result
- A 40% reduction in inference spend, with no measurable drop in answer quality.