The challenge
What was at stake
An enterprise software group wanted to add AI automation across 27 legacy internal applications, but constant frontier-model routing had pushed monthly AI API spend to $41,000.
Their existing infrastructure also could not support always-on GPU workloads, and a full rewrite of upstream applications was not acceptable.
The approach
What we built
We built a hybrid compute routing layer that classifies prompt complexity, offloads lightweight inference to existing on-premise CPU hardware, routes heavier tasks to optimized GPU tiers, and uses frontier APIs only when the workload requires them.
Inside the system
Key capabilities
Hybrid CPU/GPU Routing
Prompt complexity classification routes lightweight classification and extraction tasks to quantized local models while reserving GPU and API calls for higher-complexity reasoning.
Non-Invasive Legacy Integration
Light REST/gRPC middleware wrappers connect 27 existing internal applications to the new routing layer with zero changes to upstream application code — full deployment completed in 11 working days.
Real-Time Cost Monitoring Dashboard
Live token consumption and per-tier cost tracking with configurable budget guardrails — automatically escalating or throttling routing decisions when monthly spend approaches defined thresholds.
Under the hood
Technology
- llama.cpp inference engine
- Mistral-7B Q4_K_M
- Phi-3-mini Q5
- Mixtral-8x7B Q4 on spot GPU
- custom prompt complexity classifier
- FastAPI routing middleware
- OpenAI/Anthropic API fallback
The outcome
What changed
Monthly AI API spend reduced from $41,000 to $11,500 (72% reduction) within 60 days of deployment. 68% of all AI requests now processed entirely on existing on-premise hardware. Average P95 response latency for CPU-tier tasks: 1.1 seconds. Deployment completed in 11 working days with zero changes to existing application code. ROI on the engagement achieved in under 4 months.
- $41K → $11.5K
- Monthly cost
- 68%
- Local offload rate
- 11 Days
- Deployment time