3 posts
An AI API cache hit is not a response cache that hands back a stored answer. It reuses the KV computed for the start of the input. This post works out which requests hit and how high the hit rate must be to break even once write costs are counted, then compares an agent and a chatbot using Claude Code measurements and a Slack chatbot's production logs. In the agent's sessions, 97.6% of input tokens were cache reads.
I attached Prometheus, Grafana and Langfuse to a LiteLLM proxy and ran the same mock traffic through all of them for 14 minutes. The built-in Admin UI goes as far as who spent how much; time series and alerts need Grafana, and request bodies, sessions, traces and scores need Langfuse. Here is where the Admin UI is enough and the order to add the rest.
stdio MCP servers duplicated per session added up to 169 processes and 6.7GB of RSS. I built a proxy to share one process across sessions — and finished with a bigger question. Do these servers still need to be MCP at all?