SillyTavern beat us on cache hits, 91.5% to 46.5%. Half the fix was copying its defaults.

In late July we put our own app on a bench next to SillyTavern and made the rules deliberately paranoid. Same model (deepseek-v4-flash), same API key, the same three character cards imported on both sides, user messages byte-identical across every run, and every single call routed through one metering gateway so neither side could fudge a number. Four contestants: stock SillyTavern, a modded SillyTavern loaded the way the community actually runs it, our normal chat mode, and our agentic roleplay mode. Three independent repeats per cell.

We built the bench to answer “where exactly are we behind SillyTavern.” The first thing it nailed to the wall was us.

The number that hurt

The scouting run used one card (Nie Xiaoqian, a classic Chinese ghost-story heroine) and fifteen fixed turns. Stock SillyTavern: 88.9% cache hit rate. Us: 27.7%.

Why that number is money: DeepSeek caches any prefix that is byte-identical to a previous request and bills hits at $0.0028 per million tokens against $0.14 for misses. Two cents on the dollar. Character chat resends the system prompt, the card, and the entire history every turn, which makes it about the most cache-friendly workload in existence. Across the full matrix, the same nine 15-turn sessions cost stock SillyTavern $0.0241 and cost us $0.0664. Same model, same key, 2.8x.

The absolute figures are cents, because fifteen turns is short and v4-flash is cheap. The multiplier is what scales: longer sessions, pricier models, and, since a lot of our users bring their own key, somebody else’s bill.

Full-matrix hit rates, three cards by four contestants by three repeats: stock SillyTavern 91.5%, modded 74.6%, our normal mode 46.5%, our agentic mode 42.3%. The full matrix also disabled SillyTavern’s built-in summarize side-calls per best practice, which is why stock scores even better here than in the scouting run. It deserved the fairer setup.

Three lorebook structures, one variable

46.5% was an average hiding something worse. Same build, same pipeline, three cards: 22.6%, 27.7%, 74.3%.

The only structural difference between those cards is the lorebook. Two of them (an original deep-lore card whose world exists nowhere in any training data, plus the ghost-story card) use keyword-triggered entries exclusively. The third, a modern slice-of-life card, carries two constant entries that sit in the prompt every turn. Constant card: 74.3%. All-triggered cards: 22.6% and 27.7%. Case closed. Triggered entries walk in and out of the prompt head as keywords come and go, DeepSeek matches prefixes strictly from token zero, and one changed byte near the top reprices everything after it.

The worst shape was the deep-lore card: eight to ten turns out of fifteen missed the cache entirely, from the very first token. Its per-turn hit rates read like an ECG. 0, 0, 91, 14, then eight turns flat between 11 and 15, one spike to 94, back down to 8. An entry arrives, the cache dies. It leaves, the cache dies again.

Half the fix was copying SillyTavern’s homework

SillyTavern is open source and we tested v1.18.0, so once we knew what to look for, we read how it behaves. The fix came out as three alignments to things SillyTavern ships correctly out of the box, plus one mechanism of our own.

First, recursive scanning. Lorebook entries can reference each other, and SillyTavern enables recursion by default, so cross-referencing entries all arrive on turn one and the prompt head stays put afterwards. Our default was off. Entries trickled in one turn at a time, and every arrival broke the cache once.

Second, the “0 means unlimited” convention. Setting max recursion steps to 0 means no limit in SillyTavern’s semantics. Our old code read 0 as “do not recurse.” Plainly our bug.

Third, and nastier: whole-word matching versus CJK text. SillyTavern’s word-boundary check can match Chinese, Japanese and Korean; our old implementation used a boundary rule that never fires between CJK characters. In practice, a Chinese card with “match whole words” enabled had its entire lorebook silently dead: not one entry could ever trigger. Nothing to do with caching, strictly worse than the caching bug, dug up by the same benchmark.

The mechanism of our own is what we call entry residency: once an entry has triggered, it stays in the prompt for the rest of the session. SillyTavern does not have this either. It treats the underlying churn problem directly; SillyTavern’s factory layout just makes the churn less lethal. Residency sounds like it should bloat the prompt. Measured on the ghost-story card over the same fifteen turns, total prompt volume actually went down, 68,051 tokens after the fix against 71,715 before, because repeatedly missing the cache costs more than keeping entries resident.

The same benchmark run flagged two money issues unrelated to caching, fixed in the same batch. Our “new session” form defaulted to a paid summary generation when no cached summary existed; paid actions are now opt-in. And DeepSeek’s thinking output had no off switch in our app; it does now.

Verification, including the run we threw away

On the card that embarrassed us: 27.7% to 89.9%. The per-turn curve went from ECG to flatline, 92 to 98%, with a single dip to 19% on turn six, where a probe question pulls a new entry into the prompt for the first time. First arrivals miss by design; residency then pins them, and turn seven is back at 93%. We diffed consecutive request bodies byte by byte: from turn two onward, every request is a strict prefix of the next plus the new turn. That is what cache-friendly looks like.

One verification run got voided, and it is worth telling. To simulate an upgraded legacy install we pre-wrote an old config file onto the test device, got one wrapper level of the format wrong, and the app silently ignored it, so that run actually executed as a fresh install. When we noticed, we did not quietly count it as another 89.9%. We relabeled it as an independent repeat of the fresh-install run (89.3%, which usefully proves the first result was no fluke) and rebuilt the legacy scenario properly.

Then the full re-run, 18 cells, three repeats each: normal mode 46.5% to 89.9%, agentic mode 42.3% to 88.3%. Cost down 58% and 63%. The spread between best and worst card collapsed from 51.7 points to 5.8. Full-miss turns on the deep-lore card went from 8-10 per session to zero or one. Upstream time-to-first-token dropped 5 to 65 ms as a side effect: fewer misses, faster prefill.

Two comparisons for context. Stock SillyTavern still sits at 91.5%, 1.6 points ahead, and the gap has a concrete explanation: our scripts include probe questions that deliberately dodge trigger keywords. On our side the app still retrieves the right entry into the prompt (a retrieval story for another post), and each first arrival is a designed miss. On SillyTavern’s side the entry simply never shows up: kind to the cache, unkind to the answer. Modded SillyTavern at 74.6% now trails our fixed builds by 14 to 15 points. The popular preset-plus-plugins stack pushes prompt volume to 2.46x stock and drops the hit rate 17 points. That is not a dunk. The stack’s design goals are style and gameplay, not billing. But the plugin tax is real and worth knowing about before you install.

And the “special benchmark build” question, answered up front: the pre-fix baseline and the post-fix package are both pinned by version code and APK hash, and the post-fix package was built from a clean source snapshot. It is the code that ships.

The honest ledger

Existing users do not get 89.9% yet. The recursion default flip only reaches fresh installs; old installs have the old default baked into their config, so upgrades run residency alone, measured at 73.1%. That is 2.6x the broken baseline and 17 points short of fresh, and the shortfall is entirely the cross-referencing entries that cannot be pulled in on turn one. Whether to migrate existing configs is a separate product decision we have not made.

Cost figures are mechanical conversions at list price, meant for comparing groups rather than forecasting your bill. Every cell ran three times. The 36-cell baseline matrix was independently recomputed end to end with zero discrepancies, and the 18-cell re-run went through the same audit tooling.

The fix itself caused one regression, and the same benchmark caught it: residency drained the candidate pool of our agentic mode’s lore retrieval, and the deep-lore card’s 12-question lore quiz dropped from 12/12 to 9/12. Root-caused and fixed the same night; the retest shows retrieval restored and the cache holding. Once you have a bench, you have a regression net.

And we will not claim “cheaper than SillyTavern.” Stock SillyTavern at 91.5% is still the cache benchmark of this test, earned by a factory prompt layout that is append-only by construction. What we can claim: same model, same price card, from 45 points behind to even. The discipline that got us there (stable content first, arrived entries pinned) now ships as the default.

There is a residency switch in settings now, on by default. You can check our math without trusting us: run fifteen turns on a trigger-heavy card and watch the cached-tokens field in the per-request log. It should sit near 90% and dip exactly once per newly arrived entry. If it starts dipping in a pattern we did not describe here, that is a churn shape we have not met yet. The bench is still standing; tell us.