Foreverse Research · Benchmark
Foreverse vs SillyTavern:
an instrumented four-way benchmark
Start with the first problem this benchmark caught — it was ours: Foreverse's chat cache hit rate measured 46.5%, meaning the same 15-round session cost 2.8× what bare SillyTavern paid. We traced and fixed the root cause the same night; re-verification reached 89.9%. The full four-way data, method, and limits follow.
Data collected: 2026-07-23 to 07-24 · Page published: 2026-07-24
Same conversation, replayed on both machines
The benchmark's ground rule: same model, same API key, same byte-identical message script. Below, that script (adapted) replays on both machines in sync — watch each round's cache hits, then the bill.
Full disclosure first: before the fix, the losing curve was ours — 46.5% cache hits vs 91.5% for bare SillyTavern, so the same 15-round chat cost 2.8× as much. We traced and merged the fix that night; re-verification reached 89.9%. Flip the switch back to “pre-fix” any time — the original curve stays on the record.
Ready · auto-plays when scrolled into view
Cache hit rate · measured per round
Full 15-round aggregate (main matrix, 3 cards × 3 repeats): bare SillyTavern 91.5% · Foreverse 46.5% pre-fix (2026-07-23) → 89.9% post-fix (2026-07-24)
Measured total cost, same conversations (9-cell matrix total)
Cumulative prompt tokens (single card, full 15 rounds, measured)
of which cache hits: ≈0 (pre-fix)
Number sources and scope
- Per-round hit rates = the first 8 of 15 measured rounds on the Nie Xiaoqian card: bare tavern and “pre-fix” from the 2026-07-23 baseline, “post-fix” from the 2026-07-24 re-verification; that single-card session aggregates 88.9% / 27.7% / 89.9% (fix report §4.1).
- Panel aggregates 91.5% / 46.5% → 89.9% and costs $0.0241 / $0.0664 (2.8×) → $0.0278 (−58%) = main-matrix totals over 3 cards × 3 repeats, 9 cells (W2 2026-07-23 / W2.5 2026-07-24, final report §3).
- Cumulative prompt tokens 75,303 / 71,715 / 68,051 and cached portions 19,840 / 61,184 = the full measured 15-round single-card session (fix report §4.1).
- Dialogue text is adapted from the benchmark's message script (first 8 turns, same semantics; Nie Xiaoqian card, a public-domain Liaozhai setting). In the benchmark each side's replies came from its own system; here they are unified for synchronized replay and score nothing.
- The simultaneous dip at round 6 = a by-design first-entry miss when round 5's question pulls a new lore entry into context — symmetric on both sides. The pre-fix curve's occasional full-hit round (e.g. 91% at round 3) means the activation set happened not to change that round — sawtooth, not recovery (fix report §4.2, W2 report §4.1).
- Post-fix structural evidence: from round 2 through 15, every request is a strict prefix of the previous one plus the new turn (request-body level, fix report §4.2).
- The Smart Play arm (42.3% → 88.3%) is not one of the two machines here; the full four-way data is in the tables below.
Headline findings
- Our own cache hit rate was 46.5% before the fix vs 91.5% for bare SillyTavern — same 15-round session, 2.8× the cost (measured 2026-07-23). Root cause: the position where we injected lorebook entries drifted every round, breaking the provider's ~2%-priced repeated-prefix discount. Fixed and merged the same night; re-verified at 89.9% / 88.3%, matching bare ST, with 58-63% lower cost (2026-07-24).
- Bare SillyTavern hits 91.5% cache discount with zero plugins, straight out of the box — its prompt structure is naturally stable. That number was the bar we fixed our own app against, not something we beat.
- Fully-modded SillyTavern (the community's most popular preset + helper plugin + table-memory combo) pays a “plugin tax”: 2.5× prompt size, 2.8× the total cost of bare ST, and 12 of 135 rounds (8.9%) returned empty upstream replies that were still billed (2026-07-23). That's the cost of the configuration stack, not a flaw in SillyTavern itself.
- Deep-lore probe, 12 questions on a fully original worldbook card, deliberately avoiding trigger keywords: Foreverse Smart Play answered 12/12 with zero extra paid calls; bare ST 1/12, modded 2/12, Foreverse standard 5/12 (2026-07-23). The cost belongs in the same sentence: Smart Play sessions run about +35% (+18% measured after the cache fix).
- 27-question memory exam after injecting a real 300-round chat (single run): bare ST scored 16 — steadiest on recent facts and honestly said “I don't know” on 3 of 4 never-mentioned questions; Smart Play scored 13 and was the only setup to answer facts from ~270 rounds back; Foreverse standard scored 10 but fabricated all 4 never-mentioned answers — a real product gap this benchmark caught in our own app (2026-07-24).
- Blind-review leaning from calibrated AI judges: Smart Play > bare ST > Foreverse standard > modded ST (16:0 against our own standard mode, three judges agreeing). But the judges' qualification exam also produced a case where every judge was confidently wrong, so until 30 human spot-reads finish, this stays a leaning — not a conclusion (2026-07-24).
- Group-chat crosstalk special (same three-character group × byte-identical 12-round script × 2 repeats): all four setups landed zero crosstalk on disk. The paths differ, though — our model actually attempted whole-turn impersonation 21 times at the generation layer, every one intercepted before persisting; on @-mention dispatch we scored 11/12 vs SillyTavern's 7/12 (2026-07-24).
- Instruction-following special (six instruction groups, 24 judgment points × 2 repeats, machine first-pass + full human adjudication): Smart Play 94%, first of four; bare ST 90%, Foreverse standard 85%, modded ST 78%. Honest scope: single chat, single model — and our standard mode's weakness (it couldn't step out of character when asked) is baked into that number (2026-07-24).
- The whole benchmark used ~1,530 real AI calls, roughly $1.07 at list prices; ~952 judge calls ran on a separate channel (2026-07-23/24).
Method
- Four setups: A — bare SillyTavern (v1.18.0, stock config); B — fully-modded SillyTavern (the community's most popular preset + helper plugin + table memory, configured per best practice with every change logged); C — Foreverse standard; C+ — Foreverse Smart Play.
- Same model (deepseek-v4-flash), same API key, same machine, same character cards (chara_card_v2 imported on both sides), same byte-identical message scripts; all four streamed.
- Every AI call passed through one self-hosted transparent metering gateway recording token usage, cache hits, and time-to-first-byte. The ST side and the app side each kept an independent record; a cell only counts after all three ledgers agree.
- Main matrix = 3 cards × 4 setups × 3 independent repeats = 36 cells (pre-fix baseline), plus an 18-cell post-fix comparison. Long-chat and tabletop-tooling runs are single-run specials (n=1, marked as such).
- AI judges had to pass a grading exam first: ≥80% on 26 known-answer samples to earn a vote — 3 of 6 judges qualified. Every verdict was judged twice in both presentation orders; inconsistent pairs were discarded.
- Test dates: 2026-07-23 to 07-24. Every number can be recomputed from the per-call raw logs.
Core data
Cache hits and cost (15-round sessions, 3 cards × 3 repeats)
| Bare SillyTavern | Fully-modded ST | Foreverse standard | Foreverse Smart Play | |
|---|---|---|---|---|
| Cache hit rate · pre-fix (2026-07-23) | 91.5% | 74.6% | 46.5% | 42.3% |
| Cache hit rate · post-fix (2026-07-24) | — | — | 89.9% | 88.3% |
| Total cost, 9 cells · pre-fix | $0.0241 | $0.0676 | $0.0664 | $0.0898 |
| Total cost, 9 cells · post-fix | — | — | $0.0278 | $0.0328 |
| Time to first token (median) | 162ms | 220ms | 246→223ms | 235→203ms |
| Empty upstream replies (still billed) | 0/135 | 12/135 (8.9%) | 0/135 | 0/142 |
“Pre-fix” is the baseline where this benchmark caught our own defect; we keep it on the record. The fix shipped in a release build. Cost is a mechanical conversion at DeepSeek list prices, for between-group comparison only.
Deep-lore probe (original worldbook card, 12 questions avoiding trigger keywords)
| Bare SillyTavern | Fully-modded ST | Foreverse standard | Foreverse Smart Play | |
|---|---|---|---|---|
| Correct | 1/12 | 2/12 | 5/12 | 12/12 |
Smart Play's 12/12 has request-body-level evidence (the target lore entries verifiably entered the request) and used zero extra paid calls; the trade-off is +35% session cost (+18% measured post-fix).
Honest addendum: after the cache fix merged, a retest regressed to 9/12 — the cost-saving change had drained the retrieval candidate pool. The interaction was fixed the same night and re-verified. That's the second case of the benchmark feeding fixes back into the product.
Long-chat memory (real 300-round chat injected + 27 questions, single run, n=1)
| Bare SillyTavern | Fully-modded ST | Foreverse standard | Foreverse Smart Play | |
|---|---|---|---|---|
| Total score | 16/27 | 6/27 | 10/27 | 13/27 |
| Very-far questions (~270 rounds back) | 0/6 | 0/6 | 0/6 | 2/6 (only scorer) |
| Honest “I don't know” on never-mentioned facts | 3/4 | 1/4 | 0/4 (all fabricated) | 2/4 |
Our standard mode's 0/4 is the ugliest number in this run: it invented a blood type and a birthday that were never mentioned. Filed as a product gap. Modded ST's score of 6 includes 4 questions answered with blanks or raw jailbreak JSON.
What this benchmark caught in our own app
The worst benchmark is one that only reports the dimensions you win. This one caught six real defects in Foreverse, all fixed and shipped:
- The cache cliff (46.5% vs 91.5%) — lorebook injection position drifted every round; 89.9% after the fix.
- “New session” billed one paid summary call even with the recap box unchecked — default flipped so paid summaries are explicit opt-in.
- Long-chat auto-compaction randomly stalled on thinking models — reasoning could eat the whole output budget; fallback added.
- Lore recall truncated at 400 chars produced confident half-wrong answers, and option chips occasionally failed to render — both fixed.
- The cache fix and the retrieval feature stepped on each other, regressing the deep-lore score from 12/12 to 9/12 — fixed the same night, re-verified.
- Long-chat compaction summaries had no channel to preserve rules the user laid down — once a rule was compacted away, its survival was pure luck; the preservation checklist now carries a user-rules section.
What we learned from the SillyTavern ecosystem
- Its stock prompt structure is naturally append-only: 91.5% cache hits with nothing installed. Our fix essentially aligned with it, then added activation-set stability on top.
- Table memory's “structured output riding inside the reply” persists state far more reliably than separate tool calls — modded ST was the only setup that truly wrote its state ledger to disk in our tooling run. We've filed that pattern to borrow.
- A player-facing manual dice slot (/roll command, dice macros, quick buttons) is a complete tabletop ritual we currently don't have.
How to cite this page
Foreverse Research, “Foreverse vs SillyTavern: an instrumented four-way benchmark,” 2026-07. https://foreverse.app/research/foreverse-vs-sillytavern
Honest limits
- Single model: everything above is deepseek-v4-flash; magnitudes may differ on other models.
- Long-chat and tooling runs are single-run specials (n=1): large between-group gaps are credible, single-digit gaps need repeat runs.
- Cost is a mechanical conversion at list prices for between-group comparison — not a prediction of your bill.
- Blind review has not passed human final adjudication: the judges passed a qualification exam, but that exam also produced unanimous wrong calls, so 16:0 stays a leaning.
- Fully-modded SillyTavern isn't designed to win on cost or caching — large presets optimize for style control. We record its costs honestly and say honestly that this isn't its race.
- Existing users who upgrade measure about 73.1% cache hits, not 89.9% — closing that gap touches stored configs and is a separate product decision; the page numbers are for fresh installs.
- SillyTavern is an excellent open-source project the community loves. No setup won everything: bare ST wins on stock structure and honest recent memory, modded ST on state persistence, Foreverse standard (post-fix) on cost-effectiveness, Smart Play on retrieval and long-range memory.