Open source · AI & LLM

Model Fallback Simulator

Exercise fallback models for quality, latency, cost and error behavior.

v0.1.0 · Node.js 22+ · MIT

Browse the public repository · View releases

Replay a local model-response fixture against explicit fallback order, required capabilities, retries, quality and cumulative latency/cost budgets. This is a reporter over recorded scenarios, not a router: it makes no provider call, opens no socket, sends no request and changes no account.

This walkthrough uses the tool's public README and checked-in example files. Run the command from a repository checkout with Node.js 22+; inspect the source before using it on your own files.

Run the checked-in example

node bin/model-fallback-simulator.mjs --root examples/clean --scenario scenario.json --json
node bin/model-fallback-simulator.mjs --root examples/denied --scenario scenario.json --json
node bin/model-fallback-simulator.mjs --root examples/uncertain --scenario scenario.json --json
npm run check

Read the result

The examples exit 0 (capable fallback selected), 1 (budget denied before a fallback can run), and 2 (missing quality evidence). The CLI writes the JSON report to stdout; without --json it adds a short summary on stderr. Bad configuration exits 2 with empty stdout; an unreadable/invalid named input exits 2 with an incomplete JSON report. --help lists every flag. Library callers use simulateFallback({ root, scenario, limits, now }) and exitCodeFor(report) from src/index.mjs; now is an injected millisecond clock, default Date.now.

Where this check stops

Each configured bound is tested on both sides. Checks of elapsed time are cooperative; one filesystem read, JSON parse, hash or sort can overrun before the next checkpoint. This is not a latency benchmark, live failover engine, quality grader, billing source or deployment control. It executes no model response and writes no output artifact.

Before adapting the command to your own workflow, review the accepted inputs, exit codes and safety boundaries in the README.

Compiled with AI assistance from checked-in public documentation and example scripts. Run the example and review the repository's current documentation before relying on its result.