Open source · AI & LLM

Retrieval Chunk Auditor

Find chunks that are too large, duplicated, stale or missing provenance.

v0.1.0 · Node.js 22+ · MIT

Browse the public repository · View releases

Audit a local retrieval export before it is indexed. A chunk that has lost its source, repeats another span, carries an old source hash, runs too large, overlaps too much, or cuts a Markdown heading can make retrieval less trustworthy. This tool reports those conditions from exported documents only; it never builds an embedding, calls a model, or contacts a service.

This walkthrough uses the tool's public README and checked-in example files. Run the command from a repository checkout with Node.js 22+.

Run the checked-in example

node bin/retrieval-chunk-auditor.mjs --input examples/clean.json --json
node bin/retrieval-chunk-auditor.mjs --input examples/duplicate.json --json
npm run check

Read the result

The second command emits a fail report and exits 1. examples/stale.json shows an incomplete provenance report and exit 2. The CLI writes only a JSON report to stdout. Without --json, a short operational summary goes to stderr. It never writes or modifies an input or index.

Scope

Each bound is inclusive: exactly N is allowed and N+1 is reported. A limit that prevents evidence is incomplete and exit 2. The clock is injected into the library for deterministic tests; the CLI uses a monotonic clock. The deadline is cooperative, not preemptive: a single JSON parse, filesystem call, native string search or sort can overrun it before the next checkpoint. The report contains no timestamp or elapsed duration.