Reproducing the benchmarks

The commands below are the ones that produced every number in this section — run them from a clean checkout and they work, or that is a bug.

A fresh run will not always reproduce the printed number. Nothing on the benchmark method, competitive or string distance pages has been re-measured against Verbora 0.2.0, and several of the kernels those pages' numbers measured have since been replaced. Where a table or row is marked **pending re-measurement**, running the matching command here reports whatever the current code does, not the figure sitting next to it — that gap is the open work, not a bug in the command. See Upgrading from 0.1 to 0.2 for what changed.

Prerequisites

Rust	1.85+ (the workspace is edition 2024)
Python	3.8+ (generates the shared benchmark inputs)

git clone https://github.com/addlayerio/verbora
cd verbora

1. Generate the shared inputs

Every harness reads the same files. Run this once:

python3 tools/bench-data/generate.py

It writes benches/data/:

benches/data/words.json            word lists at several lengths
benches/data/distance-pairs.json   the string pairs the metrics compare

No harness generates its own inputs, so none can be tuned to a distribution that flatters it.

2. Run a crate's own benchmarks

cargo bench -p verbora-distance

Criterion writes HTML reports to target/criterion/ and stores a baseline it compares against on the next run. Eighteen crates ship a benches/ directory; verbora-distance is above, and the other seventeen are:

cargo bench -p verbora-analyzers
cargo bench -p verbora-classifiers
cargo bench -p verbora-inflectors
cargo bench -p verbora-language
cargo bench -p verbora-ngrams
cargo bench -p verbora-normalizers
cargo bench -p verbora-phonetics
cargo bench -p verbora-sentiment
cargo bench -p verbora-spellcheck
cargo bench -p verbora-stemmers
cargo bench -p verbora-tagger
cargo bench -p verbora-tfidf
cargo bench -p verbora-tokenizers
cargo bench -p verbora-transliterators
cargo bench -p verbora-trie
cargo bench -p verbora-util
cargo bench -p verbora-wordnet

3. Run the competitive suite

Head-to-head against the pinned third-party crates, with its own structured results:

./scripts/competitive-benchmarks.sh              # every module
./scripts/competitive-benchmarks.sh distance     # one module

It writes:

benchmarks/competitive/results/results.json      structured summary
benchmarks/competitive/results/raw/              Criterion's own estimates
benchmarks/competitive/results/metadata.json     machine and toolchain

The raw estimates are kept so the summary can be re-derived rather than taken on trust.

Comparing two runs

Criterion compares against its stored baseline automatically and reports "Performance has improved" or "has regressed" per benchmark. To name a baseline explicitly:

cargo bench -p verbora-distance -- --save-baseline before
# ... make a change ...
cargo bench -p verbora-distance -- --baseline before

This is how the Jaro–Winkler fix was confirmed. Small movements between runs are noise and are not gated in CI; material changes to tokenizer throughput, stemming, Levenshtein, TF-IDF, WordNet lookup, classifier prediction and sentiment analysis are recorded with the benchmark result and the commit that caused them.

Profiling

The bench profile inherits release and adds debug symbols, so perf and samply resolve frames:

cargo bench -p verbora-distance --no-run
perf record -g ./target/release/deps/distance-<hash> --bench
perf report

For maximum runtime speed at a significant compile-time cost:

cargo bench --profile release-max -p verbora-distance

Published tables use the ordinary release settings (opt-level = 3, lto = "thin", codegen-units = 16), because that is what most people build.

Before you trust a result

Re-run the test suite. A benchmark whose faster side computes something cheaper is not a benchmark:

cargo test --workspace

Quiet the machine. Close other work, and prefer a fixed CPU governor. On Linux:

sudo cpupower frequency-set --governor performance

Run it more than once. Criterion's confidence intervals tell you whether a difference is real.

Check the input size. A ratio measured on four-character strings and a ratio measured on 1024-character strings are both true statements about the same function, and they can differ by orders of magnitude. Quoting either alone is misleading.

Adding a benchmark

1. Add the Criterion group to crates/<crate>/benches/<name>.rs.
2. If a competitor exists, add it to benchmarks/competitive/rust-competitors/benches/<name>.rs and register the module in scripts/competitive-benchmarks.sh's MODULE_SPECS.
3. If new inputs are needed, generate them in tools/bench-data/generate.py, so every harness reads the same bytes.
4. Run it, and publish the reviewed result with the hardware, the toolchain versions and the commands — see Documentation is part of the code.