Verbora

The whole NLP stack, in one Rust toolkit.

Tokenizing, normalizing, stemming, string distance, phonetics, n-grams, TF-IDF, sentiment, classifiers, POS tagging and WordNet — nineteen focused verbora-* crates behind one coherent design. Iterators first, borrowed tokens, and no allocation you did not ask for.

Get startedBrowse the features

19 cratesDepend on what you useIterators firstLazy, borrowed tokensNo unsafeDenied workspace-wideReproducibleHardware and commands

Quick start

[dependencies]
verbora-tokenizers = "0.2"
verbora-distance = "0.2"

use verbora_tokenizers::{BorrowingTokenizer, WordTokenizer};

let tokens: Vec<&str> = WordTokenizer.tokenize_borrowed("Verbora reads text without copying it");

assert_eq!(tokens[0], "Verbora");
assert_eq!(tokens.len(), 6);

Every crate stands alone: depend on the ones you use. Git and path dependencies are covered in Installation.

Start where your work starts

Clear paths in

Learn the primitives, pick an API by workload, or go straight to the exact Rust contract. Every path leads to the same composable core.

01

Get started

Add a crate, run your first program, and read the workspace map.

Installation →02

Choose an API

Ergonomic, lazy, reusable-buffer or batch — with the trade-off behind each.

Choosing guide →03

Read the reference

Exact signatures, types, defaults and crate-level documentation.

Rust API reference →

One toolkit. Every layer of language.

Four families over one shared core. Every leaf is a crate you can use on its own, behind a fast, idiomatic Rust API.

01Prepare

Raw text into comparable units

• Tokenizers3, UAX #29
• Normalizers5, all Cow
• Inflectors6
• Transliteratorskana → romaji
• Stemmers16

02Match

Similarity, sound and lookup

• String distance7 metrics
• Phonetics12 encoders
• Phonetic indexnative
• Trieprefix + path
• Spellcheckcorrection + index

03Weigh

Statistics and trained models

• N-gramswindows + padding
• TF-IDFsparse, interned
• Sentiment14 lexicons
• Classifiers3 models

04Understand

Grammar, structure and meaning

• WordNet4 storages
• Language detectionnative
• POS taggerBrill
• Sentence analysis

High-performance language toolkit

All four families rest on verbora-core — five traits and StopWords — and on verbora-util: abbreviations, graphs, path trees.

Everything on the map ships today; none of it is roadmap. The two entries marked native — the phonetic index and language detection — are Verbora's own designs rather than implementations of a published algorithm, and are tested and benchmarked like the rest. Two things to know before you plan around it: WordNet's database is separately licensed and not bundled with the crate, and the POS tagger is an engine — it ships no dictionary, so you supply the lexicon.

One operation, four shapes

A tokenizer called once per HTTP request and one called forty million times in a batch job are not the same problem. Each shape below is a real API; what differs is what it does with memory.

Ergonomic

tokenizer.tokenize(text)

Returns a Vec of tokens. The right call for most programs — and it is not the "slow" one.

Lazy · zero-copy

tokenizer.tokens(text)

An iterator. Each token borrows the input, nothing is materialised, and you can stop early.

Buffer reuse

tokenizer.tokenize_into(text, &mut buf)

Appends into a buffer you own, so a hot loop amortises one allocation across a whole corpus.

Scale

Tokenizer::tokenize_batch(&docs)

Sequential by default. Most crates also ship an opt-in par_* batch API behind a parallel Cargo feature — see Parallelism.

Choosing the right API has the comparison tables and decision trees for every subsystem that offers more than one shape.

Why it is fast

• Tokenizers are iterators first. tokens() is the primitive; tokenize() and tokenize_into() are written on top of it, so there is one implementation and no second copy to drift.
• Nothing is copied that need not be. Every token all three tokenizers yield is a &str slice of your input; all five normalizers return Cow<'_, str> and allocate only when the result would differ from what you passed in.
• The per-call path has almost nothing to fail on. Every phonetic encoder and every inflector method is total: no Result, no panic, on any &str. An input the algorithm recognises nothing in yields an empty key or an unchanged word — an answer you branch on, not an error you handle. What is fallible is construction, where a bad abbreviation list or a rule that will not compile is rejected once, before the first call.
• The data layout is chosen, not inherited. The trie is a flat arena addressed by u32, not one heap object per node; unit-cost Levenshtein is bit-parallel at every length — one 64-bit word of state for a short pattern, contiguous blocks of them for a long one — and a dynamic-programming row appears only in the weighted forms, which have no bit-parallel formulation.

Every number on this site carries its hardware, its method and the command to reproduce it, and a figure whose code has changed underneath it is marked pending rather than restated from memory — see Performance and Benchmarks for the current results, including the ones Verbora loses.

Correctness and scope. Behaviour is pinned by an executable specification: each algorithm's output is recorded case by case, checked into the repository as data, and replayed by that crate's test suite asserting exact equality. Verbora is pre-1.0 — APIs may still be refined, but documented behaviour changes only together with those recordings. See Status and scope.

Go deeper

Features →Every crate: what it does, when to reach for it, the API surface and its guarantees.Performance →Borrowing, laziness, Cow, buffer reuse, batching and parallelism — plus the measured results.Recipes by workload →Start from your problem — request/response, streaming, batch, huge corpora — not from a function name.

For exact type signatures see the Rust API reference; for how these pages stay in step with the code, see Documentation is part of the code.