Features overview
Every subsystem below is implemented, tested, and documented. Start from what you want to do, or from the crate map further down.
Start from the problem
| I want to… | Use |
|---|---|
| Split text into words | Tokenizers — start with AggressiveTokenizer |
| Split text into sentences | SentenceTokenizer |
| Tokenize a language other than English | Tokenizers — 16 language variants |
| Measure how similar two strings are | String distance |
| Correct a typo | levenshtein, usually after narrowing candidates |
| Correct spelling against a corpus | Spellcheck |
| Match names that sound alike | Phonetics |
| Search a whole dictionary for sound-alikes | Phonetic neighbors |
| Match names across language boundaries (genealogy) | Beider-Morse |
| Match names with a shared prefix | jaro_winkler |
| Work out which encoder or stemmer a text even needs | Language |
| Build an autocomplete index | Trie |
| Find repeated phrases | N-grams |
| Strip accents before comparing | remove_diacritics |
| Expand English contractions | normalize |
| Normalise Japanese width and kana | normalize_ja |
| Romanise Japanese kana | Transliterators |
| Pluralise a noun, or write "23rd" | Inflectors |
| Reduce words to a stem | Stemmers |
| Rank documents by relevance | TF-IDF |
| Classify documents into categories | Classifiers |
| Score sentiment over a token list | Sentiment |
| Look up synonyms and word relations | WordNet |
| Tag parts of speech | POS tagger |
| Split a tagged sentence into subject and predicate | Sentence analyzers |
| Write generic code over any tokenizer | Core vocabulary |
The crates
| Subsystem | Crate | Public surface |
|---|---|---|
| Tokenizers | verbora-tokenizers | 25 tokenizers, Tokenize, Utf16Token |
| String distance | verbora-distance | 8 metrics, 5 StringMetric impls |
| Phonetics | verbora-phonetics | 11 encoders: SoundEx, Metaphone, Double Metaphone, Daitch–Mokotoff ×2, Cologne, Nysiis, Caverphone 1/2, Phonex, Refined Soundex, Match Rating |
| Phonetic neighbors | verbora-phonetics | PhoneticIndex — dictionary-wide candidate generation over any of the four core encoders |
| Beider-Morse | verbora-phonetics | Cross-language surname matching over up to 18 languages at once, with auto-detection |
| Language | verbora-language | Script detection, optional statistical language detection, and recommend() — language → phonetic encoder |
| N-grams | verbora-ngrams | window engine, stats, Chinese n-grams |
| Normalizers | verbora-normalizers | 6 normalizers, 17 Japanese converters |
| Inflectors | verbora-inflectors | 6 inflectors, runtime rules |
| Trie | verbora-trie | prefix tree, prefix and path queries |
| Transliterators | verbora-transliterators | Japanese kana → romaji, five-phase pipeline |
| WordNet | verbora-wordnet | lexical database, synsets, relation traversal, 4 storage strategies |
| TF-IDF | verbora-tfidf | term interning, incremental idf cache, listTerms/tfidf/tfidfs |
| Sentiment | verbora-sentiment | 14 lexicons across 10 languages, sticky negation |
| Classifiers | verbora-classifiers | Bayes, logistic regression, MaxEnt + GIS |
| Stemmers | verbora-stemmers | 16 stemmers: Porter/Snowball across 12 languages, Lancaster, Japanese, Indonesian |
| Spellcheck | verbora-spellcheck | frequency-ranked correction, BK-tree and deletion indexes |
| POS tagger | verbora-tagger | Brill tagging, training and evaluation; English and Dutch data |
| Sentence analyzers | verbora-analyzers | phrase annotation, subject/predicate splitting, sentence type |
| Utilities | verbora-util | stop words, abbreviations, weighted graphs and storage backends |
| Core vocabulary | verbora-core | 6 traits, Token, StopWords, whitespace helpers |
Language support
| Language | Tokenizer | Normalizer | Inflector | Phonetics |
|---|---|---|---|---|
| English | ✅ | ✅ | ✅ nouns, verbs, ordinals | ✅ all four core encoders |
| French | ✅ | ✅ nouns, ordinals | ||
| German | ✅ | |||
| Spanish | ✅ | |||
| Italian | ✅ | |||
| Portuguese | ✅ | |||
| Dutch | ✅ | |||
| Norwegian | ✅ | ✅ | ||
| Swedish | ✅ | ✅ | ||
| Danish/Nordic | ✅ | |||
| Russian | ✅ | |||
| Ukrainian | ✅ | |||
| Polish | ✅ | |||
| Persian | ✅ | |||
| Hindi | ✅ | |||
| Indonesian | ✅ | |||
| Vietnamese | ✅ | |||
| Finnish | ✅ OrthographyTokenizer | |||
| Japanese | ✅ | ✅ + 17 converters | ✅ nouns | |
| Chinese | ✅ n-grams |
Latin-script diacritic folding via remove_diacritics applies far more broadly than the Normalizer column suggests — it is a table over 820 non-ASCII characters, not a per-language rule set. Stemmers cover 12 languages on their own axis; see Stemmers.
Where else to look
- Choosing the right API — cross-subsystem decisions.
- Performance — allocation, zero-copy, parallelism.
- Benchmarks — measured numbers and how to reproduce.
- Recipes — end-to-end pipelines.
- Status and scope — what "available" means, and the boundaries.
- Exact signatures live in rustdoc; these pages cover selection, composition, costs and common mistakes.