The four API shapes

Verbora's naming is regular. Once you know the four shapes, you can predict what a function does — and what it costs — from its name alone.

Shape	Name pattern	Returns	Allocates	Lazy
Eager	verb(input)	owned collection or String	yes — the container	❌
Lazy	nouns(input), verb_iter(input), iter_*	an Iterator	no	✅
Into-buffer	verb_into(input, &mut out)	() or bool	no, after warm-up	❌
Batch	verb_batch(&[input])	Vec<…> per input	yes	❌

Naming is a promise, not a guarantee of existence. Not every subsystem has all four. Phonetics has only the eager shape; distance has only eager scalars. The feature pages say exactly which exist.

1. Eager — tokenize(), process(), pluralize(), keys_with_prefix()

OWNED

Does the work now, hands back a complete result you own.

use verbora_tokenizers::{AggressiveTokenizer, Tokenize};

let t = AggressiveTokenizer::new();
let tokens = t.tokenize("the quick brown fox");

assert_eq!(tokens.len(), 4);
assert_eq!(tokens[2], "brown");

Use it when the result is small, you need random access or a length, you are passing it somewhere that wants a slice, or you simply want the code to read well. That is most code.

Do not use it when you are about to consume the result once, in order, and throw it away — that is what the lazy shape is for — or when you are calling it in a loop millions of times and the container allocation shows up in a profile.

Cost. One container allocation, plus growth reallocations as it fills. Note what is not allocated: for the thirteen character-class tokenizers the tokens are &str slices of your input, so there is no per-token String.

2. Lazy — tokens(), ngrams_iter(), iter_keys_with_prefix()

LAZYZERO-COPY

Returns an iterator. Nothing happens until you pull.

use verbora_tokenizers::{AggressiveTokenizer, Tokenize};

let t = AggressiveTokenizer::new();

let shouty: Vec<String> = t
    .tokens("the quick brown fox")
    .filter(|w| w.len() > 3)
    .map(|w| w.to_uppercase())
    .collect();

assert_eq!(shouty, ["QUICK", "BROWN"]);

Use it when you are building a pipeline, when you might stop early, when the input is large enough that materialising every token at once matters, or when you want to hand a stream to another API that takes IntoIterator — such as phoneticize_tokens, which composes directly with a tokenizer's iterator and never builds the intermediate Vec.

Do not use it when you need the result more than once, need its length up front, or need indexing. Re-running an iterator means re-doing the work; a Vec you can read twice.

Cost. For AggressiveTokenizer and the other twelve character-class tokenizers, nothing per token — the iterator is a small struct on the stack, scanning as it goes. Not every Tokenize implementor is this lazy end to end: TreebankWordTokenizer, TokenizerJa, SentenceTokenizer, AggressiveTokenizerNo/Sv and CaseTokenizer on non-ASCII input each do some eager work inside tokens() before yielding the first item — see Tokenizers for which.

Borrow-checker note. A lazy iterator borrows the input string for as long as it lives. If you want to return tokens from a function that owns its input, you must either collect or restructure so the caller owns the text. This is not a Verbora limitation — it is the price of not copying.

3. Into-buffer — tokenize_into(), pluralize_into(), stem_into()

BUFFER REUSE

Writes into storage you own and keep between calls.

use verbora_tokenizers::{AggressiveTokenizer, Tokenize};

let t = AggressiveTokenizer::new();
let corpus = ["the quick brown fox", "jumps over the lazy dog"];

let mut buf = Vec::new();
let mut total = 0;

for document in corpus {
    buf.clear();                          // capacity survives; contents do not
    t.tokenize_into(document, &mut buf);
    total += buf.len();
}

assert_eq!(total, 9);

Use it when the same operation runs in a tight loop and the container allocation is a real cost — batch jobs, corpus indexing, offline processing.

Do not use it when you call the operation once. You have added a mutable binding and a manual clear() to your code in exchange for one saved allocation.

Cost. After the buffer reaches its high-water mark, zero allocations. Before that, the same growth pattern as the eager shape.

The two conventions. Tokenize::tokenize_intoappends — it does not clear, so you can accumulate across inputs on purpose. Stemmer::stem_into clears first. They differ because appending tokens across documents is useful and appending stem fragments is not, but you must check per API. Each one says so in its rustdoc.

4. Batch — tokenize_batch(), stem_batch()

BATCH

Takes a slice of inputs, returns a result per input.

use verbora_core::Tokenizer;
use verbora_tokenizers::AggressiveTokenizer;

let t = AggressiveTokenizer::new();
let out = t.tokenize_batch(&["one two", "three four five"]);

assert_eq!(out.len(), 2);
assert_eq!(out[1].len(), 3);

Batch is a call-site convenience, not an optimisation. The batch methods on verbora_core::Tokenizer and verbora_core::Stemmer are provided methods whose default bodies are a sequential map: one fresh Vec<String> per input, no shared buffer, no parallelism. tokenize_batch allocates more than tokenize_into does, and produces owned Strings rather than the borrowed &str that Tokenize::tokenize gives you.

Use it when you are writing generic code over the Tokenizer trait and want the batch operation to improve automatically if an implementation overrides it.

Do not use it when you want throughput today. Reach for verbora-tokenizers's own Tokenize::par_tokenize_batch (behind its parallel feature) or tokenize_into with a reused buffer, or rayon over your own slice for anything without a built-in par_* variant — see Parallelism.

Two more shapes you will meet

These are not "levels" — they are type choices that change what you can do with a result.

Cow-returning functions

COW

Four of the six top-level normalizers, plus every one of the 17 ja::converters and Stemmer::stem, return Cow<'_, str>: borrowed when nothing changed, owned when something did. (normalize and normalize_token are the two exceptions — see Normalizers for why: one contraction can expand into several tokens, so the result is a Vec<String> regardless.)

use std::borrow::Cow;
use verbora_normalizers::remove_diacritics;

// Nothing to fold — no allocation at all.
assert!(matches!(remove_diacritics("plain ascii"), Cow::Borrowed(_)));

// A fold happened — one String.
assert!(matches!(remove_diacritics("café"), Cow::Owned(_)));
assert_eq!(remove_diacritics("café"), "cafe");

This matters because these functions are usually called on text that needs no change — a Latin sentence handed to the katakana converter, an ASCII token handed to the diacritic folder. See Zero-copy and Cow.

Option-returning tokenizers

FALLIBLE

The four regex-driven tokenizers return Option<…>, because "no match at all" is observably different from "matched, but produced no tokens" — and Option is how Verbora keeps that distinction expressible.

use verbora_tokenizers::WordTokenizer;

let t = WordTokenizer::new();
assert!(t.tokenize("hello world").is_some());

Summary

You want	Shape	Example calls
a result you can hold, index or pass on	eager	tokenize(), process(), keys_with_prefix()
to consume it once, in order — maybe not all of it	lazy	tokens(), ngrams_iter(), iter_keys_with_prefix()
to do this millions of times with the same shape of output	into-buffer	tokenize_into(), pluralize_into()
generic code over the trait	batch	tokenize_batch() (sequential today)

Next: the same reasoning applied concretely, in Tokenization.