huggingface/tokenizers · warning · DeprecationWarning

Deprecated in

Error message

Deprecated in {version}: {message}

What it means

This is the message format for `DeprecationWarning` emitted by the Python bindings of tokenizers (`error.rs`). `deprecation_warning` formats `Deprecated in {version}: {message}` and raises a Python `DeprecationWarning` via `pyo3::PyErr::warn`. It is not a hard failure — the API still works — but signals that the called function (e.g. `get_words`, `save` in the training paths) will be removed in the given future version.

Solutions

1. Read the warning's `{version}` and `{message}` fields, then switch to the replacement API named in the message (e.g. call the trainer-based `train_from_files`/`Trainer` API instead of `get_words`, and the current `save` signature).
2. Run with `python -W error::DeprecationWarning` in a test run to find every call site emitting this warning before upgrading further.
3. Pin `tokenizers` to the version where the API still works if you cannot migrate yet, and schedule the migration before that version.
4. Update any third-party wrappers/libraries that call the deprecated methods — the warning may originate inside their code.

Example fix

// before
words, counts = tokenizer.get_words(texts)

// after
from tokenizers import trainers
trainer = trainers.WordPieceTrainer(vocab=...
trainer.train(tokenizer, texts)

import warnings
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    # exercise your training/save code paths here
assert not [w for w in caught if issubclass(w.category, DeprecationWarning)]

import warnings
with warnings.catch_warnings():
    warnings.filterwarnings("ignore", message="Deprecated in .*", category=DeprecationWarning)
    tokenizer.save(path)  # until migration is done

When it happens

Trigger: Calling a deprecated Python API of the tokenizers bindings, notably the legacy `tokenizer.train`-style paths such as `get_words` (word counting for trainers) and the old `save` behavior, in a release that marks them deprecated for a stated future version.

Common situations: Hit after upgrading the tokenizers Python package while keeping old training code: scripts that pre-compute word counts via `get_words`, or notebooks calling the old `save` signature; also surfaces in CI logs once warnings are enabled.

Understand the failure class

Background: "is deprecated and will be removed" — deprecation warnings for old API names, keywords, and options, and how to migrate before the removal release — this error's family across 29 libraries.

Related errors

• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`
• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`
• None input is not valid. Should be a list of integers.

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/51097ab0e47b3589. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/src/error.rs:41

}
impl std::error::Error for PyError {}

pub struct ToPyResult<T>(pub Result<T>);
impl<T> From<ToPyResult<T>> for PyResult<T> {
    fn from(v: ToPyResult<T>) -> Self {
        v.0.map_err(|e| exceptions::PyException::new_err(format!("{e}")))
    }
}
impl<T> ToPyResult<T> {
    pub fn into_py(self) -> PyResult<T> {
        self.into()
    }
}

pub(crate) fn deprecation_warning(py: Python<'_>, version: &str, message: &str) -> PyResult<()> {
    let deprecation_warning = py.import("builtins")?.getattr("DeprecationWarning")?;
    let full_message = format!("Deprecated in {version}: {message}");
    pyo3::PyErr::warn(py, &deprecation_warning, &CString::new(full_message)?, 0)
}

View on GitHub (pinned to 6cfd9d385c)