{"record":{"id":"51097ab0e47b3589","repo":"huggingface/tokenizers","slug":"deprecated-in-version-message","errorCode":null,"errorMessage":"Deprecated in {version}: {message}","messagePattern":"Deprecated in (.+?): (.+?)","errorType":"console","errorClass":"DeprecationWarning","httpStatus":null,"severity":"warning","filePath":"bindings/python/src/error.rs","lineNumber":41,"sourceCode":"}\nimpl std::error::Error for PyError {}\n\npub struct ToPyResult<T>(pub Result<T>);\nimpl<T> From<ToPyResult<T>> for PyResult<T> {\n    fn from(v: ToPyResult<T>) -> Self {\n        v.0.map_err(|e| exceptions::PyException::new_err(format!(\"{e}\")))\n    }\n}\nimpl<T> ToPyResult<T> {\n    pub fn into_py(self) -> PyResult<T> {\n        self.into()\n    }\n}\n\npub(crate) fn deprecation_warning(py: Python<'_>, version: &str, message: &str) -> PyResult<()> {\n    let deprecation_warning = py.import(\"builtins\")?.getattr(\"DeprecationWarning\")?;\n    let full_message = format!(\"Deprecated in {version}: {message}\");\n    pyo3::PyErr::warn(py, &deprecation_warning, &CString::new(full_message)?, 0)\n}\n","sourceCodeStart":23,"sourceCodeEnd":43,"githubUrl":"https://github.com/huggingface/tokenizers/blob/6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607/bindings/python/src/error.rs#L23-L43","documentation":"This is the message format for `DeprecationWarning` emitted by the Python bindings of tokenizers (`error.rs`). `deprecation_warning` formats `Deprecated in {version}: {message}` and raises a Python `DeprecationWarning` via `pyo3::PyErr::warn`. It is not a hard failure — the API still works — but signals that the called function (e.g. `get_words`, `save` in the training paths) will be removed in the given future version.","triggerScenarios":"Calling a deprecated Python API of the tokenizers bindings, notably the legacy `tokenizer.train`-style paths such as `get_words` (word counting for trainers) and the old `save` behavior, in a release that marks them deprecated for a stated future version.","commonSituations":"Hit after upgrading the tokenizers Python package while keeping old training code: scripts that pre-compute word counts via `get_words`, or notebooks calling the old `save` signature; also surfaces in CI logs once warnings are enabled.","solutions":["Read the warning's `{version}` and `{message}` fields, then switch to the replacement API named in the message (e.g. call the trainer-based `train_from_files`/`Trainer` API instead of `get_words`, and the current `save` signature).","Run with `python -W error::DeprecationWarning` in a test run to find every call site emitting this warning before upgrading further.","Pin `tokenizers` to the version where the API still works if you cannot migrate yet, and schedule the migration before that version.","Update any third-party wrappers/libraries that call the deprecated methods — the warning may originate inside their code."],"exampleFix":"// before\nwords, counts = tokenizer.get_words(texts)\n\n// after\nfrom tokenizers import trainers\ntrainer = trainers.WordPieceTrainer(vocab=...\ntrainer.train(tokenizer, texts)","handlingStrategy":"try-catch","validationCode":"import warnings\nwith warnings.catch_warnings(record=True) as caught:\n    warnings.simplefilter(\"always\")\n    # exercise your training/save code paths here\nassert not [w for w in caught if issubclass(w.category, DeprecationWarning)]","typeGuard":null,"tryCatchPattern":"import warnings\nwith warnings.catch_warnings():\n    warnings.filterwarnings(\"ignore\", message=\"Deprecated in .*\", category=DeprecationWarning)\n    tokenizer.save(path)  # until migration is done","preventionTips":["Treat DeprecationWarning as an error in CI (`python -W error::DeprecationWarning`) to catch removals early.","Read the version in the warning and migrate to the replacement API before that release.","After upgrading tokenizers, scan logs for this warning before promoting the new version.","Keep training code aligned with the current Trainer API instead of legacy `get_words`/`save` paths."],"tags":["python","deprecation","tokenizers","pyo3"],"backgroundTag":"deprecated-api-usage","analyzedSha":"6cfd9d385ca0ed91c10b49f0ce97d02cfde1b607","analyzedAt":"2026-09-09T11:43:25.027Z","contentChangedAt":"2026-09-09T11:43:25.027Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}