{"record":{"id":"c178284b92702d10","repo":"hiyouga/LlamaFactory","slug":"special-token-escape-failed-the-tokenizer-normali","errorCode":null,"errorMessage":"special-token escape failed: the tokenizer normalized away the break char; user text contains a literal control token that cannot be safely neutralized.","messagePattern":"special-token escape failed: the tokenizer normalized away the break char; user text contains a literal control token that cannot be safely neutralized\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/rendering/escape.py","lineNumber":55,"sourceCode":"\n\ndef _escape_special(text: str, specials: list[str], special_ids: set[int], tokenizer) -> str:\n    \"\"\"Break any special-token string in user text by inserting U+200B after its first char.\n\n    No-op (no tokenization cost) when the text contains no special-token string. When it does,\n    self-validate that the result no longer encodes to a special id -- some normalizers strip\n    zero-width chars and would resurrect the collision -- and raise if it does.\n    \"\"\"\n    if not any(sp in text for sp in specials):\n        return text\n    out = text\n    for sp in specials:\n        if sp in out:\n            # Insert a zero-width space (U+200B) after the first char to break the exact\n            # special-token string match while keeping the text visually/semantically intact.\n            out = out.replace(sp, sp[0] + \"\\u200b\" + sp[1:])\n    if special_ids.intersection(tokenizer(out, add_special_tokens=False)[\"input_ids\"]):\n        raise ValueError(\n            \"special-token escape failed: the tokenizer normalized away the break char; \"\n            \"user text contains a literal control token that cannot be safely neutralized.\"\n        )\n    return out\n\n\ndef _escape_special_in_messages(\n    messages: list[Message], specials: list[str], special_ids: set[int], tokenizer\n) -> list[Message]:\n    \"\"\"Return messages with special-token strings neutralized in user-controlled literal text.\n\n    Covers ``text``/``reasoning`` block values and string values inside ``tool_call`` arguments.\n    \"\"\"\n    if not specials:\n        return messages\n    escaped: list[Message] = []\n    for message in messages:\n        new_content = []","sourceCodeStart":37,"sourceCodeEnd":73,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/rendering/escape.py#L37-L73","documentation":"The v1 rendering pipeline escapes user text that literally contains special-token strings (e.g. '<|im_start|>') by inserting a zero-width space (U+200B) to break the exact match. It then re-tokenizes and asserts none of the inserted ids are special ids. This error means the tokenizer's normalizer stripped the zero-width space (or otherwise re-created the special token), so the escape failed and the text cannot be safely neutralized.","triggerScenarios":"A dataset sample (or tools string) whose text/reasoning/tool_call value literally contains a special-token string that the tokenizer has registered as special, combined with a tokenizer normalizer that removes zero-width characters (e.g. some BERT-style or NFKC-based normalizers).","commonSituations":"Training on scraped chat logs or red-team corpora that contain leaked control tokens like <|endoftext|>, <|im_end|>, <｜end▁of▁sentence｜>; using a tokenizer whose pre-tokenizer/normalizer deletes U+200B.","solutions":["Clean the dataset: replace or strip literal special-token strings from user-controlled text before feeding samples to the renderer","Pre-emptively substitute a visible placeholder (e.g. 'im_start' or aescaped form) for the control token in your preprocessing script","If the text is legitimately required verbatim, drop the sample; it cannot be encoded safely","Do not try to defeat the check by inserting different invisible characters — the self-validation exists because normalizers can resurrect the collision"],"exampleFix":"# before (raw data leaks control tokens)\nsample_text = \"user typed <|im_end|> in chat\"\n\n# after (sanitize before training)\nimport re\nspecials = [t.content for t in tokenizer.added_tokens_decoder.values() if t.special]\nfor sp in sorted(specials, key=len, reverse=True):\n    sample_text = sample_text.replace(sp, sp.replace(\"<\", \"[\"))","handlingStrategy":"validation","validationCode":"def is_escaped_safe(text: str, tokenizer, special_ids: set[int]) -> bool:\n    escaped = text\n    for sp in [t.content for t in tokenizer.added_tokens_decoder.values() if getattr(t, \"special\", False)]:\n        if sp in escaped:\n            escaped = escaped.replace(sp, sp[0] + \"\\u200b\" + sp[1:])\n    return not special_ids.intersection(tokenizer(escaped, add_special_tokens=False)[\"input_ids\"])","typeGuard":null,"tryCatchPattern":"try:\n    renderer.render_messages(msgs)\nexcept ValueError as e:\n    if \"special-token escape failed\" in str(e):\n        drop_or_flag_sample(sample)  # do not retry with the same text","preventionTips":["Sanitize special-token strings out of scraped corpora during data prep","Run a one-off scan of the dataset for tokenizer.special_tokens_map values before training"],"tags":["tokenizer","special-tokens","data-cleaning","prompt-injection","rendering"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}