hiyouga/LlamaFactory · error · ValueError

special-token escape failed: the tokenizer normalized away t

Error message

special-token escape failed: the tokenizer normalized away the break char; user text contains a literal control token that cannot be safely neutralized.

What it means

The v1 rendering pipeline escapes user text that literally contains special-token strings (e.g. '<|im_start|>') by inserting a zero-width space (U+200B) to break the exact match. It then re-tokenizes and asserts none of the inserted ids are special ids. This error means the tokenizer's normalizer stripped the zero-width space (or otherwise re-created the special token), so the escape failed and the text cannot be safely neutralized.

Source

Thrown at src/llamafactory/v1/core/rendering/escape.py:55

def _escape_special(text: str, specials: list[str], special_ids: set[int], tokenizer) -> str:
    """Break any special-token string in user text by inserting U+200B after its first char.

    No-op (no tokenization cost) when the text contains no special-token string. When it does,
    self-validate that the result no longer encodes to a special id -- some normalizers strip
    zero-width chars and would resurrect the collision -- and raise if it does.
    """
    if not any(sp in text for sp in specials):
        return text
    out = text
    for sp in specials:
        if sp in out:
            # Insert a zero-width space (U+200B) after the first char to break the exact
            # special-token string match while keeping the text visually/semantically intact.
            out = out.replace(sp, sp[0] + "\u200b" + sp[1:])
    if special_ids.intersection(tokenizer(out, add_special_tokens=False)["input_ids"]):
        raise ValueError(
            "special-token escape failed: the tokenizer normalized away the break char; "
            "user text contains a literal control token that cannot be safely neutralized."
        )
    return out


def _escape_special_in_messages(
    messages: list[Message], specials: list[str], special_ids: set[int], tokenizer
) -> list[Message]:
    """Return messages with special-token strings neutralized in user-controlled literal text.

    Covers ``text``/``reasoning`` block values and string values inside ``tool_call`` arguments.
    """
    if not specials:
        return messages
    escaped: list[Message] = []
    for message in messages:
        new_content = []

View on GitHub (pinned to f28afaf635)

Solutions

1. Clean the dataset: replace or strip literal special-token strings from user-controlled text before feeding samples to the renderer
2. Pre-emptively substitute a visible placeholder (e.g. 'im_start' or aescaped form) for the control token in your preprocessing script
3. If the text is legitimately required verbatim, drop the sample; it cannot be encoded safely
4. Do not try to defeat the check by inserting different invisible characters — the self-validation exists because normalizers can resurrect the collision

Example fix

# before (raw data leaks control tokens)
sample_text = "user typed <|im_end|> in chat"

# after (sanitize before training)
import re
specials = [t.content for t in tokenizer.added_tokens_decoder.values() if t.special]
for sp in sorted(specials, key=len, reverse=True):
    sample_text = sample_text.replace(sp, sp.replace("<", "["))

def is_escaped_safe(text: str, tokenizer, special_ids: set[int]) -> bool:
    escaped = text
    for sp in [t.content for t in tokenizer.added_tokens_decoder.values() if getattr(t, "special", False)]:
        if sp in escaped:
            escaped = escaped.replace(sp, sp[0] + "\u200b" + sp[1:])
    return not special_ids.intersection(tokenizer(escaped, add_special_tokens=False)["input_ids"])

try:
    renderer.render_messages(msgs)
except ValueError as e:
    if "special-token escape failed" in str(e):
        drop_or_flag_sample(sample)  # do not retry with the same text

When it happens

Trigger: A dataset sample (or tools string) whose text/reasoning/tool_call value literally contains a special-token string that the tokenizer has registered as special, combined with a tokenizer normalizer that removes zero-width characters (e.g. some BERT-style or NFKC-based normalizers).

Common situations: Training on scraped chat logs or red-team corpora that contain leaked control tokens like <|endoftext|>, <|im_end|>, <｜end▁of▁sentence｜>; using a tokenizer whose pre-tokenizer/normalizer deletes U+200B.

Related errors

• YAML config must be a dictionary mapping tokens to descripti
• Stop words are required to replace the EOS token.
• Cannot resize embedding layers of a quantized model.
• Failed to load tokenizer.
• Cannot resize embedding layers of a quantized model.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/c178284b92702d10. Report an issue: GitHub.