hiyouga/LlamaFactory · error · ValueError

prompt is not a token-prefix of the full sequence; the chat

Error message

prompt is not a token-prefix of the full sequence; the chat template is not prefix-stable for this turn, so diff-based labeling is unsafe.

What it means

Training labels are computed by rendering the conversation without the final assistant turn (with add_generation_prompt=True) and diffing against the full render. This requires the prompt ids to be an exact token-prefix of the full ids. Some chat templates are not prefix-stable — they re-render earlier turns differently once the final turn is appended (dynamic system prompts, dates, changed enable_thinking, whitespace) or the tokenizer merges tokens across the boundary — and then diff-based labeling would mislabel tokens, so the renderer fails loudly.

Source

Thrown at src/llamafactory/v1/core/rendering/rendering.py:155

            attention_mask=[1] * n,
            labels=[IGNORE_INDEX] * n,
            loss_weights=[0.0] * n,
        )
        _attach_multimodal(result)
        return result

    if not messages or messages[-1]["role"] != "assistant":
        raise ValueError(
            "training render expects the last message to be the supervised assistant turn; "
            "multi-turn conversations are split per turn in process_samples."
        )

    prompt_ids, _ = _encode(hf_messages[:-1], messages[:-1], add_generation_prompt=True)
    if input_ids[: len(prompt_ids)] != prompt_ids:
        # The prompt must be a token-prefix of the full sequence for the diff to be valid. If a
        # template re-renders earlier turns when the final turn is appended, fail loud rather than
        # mislabel.
        raise ValueError(
            "prompt is not a token-prefix of the full sequence; the chat template is not "
            "prefix-stable for this turn, so diff-based labeling is unsafe."
        )

    weight = messages[-1].get("loss_weight", 1.0)
    supervised = weight > 1e-6
    labels = [IGNORE_INDEX] * len(prompt_ids)
    loss_weights = [0.0] * len(prompt_ids)
    for tid in input_ids[len(prompt_ids) :]:
        labels.append(tid if supervised else IGNORE_INDEX)
        loss_weights.append(weight)

    result = ModelInput(
        input_ids=input_ids,
        attention_mask=[1] * n,
        labels=labels,
        loss_weights=loss_weights,
    )

View on GitHub (pinned to f28afaf635)

Solutions

1. Make the template deterministic and prefix-stable: remove date/time inserts and history-dependent reformatting
2. Guarantee identical template kwargs for both renders (the code already forwards enable_thinking; ensure your template does not derive behavior from the presence of the last turn)
3. Insert an explicit structural break before the assistant header (e.g. newline/special token the tokenizer cannot merge across)
4. Test with the model's stock template; if it passes, the issue is in your custom Jinja

Example fix

# before (template fragment, not prefix-stable)
"{%- if messages[-1].role == 'assistant' %}<|old_history|>{%- endif %}"

# after (deterministic rendering of history)
"{%- for m in messages %}{{ '<|im_start|>' + m.role }}{%- endfor %}"

def template_is_prefix_stable(template, tokenizer, msgs) -> bool:
    full = tokenizer.apply_chat_template(msgs, tokenize=True)
    prompt = tokenizer.apply_chat_template(msgs[:-1], tokenize=True, add_generation_prompt=True)
    return full[: len(prompt)] == prompt

When it happens

Trigger: Using a chat template whose output for messages[:-1] is not a strict prefix of the output for messages (e.g. templates that inject the current date, reorder or trim history based on the last turn, or toggle a thinking block); tokenizers where the generation-prompt boundary merges with the preceding token (e.g. no space before the assistant header).

Common situations: Custom or third-party Jinja chat templates with time-dependent or turn-count-dependent behavior; Qwen3-style thinking templates where enable_thinking differs between the two renders; BPE tokenizers whose pretokenizer crosses the assistant-tag boundary.

Related errors

• training render expects the last message to be the supervise
• Not allowed

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/24ce4a41931bcbfd. Report an issue: GitHub.