{"record":{"id":"24ce4a41931bcbfd","repo":"hiyouga/LlamaFactory","slug":"prompt-is-not-a-token-prefix-of-the-full-sequence","errorCode":null,"errorMessage":"prompt is not a token-prefix of the full sequence; the chat template is not prefix-stable for this turn, so diff-based labeling is unsafe.","messagePattern":"prompt is not a token-prefix of the full sequence; the chat template is not prefix-stable for this turn, so diff-based labeling is unsafe\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/llamafactory/v1/core/rendering/rendering.py","lineNumber":155,"sourceCode":"            attention_mask=[1] * n,\n            labels=[IGNORE_INDEX] * n,\n            loss_weights=[0.0] * n,\n        )\n        _attach_multimodal(result)\n        return result\n\n    if not messages or messages[-1][\"role\"] != \"assistant\":\n        raise ValueError(\n            \"training render expects the last message to be the supervised assistant turn; \"\n            \"multi-turn conversations are split per turn in process_samples.\"\n        )\n\n    prompt_ids, _ = _encode(hf_messages[:-1], messages[:-1], add_generation_prompt=True)\n    if input_ids[: len(prompt_ids)] != prompt_ids:\n        # The prompt must be a token-prefix of the full sequence for the diff to be valid. If a\n        # template re-renders earlier turns when the final turn is appended, fail loud rather than\n        # mislabel.\n        raise ValueError(\n            \"prompt is not a token-prefix of the full sequence; the chat template is not \"\n            \"prefix-stable for this turn, so diff-based labeling is unsafe.\"\n        )\n\n    weight = messages[-1].get(\"loss_weight\", 1.0)\n    supervised = weight > 1e-6\n    labels = [IGNORE_INDEX] * len(prompt_ids)\n    loss_weights = [0.0] * len(prompt_ids)\n    for tid in input_ids[len(prompt_ids) :]:\n        labels.append(tid if supervised else IGNORE_INDEX)\n        loss_weights.append(weight)\n\n    result = ModelInput(\n        input_ids=input_ids,\n        attention_mask=[1] * n,\n        labels=labels,\n        loss_weights=loss_weights,\n    )","sourceCodeStart":137,"sourceCodeEnd":173,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/v1/core/rendering/rendering.py#L137-L173","documentation":"Training labels are computed by rendering the conversation without the final assistant turn (with add_generation_prompt=True) and diffing against the full render. This requires the prompt ids to be an exact token-prefix of the full ids. Some chat templates are not prefix-stable — they re-render earlier turns differently once the final turn is appended (dynamic system prompts, dates, changed enable_thinking, whitespace) or the tokenizer merges tokens across the boundary — and then diff-based labeling would mislabel tokens, so the renderer fails loudly.","triggerScenarios":"Using a chat template whose output for messages[:-1] is not a strict prefix of the output for messages (e.g. templates that inject the current date, reorder or trim history based on the last turn, or toggle a thinking block); tokenizers where the generation-prompt boundary merges with the preceding token (e.g. no space before the assistant header).","commonSituations":"Custom or third-party Jinja chat templates with time-dependent or turn-count-dependent behavior; Qwen3-style thinking templates where enable_thinking differs between the two renders; BPE tokenizers whose pretokenizer crosses the assistant-tag boundary.","solutions":["Make the template deterministic and prefix-stable: remove date/time inserts and history-dependent reformatting","Guarantee identical template kwargs for both renders (the code already forwards enable_thinking; ensure your template does not derive behavior from the presence of the last turn)","Insert an explicit structural break before the assistant header (e.g. newline/special token the tokenizer cannot merge across)","Test with the model's stock template; if it passes, the issue is in your custom Jinja"],"exampleFix":"# before (template fragment, not prefix-stable)\n\"{%- if messages[-1].role == 'assistant' %}<|old_history|>{%- endif %}\"\n\n# after (deterministic rendering of history)\n\"{%- for m in messages %}{{ '<|im_start|>' + m.role }}{%- endfor %}\"","handlingStrategy":"validation","validationCode":"def template_is_prefix_stable(template, tokenizer, msgs) -> bool:\n    full = tokenizer.apply_chat_template(msgs, tokenize=True)\n    prompt = tokenizer.apply_chat_template(msgs[:-1], tokenize=True, add_generation_prompt=True)\n    return full[: len(prompt)] == prompt","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Run a prefix-stability smoke test for every new chat template before training","Avoid date/time or last-turn-dependent constructs in custom Jinja templates"],"tags":["chat-template","tokenization","prefix-stability","training-data","labels"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}