OpenBMB/MiniCPM-V · warning

Could not find instruction key `{instruction_template}` in t

Error message

Could not find instruction key `{instruction_template}` in the following instance: @===>{tokenizer.decode(res_input_ids)}<===@ Raw text is @===>{res_text}<===@Raw source is @===>{new_source}<===@This instance will be ignored in loss calculation. Note, if this happens often, consider increasing the `max_seq_length`.

What it means

The counterpart of error 14: omni_preprocess scans res_labels for the instruction (human/USER) template token ids to mask out prompt tokens from the loss. If no instruction key is found in the tokenized instance, it emits this UserWarning and the instance's prompt tokens are not excluded/marked as expected, effectively ignored in loss calculation structure. It fires when the human-turn marker is missing from the truncated or malformed token sequence.

Source

Thrown at omnilmm/train/train_utils.py:128

        if len(response_token_ids_idxs) == 0:
            warnings.warn(
                f"Could not find response key `{response_template}` in the "
                f'following instance: @===>{tokenizer.decode(res_input_ids)}<===@ '
                f'Raw text is @===>{res_text}<===@'
                f'Raw source is @===>{new_source}<===@'
                f"This instance will be ignored in loss calculation. "
                f"Note, if this happens often, consider increasing the `max_seq_length`."
            )
            res_labels[:] = ignore_index

        human_token_ids = instruction_token_ids
        for human_idx in np.where(res_labels == human_token_ids[0])[0]:
            # find the indexes of the start of a human answer.
            if human_token_ids == res_labels[human_idx: human_idx + len(human_token_ids)].tolist():
                human_token_ids_idxs.append(human_idx)

        if len(human_token_ids_idxs) == 0:
            warnings.warn(
                f"Could not find instruction key `{instruction_template}` in the "
                f'following instance: @===>{tokenizer.decode(res_input_ids)}<===@ '
                f'Raw text is @===>{res_text}<===@'
                f'Raw source is @===>{new_source}<===@'
                f"This instance will be ignored in loss calculation. "
                f"Note, if this happens often, consider increasing the `max_seq_length`."
            )
            res_labels[:] = ignore_index

        for idx, (start, end) in enumerate(zip(human_token_ids_idxs, response_token_ids_idxs)):
            # Make pytorch loss function ignore all non response tokens
            if idx != 0:
                res_labels[start:end] = ignore_index
            else:
                res_labels[:end] = ignore_index

        if len(response_token_ids_idxs) < len(human_token_ids_idxs):
            res_labels[human_token_ids_idxs[-1]:] = ignore_index

View on GitHub (pinned to 7a11e2bec4)

Solutions

1. Ensure every sample passes through wrap_question_for_omni_lmm / the conversation template so the instruction marker is present.
2. Increase max_seq_length or switch truncation to truncate from the end (keep the prompt) so the instruction key is retained.
3. Decode the sample from the warning and confirm the instruction marker string exists and tokenizes to the expected ids with the current tokenizer.
4. Clean the dataset: drop or fix samples that legitimately lack a human turn.

Example fix

// before: raw assistant-only text in dataset
sample = {"text": "The image shows a cat."}
// after: wrap with the conversation template first
sample = {"text": wrap_question_for_omni_lmm("Describe this image.", tokenizer) + "The image shows a cat."}

encoded = tokenizer(text).input_ids
instr_ids = tokenizer(instruction_template, add_special_tokens=False).input_ids
assert any(encoded[i:i+len(instr_ids)] == instr_ids for i in range(len(encoded))), "instruction key missing from tokenized sample"

import warnings
with warnings.catch_warnings(record=True) as w:
    warnings.simplefilter("always")
    batch = collator(features)
    dropped = [str(x.message) for x in w if 'Could not find instruction key' in str(x.message)]
if dropped:
    logging.warning("%d samples lack the instruction marker; re-wrap with wrap_question_for_omni_lmm", len(dropped))

When it happens

Trigger: Calling the training collator with a sample whose res_input_ids lacks instruction_template's token ids — e.g. the conversation text has no human/USER marker, or max_seq_length truncation removed the beginning/human turn, or tokenizer change broke the marker's tokenization.

Common situations: Dataset rows containing only assistant output without the wrapped question (skipped wrap_question_for_omni_lmm); truncation strategy trimming from the left and dropping the human marker; response_template masking consumed/overlapping positions; tokenizer upgrade changing special-token ids.

Related errors

• Could not find response key `{response_template}` in the fol

AI-assisted analysis of OpenBMB/MiniCPM-V@7a11e2bec4 (2026-08-30). Data as JSON: /api/errors/60da026381d551ae. Report an issue: GitHub.