{"record":{"id":"60da026381d551ae","repo":"OpenBMB/MiniCPM-V","slug":"could-not-find-instruction-key-instruction-templ","errorCode":null,"errorMessage":"Could not find instruction key `{instruction_template}` in the following instance: @===>{tokenizer.decode(res_input_ids)}<===@ Raw text is @===>{res_text}<===@Raw source is @===>{new_source}<===@This instance will be ignored in loss calculation. Note, if this happens often, consider increasing the `max_seq_length`.","messagePattern":"Could not find instruction key `(.+?)` in the following instance: @===>(.+?)<===@ Raw text is @===>(.+?)<===@Raw source is @===>(.+?)<===@This instance will be ignored in loss calculation\\. Note, if this happens often, consider increasing the `max_seq_length`\\.","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"omnilmm/train/train_utils.py","lineNumber":128,"sourceCode":"        if len(response_token_ids_idxs) == 0:\n            warnings.warn(\n                f\"Could not find response key `{response_template}` in the \"\n                f'following instance: @===>{tokenizer.decode(res_input_ids)}<===@ '\n                f'Raw text is @===>{res_text}<===@'\n                f'Raw source is @===>{new_source}<===@'\n                f\"This instance will be ignored in loss calculation. \"\n                f\"Note, if this happens often, consider increasing the `max_seq_length`.\"\n            )\n            res_labels[:] = ignore_index\n\n        human_token_ids = instruction_token_ids\n        for human_idx in np.where(res_labels == human_token_ids[0])[0]:\n            # find the indexes of the start of a human answer.\n            if human_token_ids == res_labels[human_idx: human_idx + len(human_token_ids)].tolist():\n                human_token_ids_idxs.append(human_idx)\n\n        if len(human_token_ids_idxs) == 0:\n            warnings.warn(\n                f\"Could not find instruction key `{instruction_template}` in the \"\n                f'following instance: @===>{tokenizer.decode(res_input_ids)}<===@ '\n                f'Raw text is @===>{res_text}<===@'\n                f'Raw source is @===>{new_source}<===@'\n                f\"This instance will be ignored in loss calculation. \"\n                f\"Note, if this happens often, consider increasing the `max_seq_length`.\"\n            )\n            res_labels[:] = ignore_index\n\n        for idx, (start, end) in enumerate(zip(human_token_ids_idxs, response_token_ids_idxs)):\n            # Make pytorch loss function ignore all non response tokens\n            if idx != 0:\n                res_labels[start:end] = ignore_index\n            else:\n                res_labels[:end] = ignore_index\n\n        if len(response_token_ids_idxs) < len(human_token_ids_idxs):\n            res_labels[human_token_ids_idxs[-1]:] = ignore_index","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/OpenBMB/MiniCPM-V/blob/7a11e2bec4c422236eba422f872dca5712d5c7ef/omnilmm/train/train_utils.py#L110-L146","documentation":"The counterpart of error 14: omni_preprocess scans res_labels for the instruction (human/USER) template token ids to mask out prompt tokens from the loss. If no instruction key is found in the tokenized instance, it emits this UserWarning and the instance's prompt tokens are not excluded/marked as expected, effectively ignored in loss calculation structure. It fires when the human-turn marker is missing from the truncated or malformed token sequence.","triggerScenarios":"Calling the training collator with a sample whose res_input_ids lacks instruction_template's token ids — e.g. the conversation text has no human/USER marker, or max_seq_length truncation removed the beginning/human turn, or tokenizer change broke the marker's tokenization.","commonSituations":"Dataset rows containing only assistant output without the wrapped question (skipped wrap_question_for_omni_lmm); truncation strategy trimming from the left and dropping the human marker; response_template masking consumed/overlapping positions; tokenizer upgrade changing special-token ids.","solutions":["Ensure every sample passes through wrap_question_for_omni_lmm / the conversation template so the instruction marker is present.","Increase max_seq_length or switch truncation to truncate from the end (keep the prompt) so the instruction key is retained.","Decode the sample from the warning and confirm the instruction marker string exists and tokenizes to the expected ids with the current tokenizer.","Clean the dataset: drop or fix samples that legitimately lack a human turn."],"exampleFix":"// before: raw assistant-only text in dataset\nsample = {\"text\": \"The image shows a cat.\"}\n// after: wrap with the conversation template first\nsample = {\"text\": wrap_question_for_omni_lmm(\"Describe this image.\", tokenizer) + \"The image shows a cat.\"}","handlingStrategy":"validation","validationCode":"encoded = tokenizer(text).input_ids\ninstr_ids = tokenizer(instruction_template, add_special_tokens=False).input_ids\nassert any(encoded[i:i+len(instr_ids)] == instr_ids for i in range(len(encoded))), \"instruction key missing from tokenized sample\"","typeGuard":null,"tryCatchPattern":"import warnings\nwith warnings.catch_warnings(record=True) as w:\n    warnings.simplefilter(\"always\")\n    batch = collator(features)\n    dropped = [str(x.message) for x in w if 'Could not find instruction key' in str(x.message)]\nif dropped:\n    logging.warning(\"%d samples lack the instruction marker; re-wrap with wrap_question_for_omni_lmm\", len(dropped))","preventionTips":["Route every training sample through wrap_question_for_omni_lmm so the human marker is always present.","Truncate from the end (keep the prompt) rather than the beginning of the sequence.","Pin the tokenizer version in requirements so special-token ids stay stable."],"tags":["training","tokenization","loss-masking","truncation"],"backgroundTag":"instruction-template-not-found","analyzedSha":"7a11e2bec4c422236eba422f872dca5712d5c7ef","analyzedAt":"2026-08-30T09:49:48.273Z","schemaVersion":2},"datasetVersion":"2026-08-30T13:17:10.514Z"}