{"record":{"id":"e565b1f31fb01397","repo":"hiyouga/LlamaFactory","slug":"the-length-of-packed-example-should-be-identical-t","errorCode":null,"errorMessage":"The length of packed example should be identical to the cutoff length.","messagePattern":"The length of packed example should be identical to the cutoff length\\.","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"src/llamafactory/data/processor/supervised.py","lineNumber":232,"sourceCode":"                    packed_attention_masks += [i + 1] * len(batch_input_ids[index])  # start from 1\n                else:\n                    packed_attention_masks += [1] * len(batch_input_ids[index])\n\n            if len(packed_input_ids) < self.data_args.cutoff_len + 1:  # avoid flash_attn drops attn mask\n                pad_length = self.data_args.cutoff_len - len(packed_input_ids) + 1\n                packed_input_ids += [self.tokenizer.pad_token_id] * pad_length\n                packed_position_ids += [0] * pad_length\n                packed_labels += [IGNORE_INDEX] * pad_length\n                if self.data_args.neat_packing:\n                    packed_attention_masks += [0] * pad_length\n                else:\n                    packed_attention_masks += [1] * pad_length  # more efficient flash_attn\n\n                if requires_packing_params:\n                    sequence_boundaries.append(sequence_boundaries[-1] + pad_length)\n\n            if len(packed_input_ids) != self.data_args.cutoff_len + 1:\n                raise ValueError(\"The length of packed example should be identical to the cutoff length.\")\n\n            model_inputs[\"input_ids\"].append(packed_input_ids)\n            if requires_packing_params:\n                packing_params = PackingParams(\n                    sequence_boundaries=sequence_boundaries,\n                    image_subseq_ids=image_subseq_ids or [MAX_SU_SEQ_IDX],  # avoid dataset concat error\n                    video_subseq_ids=video_subseq_ids or [MAX_SU_SEQ_IDX],\n                    audio_subseq_ids=audio_subseq_ids or [MAX_SU_SEQ_IDX],\n                    right_padding_length=pad_length,\n                )\n                model_inputs[\"packing_params\"].append(asdict(packing_params))\n\n            model_inputs[\"attention_mask\"].append(packed_attention_masks)\n            model_inputs[\"position_ids\"].append(packed_position_ids)\n            model_inputs[\"labels\"].append(packed_labels)\n            model_inputs[\"images\"].append(packed_images or None)\n            model_inputs[\"videos\"].append(packed_videos or None)\n            model_inputs[\"audios\"].append(packed_audios or None)","sourceCodeStart":214,"sourceCodeEnd":250,"githubUrl":"https://github.com/hiyouga/LlamaFactory/blob/f28afaf6355af515454dfb16c97d728307c93897/src/llamafactory/data/processor/supervised.py#L214-L250","documentation":"During packed SFT preprocessing, every knapsack bucket must end up exactly cutoff_len + 1 tokens long (the +1 guards against flash-attention dropping the attention mask). After concatenating examples and padding, the final length check fails, meaning an example longer than cutoff_len slipped into a knapsack or the padding arithmetic produced an inconsistent length. It is raised from _prepare_packed_data in the supervised processor.","triggerScenarios":"Running SFT with packing=True (or neat_packing=True) where greedy_knapsack receives a length list containing values > cutoff_len (e.g. cutoff_len reduced after tokenization, or an example whose tokenized length exceeds cutoff_len but was not filtered because a custom processor skipped the length check), or when pad_token_id is None making the pad step produce zero-length/invalid appends.","commonSituations":"Setting a small cutoff_len in the YAML while the dataset contains long multi-turn conversations; switching templates that tokenize to different lengths; using a tokenizer whose pad token setup changed after model load; version upgrades that altered the +1 flash-attn guard.","solutions":["Check that cutoff_len in your training config is at least as large as your longest tokenized sample, or raise cutoff_len (e.g. 1024 -> 4096).","Verify tokenizer.pad_token_id is not None (LlamaFactory normally sets pad=eos via fix_special_tokens); if loading a tokenizer manually, ensure it has a pad token before the processor runs.","Re-tokenize with the same template used for training: template choice changes token counts, so a sample under cutoff in one template may exceed it in another.","If the error persists, disable packing (packing: false) to identify the offending sample from the 'Dropped lengthy example' warnings, then fix the data."],"exampleFix":"# before\ncutoff_len: 1024\npacking: true\n\n# after\ncutoff_len: 4096\npacking: true","handlingStrategy":"validation","validationCode":"# Before training, verify every tokenized sample fits the cutoff\nfrom transformers import AutoTokenizer\nfrom llamafactory.data.template import get_template_and_fix_tokenizer\nfrom llamafactory.hparams import DataArguments\n\ntok = AutoTokenizer.from_pretrained(model_path)\ntmpl = get_template_and_fix_tokenizer(tok, DataArguments(template=\"qwen\"))\nfor sample in my_samples:\n    ids, _ = tmpl._encode_data_example(...)  # or encode via the processor\n    assert len(ids) <= cutoff_len, f\"sample too long: {len(ids)} > {cutoff_len}\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set cutoff_len at or above the tokenized length of your longest sample (measure, don't guess).","Keep the same template between data inspection and training so token counts match.","Never leave tokenizer.pad_token_id None when packing; rely on fix_special_tokens or set it manually."],"tags":["data","packing","sft","config"],"backgroundTag":null,"analyzedSha":"f28afaf6355af515454dfb16c97d728307c93897","analyzedAt":"2026-08-14T21:57:28.298Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}