hiyouga/LlamaFactory · critical · ValueError

The length of packed example should be identical to the cuto

Error message

The length of packed example should be identical to the cutoff length.

What it means

During packed SFT preprocessing, every knapsack bucket must end up exactly cutoff_len + 1 tokens long (the +1 guards against flash-attention dropping the attention mask). After concatenating examples and padding, the final length check fails, meaning an example longer than cutoff_len slipped into a knapsack or the padding arithmetic produced an inconsistent length. It is raised from _prepare_packed_data in the supervised processor.

Source

Thrown at src/llamafactory/data/processor/supervised.py:232

                    packed_attention_masks += [i + 1] * len(batch_input_ids[index])  # start from 1
                else:
                    packed_attention_masks += [1] * len(batch_input_ids[index])

            if len(packed_input_ids) < self.data_args.cutoff_len + 1:  # avoid flash_attn drops attn mask
                pad_length = self.data_args.cutoff_len - len(packed_input_ids) + 1
                packed_input_ids += [self.tokenizer.pad_token_id] * pad_length
                packed_position_ids += [0] * pad_length
                packed_labels += [IGNORE_INDEX] * pad_length
                if self.data_args.neat_packing:
                    packed_attention_masks += [0] * pad_length
                else:
                    packed_attention_masks += [1] * pad_length  # more efficient flash_attn

                if requires_packing_params:
                    sequence_boundaries.append(sequence_boundaries[-1] + pad_length)

            if len(packed_input_ids) != self.data_args.cutoff_len + 1:
                raise ValueError("The length of packed example should be identical to the cutoff length.")

            model_inputs["input_ids"].append(packed_input_ids)
            if requires_packing_params:
                packing_params = PackingParams(
                    sequence_boundaries=sequence_boundaries,
                    image_subseq_ids=image_subseq_ids or [MAX_SU_SEQ_IDX],  # avoid dataset concat error
                    video_subseq_ids=video_subseq_ids or [MAX_SU_SEQ_IDX],
                    audio_subseq_ids=audio_subseq_ids or [MAX_SU_SEQ_IDX],
                    right_padding_length=pad_length,
                )
                model_inputs["packing_params"].append(asdict(packing_params))

            model_inputs["attention_mask"].append(packed_attention_masks)
            model_inputs["position_ids"].append(packed_position_ids)
            model_inputs["labels"].append(packed_labels)
            model_inputs["images"].append(packed_images or None)
            model_inputs["videos"].append(packed_videos or None)
            model_inputs["audios"].append(packed_audios or None)

View on GitHub (pinned to f28afaf635)

Solutions

1. Check that cutoff_len in your training config is at least as large as your longest tokenized sample, or raise cutoff_len (e.g. 1024 -> 4096).
2. Verify tokenizer.pad_token_id is not None (LlamaFactory normally sets pad=eos via fix_special_tokens); if loading a tokenizer manually, ensure it has a pad token before the processor runs.
3. Re-tokenize with the same template used for training: template choice changes token counts, so a sample under cutoff in one template may exceed it in another.
4. If the error persists, disable packing (packing: false) to identify the offending sample from the 'Dropped lengthy example' warnings, then fix the data.

Example fix

# before
cutoff_len: 1024
packing: true

# after
cutoff_len: 4096
packing: true

# Before training, verify every tokenized sample fits the cutoff
from transformers import AutoTokenizer
from llamafactory.data.template import get_template_and_fix_tokenizer
from llamafactory.hparams import DataArguments

tok = AutoTokenizer.from_pretrained(model_path)
tmpl = get_template_and_fix_tokenizer(tok, DataArguments(template="qwen"))
for sample in my_samples:
    ids, _ = tmpl._encode_data_example(...)  # or encode via the processor
    assert len(ids) <= cutoff_len, f"sample too long: {len(ids)} > {cutoff_len}"

When it happens

Trigger: Running SFT with packing=True (or neat_packing=True) where greedy_knapsack receives a length list containing values > cutoff_len (e.g. cutoff_len reduced after tokenization, or an example whose tokenized length exceeds cutoff_len but was not filtered because a custom processor skipped the length check), or when pad_token_id is None making the pad step produce zero-length/invalid appends.

Common situations: Setting a small cutoff_len in the YAML while the dataset contains long multi-turn conversations; switching templates that tokenize to different lengths; using a tokenizer whose pad token setup changed after model load; version upgrades that altered the +1 flash-attn guard.

Related errors

• `neat_packing` cannot be set as True except SFT.
• Streaming mode should have an integer val size.
• `max_samples` is incompatible with `streaming`.
• `mask_history` is incompatible with `train_on_prompt`.
• `predict_with_generate` cannot be set as True except SFT.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/e565b1f31fb01397. Report an issue: GitHub.