hiyouga/LlamaFactory · error · ValueError

`neat_packing` cannot be set as True except SFT.

Error message

`neat_packing` cannot be set as True except SFT.

What it means

neat_packing is a sequence-packing strategy (with position-id based attention separation) implemented only in the SFT supervised data processor. When stage is anything other than sft, the flag has no code path and the argument checker raises ValueError to prevent silently ignored packing.

Source

Thrown at src/llamafactory/hparams/parser.py:423

        model_args, data_args, training_args, finetuning_args, mb_args, generating_args = _parse_train_mbridge_args(
            args
        )
    else:
        model_args, data_args, training_args, finetuning_args, generating_args = _parse_train_args(args)
        finetuning_args.use_mca = False
        finetuning_args.use_megatron_bridge = False

    # Setup logging
    if training_args.should_log:
        _set_transformers_logging()

    # Check arguments
    if finetuning_args.stage != "sft":
        if training_args.predict_with_generate:
            raise ValueError("`predict_with_generate` cannot be set as True except SFT.")

        if data_args.neat_packing:
            raise ValueError("`neat_packing` cannot be set as True except SFT.")

        if data_args.train_on_prompt or data_args.mask_history:
            raise ValueError("`train_on_prompt` or `mask_history` cannot be set as True except SFT.")

    if finetuning_args.stage == "sft" and training_args.do_predict and not training_args.predict_with_generate:
        raise ValueError("Please enable `predict_with_generate` to save model predictions.")

    if finetuning_args.use_megatron_bridge:
        if finetuning_args.use_mca or finetuning_args.use_hyper_parallel:
            raise ValueError("Megatron Bridge cannot be used together with MCA or HyperParallel.")
        if finetuning_args.stage not in ["pt", "sft"]:
            raise ValueError("Megatron Bridge only supports the `pt` and `sft` stages.")
        if finetuning_args.finetuning_type not in ["full", "lora"]:
            raise ValueError("Megatron Bridge only supports `full` and `lora` finetuning.")
        if model_args.quantization_bit is not None:
            raise ValueError("Quantized models are not supported with Megatron Bridge.")
        if training_args.deepspeed is not None:
            raise ValueError("Megatron Bridge is incompatible with DeepSpeed.")

View on GitHub (pinned to f28afaf635)

Solutions

1. Remove neat_packing or set it to false for non-SFT stages.
2. If packing matters for throughput, ensure the run is stage: sft.
3. For long-sequence efficiency in DPO/RM, use cutoff_len and batching appropriate to that stage instead.

Example fix

# before
stage: dpo
neat_packing: true

# after
stage: dpo
neat_packing: false

if cfg.get("stage", "sft") != "sft" and cfg.get("neat_packing"):
    raise SystemExit("neat_packing is SFT-only; remove it for this stage")

When it happens

Trigger: A config with stage != sft (e.g. dpo, rm, ppo, pt, kto) and neat_packing: true in data_args, submitted via llamafactory-cli train. Also triggered programmatically through run_exp(dict) with the same combination.

Common situations: Copy-pasting an efficient SFT config (packing for throughput) into a DPO/RM run; upgrading from an older version where the flag was silently ignored and now validates strictly.

Related errors

• `predict_with_generate` cannot be set as True except SFT.
• `train_on_prompt` or `mask_history` cannot be set as True ex
• The length of packed example should be identical to the cuto
• Please enable `predict_with_generate` to save model predicti
• Megatron Bridge only supports the `pt` and `sft` stages.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/02db58655459c0ab. Report an issue: GitHub.