hiyouga/LlamaFactory · error · RuntimeError

Omni models are not supported for packed sequences for now.

Error message

Omni models are not supported for packed sequences for now.

What it means

In the multimodal collator's __call__, when packed sequences are detected (any sample's packing_params has >2 sequence boundaries) and the model is an omni model (audio+text like qwen2.5-omni), a RuntimeError is raised: the rope position-id computation for packed sequences (_compute_rope_position_ids_with_packing) is not implemented for omni inputs.

Source

Thrown at src/llamafactory/data/collator.py:445

            "qwen2_5_omni_thinker",
            "qwen3_omni_moe_thinker",
        ]

        if self.get_rope_func is not None:
            # for mmrope situation, we should calculate position_ids and rope_deltas per sample.
            # When neat_packing is on, each sample has packing_params; None means no packing for that sample.
            boundaries_list = [p.get("sequence_boundaries") if p is not None else None for p in packing_params_list]
            has_packing = any(b is not None and len(b) > 2 for b in boundaries_list)
            if has_dummy_image and has_packing:
                # FIXME: too tricky, need to be refactored @kuangdd
                features["has_dummy_image"] = True

            # When fake image/audio was injected, sequence_boundaries no longer match the tensor; use non-packing path.
            if not has_packing:
                self._compute_rope_position_ids(features, mm_inputs)
            else:
                if is_omni:  # TODO: support omni models for packed sequences @kuangdd
                    raise RuntimeError("Omni models are not supported for packed sequences for now.")

                self._compute_rope_position_ids_with_packing(
                    features,
                    mm_inputs,
                    packing_params_list,
                    batch_imglens,
                    batch_vidlens,
                    batch_audlens,
                    has_dummy_image,
                )

            # For transformers compatibility, after https://github.com/huggingface/transformers/issues/39400
            if features["position_ids"].dim() == 3:
                features["position_ids"] = torch.cat(
                    [features["position_ids"][0].unsqueeze(0), features["position_ids"]], dim=0
                )

        if (

View on GitHub (pinned to f28afaf635)

Solutions

1. Set packing: false (and neat_packing: false if set) in the training YAML for omni models.
2. If you need packing for omni, wait for/watch the upstream TODO (support omni models for packed sequences) or contribute the omni branch in _compute_rope_position_ids_with_packing.
3. Split the omni dataset out of mixed packing-enabled runs.

Example fix

# before (yaml)
packing: true
neat_packing: true

# after (yaml)
packing: false

OMNI_TYPES = {"qwen2_5_omni"}  # extend as needed
if getattr(model.config, "model_type", "") in OMNI_TYPES:
    assert not data_args.packing and not training_args.neat_packing, "omni models cannot train with packing"

When it happens

Trigger: Training an omni model (e.g. qwen2.5-omni) with packing enabled (neat_packing or default packing) so that collator batches carry packing_params, while the batch also contains multimodal inputs routed through the omni path.

Common situations: Copying a packing-enabled SFT config (written for a text or vision model) to fine-tune an audio-capable omni model; enabling efficient/packing defaults in a shared base config that omni runs also inherit.

Related errors

• Template is required for MultiModalDataCollator.
• Merged position_ids shape mismatch: got {features['position_
• {self.model.config.model_type} requires 3D position ids for
• batching_strategy={self.batching_strategy.value!r} does not
• Qwen2VL requires 3D position ids for mrope.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/84fbf921b39753f7. Report an issue: GitHub.