hiyouga/LlamaFactory · error · ValueError

{self.model.config.model_type} requires 3D position ids for

Error message

{self.model.config.model_type} requires 3D position ids for mrope.

What it means

After collation, if the model's config.model_type is in MROPE_MODELS (qwen2-vl family using 3D mrope position ids) but features lacks position_ids or its dim is not 3, the collator raises ValueError. The 3D tensor (3, batch, seq) encodes temporal/height/width positions required by mrope attention; a 2D or missing position_ids means the mrope path never ran (e.g. rope func absent) or produced the wrong rank.

Source

Thrown at src/llamafactory/data/collator.py:468

                    packing_params_list,
                    batch_imglens,
                    batch_vidlens,
                    batch_audlens,
                    has_dummy_image,
                )

            # For transformers compatibility, after https://github.com/huggingface/transformers/issues/39400
            if features["position_ids"].dim() == 3:
                features["position_ids"] = torch.cat(
                    [features["position_ids"][0].unsqueeze(0), features["position_ids"]], dim=0
                )

        if (
            self.model is not None
            and getattr(self.model.config, "model_type", None) in MROPE_MODELS
            and ("position_ids" not in features or features["position_ids"].dim() != 3)
        ):
            raise ValueError(f"{self.model.config.model_type} requires 3D position ids for mrope.")

        if (
            "cross_attention_mask" in mm_inputs and mm_inputs["cross_attention_mask"].dtype != torch.bool
        ):  # for mllama inputs when pad_to_multiple_of is enabled
            cross_attention_mask = mm_inputs.pop("cross_attention_mask")
            seq_len = features["input_ids"].size(1)
            orig_len = cross_attention_mask.size(1)
            mm_inputs["cross_attention_mask"] = F.pad(cross_attention_mask, (0, 0, 0, 0, 0, seq_len - orig_len))

        if is_moss_vl:
            mm_inputs = self.template.mm_plugin.post_process_mossvl_inputs(features, mm_inputs, self.processor)

        features.update(mm_inputs)

        if "image_bound" in features:  # for minicpmv inputs
            bsz, seq_length = features["input_ids"].shape
            features["position_ids"] = torch.arange(seq_length).long().repeat(bsz, 1)
            return {"data": features, "input_ids": features["input_ids"], "labels": features["labels"]}

View on GitHub (pinned to f28afaf635)

Solutions

Confirm model_type of your checkpoint is what you expect (print model.config.model_type) and that it matches the template family.
Upgrade/downgrade transformers to a version supported by your LlamaFactory release so get_rope_index is found (probe order: model, then model.model).
Avoid wrapping the model in custom classes that shadow get_rope_index; expose or delegate the attribute.
Ensure the multimodal collator (not a plain text collator) is used for mrope models.

Defensive patterns

Strategy: validation

Validate before calling

from llamafactory.data.collator import MROPE_MODELS
if getattr(model.config, "model_type", None) in MROPE_MODELS:
    rope_ok = hasattr(model, "get_rope_index") or (hasattr(model, "model") and hasattr(model.model, "get_rope_index"))
    assert rope_ok, "mrope model but get_rope_index not found — check transformers version"

Prevention

Pin transformers to the version matrix tested by LlamaFactory for qwen2-vl family.
Do not wrap models in custom containers that hide get_rope_index.
Match template family to checkpoint family.

When it happens

Trigger: Training/evaluating an MROPE_MODELS model where get_rope_func resolved to None (model wrapper lacking get_rope_index on both model and model.model), or a code path replaced position_ids with a 2D tensor before this check; also packing/dummy-image paths that bypassed mrope computation.

Common situations: transformers version change moving get_rope_index off the expected attribute (the collator probes model.get_rope_index then model.model.get_rope_index); wrapping the model in a custom module that hides the attribute; using a qwen2-vl template with a non-qwen2-vl checkpoint or vice versa.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/5a44e12f7b5e6dff. Report an issue: GitHub.