hiyouga/LlamaFactory · error · ValueError

Unsupported torch version detected: torch 2.9.x with Conv3D.

Error message

Unsupported torch version detected: torch 2.9.x with Conv3D. This combination is known to cause severe performance regression. Please downgrade torch to <2.9 or remove Conv3D. See https://github.com/pytorch/pytorch/issues/166122

What it means

LlamaFactory refuses to load/train a model when torch 2.9.x is installed together with any torch.nn.Conv3d module in the model. Upstream PyTorch issue 166122 documents a severe performance regression for Conv3D kernels in the 2.9 series. The check runs after model loading in load_tokenizer_module/load flow in loader.py and raises a hard ValueError so the user does not silently train at degraded speed.

Source

Thrown at src/llamafactory/model/loader.py:200

    if add_valuehead:
        model = AutoModelForCausalLMWithValueHead.from_pretrained(model)
        patch_valuehead_model(model)

        if model_args.adapter_name_or_path is not None:
            vhead_path = model_args.adapter_name_or_path[-1]
        else:
            vhead_path = model_args.model_name_or_path

        vhead_params = load_valuehead_params(vhead_path, model_args)
        if vhead_params is not None:
            model.load_state_dict(vhead_params, strict=False)
            logger.info_rank0(f"Loaded valuehead from checkpoint: {vhead_path}")

    # Conv3D is not recommended when using torch 2.9.x
    if is_torch_version_greater_than("2.9.0") and not is_torch_version_greater_than("2.10.0"):
        if any(isinstance(m, torch.nn.Conv3d) for m in model.modules()):
            raise ValueError(
                "Unsupported torch version detected: torch 2.9.x with Conv3D. "
                "This combination is known to cause severe performance regression. "
                "Please downgrade torch to <2.9 or remove Conv3D. "
                "See https://github.com/pytorch/pytorch/issues/166122"
            )

    if not is_trainable:
        model.requires_grad_(False)
        model.eval()
    else:
        model.train()

    # Borrowing the kernel plugins ability of v1 to temporarily apply the NPU fusion operator to v0,
    # it is turned off by default, and can be discarded after the transition period ends.
    if model_args.use_v1_kernels and is_trainable:
        logger.warning_rank0(
            "You are try to using future feature about kernels, please note that this feature "
            "is not supported for all models. If get any error, please disable this feature, or report the issue."

View on GitHub (pinned to f28afaf635)

Solutions

1. Downgrade torch to <2.9 (e.g. pip install 'torch<2.9') and keep other CUDA deps compatible.
2. Upgrade to torch >=2.10 where the regression is fixed, once LlamaFactory/transformers support it.
3. Use a model variant without Conv3d modules (pure 2D vision encoder) if you must stay on torch 2.9.x.
4. If you have benchmarked your exact workload and accept the regression, patch the guard locally — but this leaves you exposed to the perf bug.

Example fix

# before (env)
# torch==2.9.0 + video model with Conv3d -> ValueError

# after
pip install "torch<2.9"
# or in pyproject/requirements: torch>=2.6,<2.9

import torch
from packaging.version import Version

def torch_conv3d_ok() -> bool:
    v = Version(torch.__version__.split("+")[0])
    return not (Version("2.9.0") <= v < Version("2.10.0"))

assert torch_conv3d_ok(), "torch 2.9.x detected; downgrade to <2.9 or upgrade to >=2.10 before loading Conv3D models"

When it happens

Trigger: Loading a model that contains nn.Conv3d layers (typically video/multi-modal models, e.g. video-Llava-style encoders) while torch.__version__ is >=2.9.0 and <2.10.0. The check is any(isinstance(m, torch.nn.Conv3d) for m in model.modules()) combined with is_torch_version_greater_than('2.9.0') and not is_torch_version_greater_than('2.10.0').

Common situations: Fresh environment installs that pull the latest torch (2.9.x) while fine-tuning a video-LLM whose encoder uses Conv3d. CI images updated to torch 2.9 suddenly failing on models that worked before. Mixed vision towers that include 3D convolutions.

Related errors

• Processor did not emit {modality} placeholder tokens for the
• Please upgrade `transformers` to 4.34.0
• Unable to process key {key}
• Invalid length
• Invalid input type {input_item.type}.

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/f075cad8c77c25a8. Report an issue: GitHub.