hiyouga/LlamaFactory · error · ValueError

{self.class.name} does not support gradient checkpoi

Error message

{self.__class__.__name__} does not support gradient checkpointing.

What it means

LlamaFactory monkey-patches transformers' _gradient_checkpointing_enable (checkpointing.py:135) to support custom checkpointing funcs and block-wise optimizers. Before enabling, it checks self.supports_gradient_checkpointing on the PreTrainedModel; if the model class declares no gradient-checkpointing support, it raises ValueError. This mirrors transformers' own behavior but surfaces during patching/enable when the training args request gradient_checkpointing=true.

Source

Thrown at src/llamafactory/model/model_utils/checkpointing.py:135

        else:
            return func(*args, **kwargs)

    return custom_gradient_checkpointing_func


def _gradient_checkpointing_enable(
    self: "PreTrainedModel",
    gradient_checkpointing_kwargs: Optional[dict[str, Any]] = None,
    use_unsloth_gc: bool = False,
) -> None:
    r"""Activates gradient checkpointing for the current model.

    Modification of the original method to enable gradient checkpointing for block-wise optimizer.
    """
    from torch.utils.checkpoint import checkpoint

    if not self.supports_gradient_checkpointing:
        raise ValueError(f"{self.__class__.__name__} does not support gradient checkpointing.")

    if gradient_checkpointing_kwargs is None:
        gradient_checkpointing_kwargs = {"use_reentrant": True}

    if use_unsloth_gc:
        gradient_checkpointing_func = get_unsloth_gradient_checkpointing_func()
    else:
        gradient_checkpointing_func = partial(checkpoint, **gradient_checkpointing_kwargs)

    gradient_checkpointing_func = get_custom_gradient_checkpointing_func(gradient_checkpointing_func)
    if "value" in inspect.signature(self._set_gradient_checkpointing).parameters:  # old GC format
        self.apply(partial(self._set_gradient_checkpointing, value=True))
        self.enable_input_require_grads()
        logger.warning_rank0_once("You are using the old GC format, some features (e.g. BAdam) will be invalid.")
    else:  # have already enabled input require gradients
        self._set_gradient_checkpointing(enable=True, gradient_checkpointing_func=gradient_checkpointing_func)

View on GitHub (pinned to f28afaf635)

Solutions

Disable gradient checkpointing in the training config (gradient_checkpointing: false) — usually only viable if VRAM is sufficient.
If it is your own model class, set supports_gradient_checkpointing = True on the PreTrainedModel subclass and ensure layers use checkpoint-able modules.
Check whether a newer transformers/LlamaFactory version adds GC support for that model_type.
Switch to a supported model variant or reduce batch/sequence length to fit without checkpointing.

Example fix

# before (yaml)
gradient_checkpointing: true   # custom model without GC support -> ValueError

# after (yaml)
gradient_checkpointing: false
per_device_train_batch_size: 1

Defensive patterns

Strategy: validation

Validate before calling

model = AutoModelForCausalLM.from_pretrained(name, trust_remote_code=True)
if training_args.gradient_checkpointing:
    assert getattr(model, "supports_gradient_checkpointing", False), (
        f"{model.__class__.__name__} lacks GC support; disable gradient_checkpointing"
    )

Prevention

Check supports_gradient_checkpointing right after loading, before trainer setup.
For custom model classes, declare supports_gradient_checkpointing = True and test checkpointing explicitly.

When it happens

Trigger: Running training with gradient_checkpointing: true on a model class whose supports_gradient_checkpointing attribute is False (custom models, some multimodal wrappers, models converted without the flag). Called from _enable_gradient_checkpointing during trainer/model setup.

Common situations: Fine-tuning a custom or newly added model architecture that forgot to set supports_gradient_checkpointing = True; loading a community model whose modeling file does not declare GC support; using a wrapper model class not registered as GC-capable.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/325adff8ecbc573d. Report an issue: GitHub.