unslothai/unsloth · error · ValueError

features {features} not divisible by ConvRot group {group_si

Error message

features {features} not divisible by ConvRot group {group_size}

What it means

rotate_convrot_activation() applies x @ H blockwise over the last dimension, which requires the feature dimension to be an exact multiple of the Hadamard group size. A non-divisible feature count cannot be reshaped into blocks and raises before matmul.

Source

Thrown at studio/backend/core/inference/diffusion_convrot.py:141

        dtype = dtype,
        device = device,
    )
    h = h4
    current = 4
    while current < size:
        h = torch.kron(h, h4)
        current *= 4
    h = h / (size**0.5)
    _HADAMARD_CACHE[key] = h
    return h


def rotate_convrot_activation(x: Any, h: Any, group_size: int) -> Any:
    """``x @ H`` blockwise over the last dimension."""
    shape = x.shape
    features = shape[-1]
    if features % group_size != 0:
        raise ValueError(f"features {features} not divisible by ConvRot group {group_size}")
    grouped = x.reshape(-1, features // group_size, group_size)
    return grouped.matmul(h.to(dtype = x.dtype, device = x.device)).reshape(shape)


def rotate_convrot_weight_(module: Any, group_size: int) -> None:
    """``W <- W @ blockdiag(H).T`` in place, accumulated in float32 and cast back.

    float32 regardless of the stored dtype: each output element becomes a ``group_size``-term dot
    product, and accumulating that in bfloat16 would spend a visible part of the error budget the
    rotation exists to save. The offline half only runs once, so the upcast is free."""
    import torch

    weight = module.weight.data
    out_features, in_features = weight.shape
    if in_features % group_size:
        raise ValueError(
            f"in_features {in_features} is not divisible by the ConvRot group {group_size}"
        )

View on GitHub (pinned to 203007d190)

Solutions

Ensure in_features of every rotated layer is a multiple of group_size (pad the Linear first, e.g. apply_small_m_padding before/after per the documented ordering)
Choose a smaller power-of-four group (16 or 4) that divides the feature count
Exclude the offending layer from the rotation fqn list when rebuilding the checkpoint

Example fix

# before
rotate_convrot_activation(x, h, group_size=64)  # x[..., 70]: 70 % 64 != 0

# after
# pad Linear in_features to a multiple of the group first
apply_small_m_padding(transformer, multiple=64)
rotate_convrot_activation(x, h, group_size=64)

Defensive patterns

Strategy: validation

Validate before calling

def activation_rotatable(x, group_size: int) -> bool:
    return x.shape[-1] % group_size == 0

Try / catch

try:
    y = rotate_convrot_activation(x, h, group_size)
except ValueError as e:
    raise ModelShapeError(str(e)) from e  # config bug: pad features or shrink the group

Prevention

Pad feature dimensions to a multiple of the ConvRot group before the offline build
Keep checkpoint-recorded group sizes and runtime config in sync
Fail the whole build on the first non-divisible layer instead of skipping it — a half-rotated model renders wrong silently

When it happens

Trigger: Calling rotate_convrot_activation(x, h, group_size) where x.shape[-1] % group_size != 0 — e.g. 70 features with group 16, or a Linear whose in_features was padded to a multiple of 8 (common for quantization) but not to the ConvRot group.

Common situations: Mixing quantization padding (pad to 256/multiple-of-8) with a ConvRot group of 64 on models with unusual hidden sizes; applying the online rotation to a tensor that skipped the apply_small_m_padding step; mismatched group size between checkpoint metadata and runtime config.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/9a86f90ca95eed09. Report an issue: GitHub.