unslothai/unsloth · error · ValueError

in_features {in_features} is not divisible by the ConvRot gr

Error message

in_features {in_features} is not divisible by the ConvRot group {group_size}

What it means

rotate_convrot_weight_() must rewrite each weight row as W @ blockdiag(H).T, which requires the Linear's in_features to be an exact multiple of the ConvRot group size. A non-divisible input dimension cannot be block-rotated and the offline rotation aborts before touching the weight.

Source

Thrown at studio/backend/core/inference/diffusion_convrot.py:157

    features = shape[-1]
    if features % group_size != 0:
        raise ValueError(f"features {features} not divisible by ConvRot group {group_size}")
    grouped = x.reshape(-1, features // group_size, group_size)
    return grouped.matmul(h.to(dtype = x.dtype, device = x.device)).reshape(shape)


def rotate_convrot_weight_(module: Any, group_size: int) -> None:
    """``W <- W @ blockdiag(H).T`` in place, accumulated in float32 and cast back.

    float32 regardless of the stored dtype: each output element becomes a ``group_size``-term dot
    product, and accumulating that in bfloat16 would spend a visible part of the error budget the
    rotation exists to save. The offline half only runs once, so the upcast is free."""
    import torch

    weight = module.weight.data
    out_features, in_features = weight.shape
    if in_features % group_size:
        raise ValueError(
            f"in_features {in_features} is not divisible by the ConvRot group {group_size}"
        )
    h = build_convrot_hadamard(group_size, device = weight.device, dtype = torch.float32)
    rotated = torch.matmul(
        weight.float().reshape(out_features, in_features // group_size, group_size), h.T
    ).reshape(out_features, in_features)
    module.weight.data = rotated.to(weight.dtype)


@lru_cache(maxsize = None)
def convrot_linear_class() -> Any:
    """The ``nn.Linear`` subclass that rotates its input, built lazily so importing this module
    never imports torch, and built exactly ONCE.

    The cache is not a micro-optimisation, it is the difference between one compiled graph and
    dozens. A class defined inside a function is a NEW class object on every call, and
    ``torch.compile`` guards each frame on ``___check_type_id`` of the modules it closes over. So
    handing every rotated projection its own ConvRotLinear made each one look like a different

View on GitHub (pinned to 203007d190)

Solutions

Pick a power-of-four group that divides in_features (64, 16, or 4 — try the largest that divides)
Pad the Linear's in_features to a multiple of the group (apply_small_m_padding) before the offline rotation
Drop the non-divisible layer from the fqn list; the builder records only the fqns actually rotated

Example fix

# before
rotate_convrot_weight_(module, group_size=64)  # module.in_features = 70

# after
group = next(g for g in (64, 16, 4) if module.in_features % g == 0)
rotate_convrot_weight_(module, group_size=group)

Defensive patterns

Strategy: validation

Validate before calling

def linear_rotatable(module, group_size: int) -> bool:
    return isinstance(module, nn.Linear) and module.in_features % group_size == 0

Try / catch

try:
    rotate_convrot_weight_(module, group_size)
except ValueError as e:
    raise BuildError(f"{fqn}: {e}") from e  # builder must record only what it actually rotated

Prevention

Filter the fqn list up front: only nn.Linear modules with in_features divisible by the group
Run small-m padding before rotation so widths are multiples of the group
Choose the largest power-of-four group that divides every target's in_features

When it happens

Trigger: Calling rotate_convrot_weight_(module, group_size) on an nn.Linear whose weight.shape[1] % group_size != 0; reached from apply_rotation/rotate_weights over an fqn list during checkpoint build.

Common situations: Rotating a model whose hidden size is not divisible by the chosen group (e.g. hidden 11008 with group 16 is fine, but 70 x group 64 is not); forgetting to run small-m padding before weight rotation; group size from a different model's config.

Related errors

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/2d50b80e4cb83c95. Report an issue: GitHub.