unslothai/unsloth · error · ValueError

ConvRot group size must be a power of 4, got {size}

Error message

ConvRot group size must be a power of 4, got {size}

What it means

build_convrot_hadamard() constructs a scaled Hadamard matrix via repeated Kronecker products of a 4x4 base, which is only defined for sizes 4, 16, 64, ... (powers of four). Any other size raises before any tensor work.

Source

Thrown at studio/backend/core/inference/diffusion_convrot.py:120

    device: Any = "cpu",
    dtype: Any = None,
) -> Any:
    """The normalized regular Hadamard matrix ConvRot rotates by. Cached per (size, device, dtype).

    Built as ``kron(H4, H4, ...) / sqrt(size)``, which is both symmetric and orthogonal -- the
    property the offline/online pair relies on, since it means the same matrix undoes itself and
    the weight side can use ``H.T`` interchangeably with ``H``. Building directly in ``dtype`` is
    exact for every float type: the entries are +-1 and the normalizer is a power of two."""
    import torch

    if dtype is None:
        dtype = torch.float32
    key = (size, str(device), dtype)
    cached = _HADAMARD_CACHE.get(key)
    if cached is not None:
        return cached
    if not is_power_of_four(size):
        raise ValueError(f"ConvRot group size must be a power of 4, got {size}")
    h4 = torch.tensor(
        [[1, 1, 1, -1], [1, 1, -1, 1], [1, -1, 1, 1], [-1, 1, 1, 1]],
        dtype = dtype,
        device = device,
    )
    h = h4
    current = 4
    while current < size:
        h = torch.kron(h, h4)
        current *= 4
    h = h / (size**0.5)
    _HADAMARD_CACHE[key] = h
    return h


def rotate_convrot_activation(x: Any, h: Any, group_size: int) -> Any:
    """``x @ H`` blockwise over the last dimension."""
    shape = x.shape

View on GitHub (pinned to 203007d190)

Solutions

1. Use a power of four: 4, 16, 64 (the Kronecker construction cannot build anything else)
2. Validate group sizes at config load with is_power_of_four() so the mistake is caught at startup, not mid-inference
3. If a smaller/larger rotation is needed, pick the nearest power of four and pad/choose divisible in_features accordingly

Example fix

# before
h = build_convrot_hadamard(32)   # ValueError: power of two, not four

# after
h = build_convrot_hadamard(64)    # 4, 16, 64, ... only

from studio.backend.core.inference.diffusion_convrot import is_power_of_four

def hadamard_size_valid(size: int) -> bool:
    return is_power_of_four(size)  # 4, 16, 64, ...

try:
    h = build_convrot_hadamard(group_size, device=dev, dtype=dt)
except ValueError as e:
    raise ConfigError(str(e)) from e  # fail the build immediately, never fall back to a wrong matrix

When it happens

Trigger: Calling build_convrot_hadamard(size) with size not a power of four (e.g. 8, 32, 48, 0); usually reached from rotate_convrot_weight_/activation-rotation code paths that pass an unvalidated group_size.

Common situations: Copy-pasting a SpinQuant/Quarot-style config that uses group 128 (power of two, not four); tweaking DEFAULT_CONVROT_GROUPSIZE to 8 or 32; computing group size from a formula (in_features/32) that lands off the power-of-four ladder.

Related errors

• features {features} not divisible by ConvRot group {group_si
• in_features {in_features} is not divisible by the ConvRot gr
• ConvRot group size must be a power of 4, got {group_size!r}
• cannot rotate {fqn!r}: not an nn.Linear on this model
• unsupported activation rotation {kind!r} (this build impleme

AI-assisted analysis of unslothai/unsloth@203007d190 (2026-08-15). Data as JSON: /api/errors/b407326bd8537f5b. Report an issue: GitHub.