{"record":{"id":"9a86f90ca95eed09","repo":"unslothai/unsloth","slug":"features-features-not-divisible-by-convrot-group","errorCode":null,"errorMessage":"features {features} not divisible by ConvRot group {group_size}","messagePattern":"features (.+?) not divisible by ConvRot group (.+?)","errorType":"exception","errorClass":"ValueError","httpStatus":400,"severity":"error","filePath":"studio/backend/core/inference/diffusion_convrot.py","lineNumber":141,"sourceCode":"        dtype = dtype,\n        device = device,\n    )\n    h = h4\n    current = 4\n    while current < size:\n        h = torch.kron(h, h4)\n        current *= 4\n    h = h / (size**0.5)\n    _HADAMARD_CACHE[key] = h\n    return h\n\n\ndef rotate_convrot_activation(x: Any, h: Any, group_size: int) -> Any:\n    \"\"\"``x @ H`` blockwise over the last dimension.\"\"\"\n    shape = x.shape\n    features = shape[-1]\n    if features % group_size != 0:\n        raise ValueError(f\"features {features} not divisible by ConvRot group {group_size}\")\n    grouped = x.reshape(-1, features // group_size, group_size)\n    return grouped.matmul(h.to(dtype = x.dtype, device = x.device)).reshape(shape)\n\n\ndef rotate_convrot_weight_(module: Any, group_size: int) -> None:\n    \"\"\"``W <- W @ blockdiag(H).T`` in place, accumulated in float32 and cast back.\n\n    float32 regardless of the stored dtype: each output element becomes a ``group_size``-term dot\n    product, and accumulating that in bfloat16 would spend a visible part of the error budget the\n    rotation exists to save. The offline half only runs once, so the upcast is free.\"\"\"\n    import torch\n\n    weight = module.weight.data\n    out_features, in_features = weight.shape\n    if in_features % group_size:\n        raise ValueError(\n            f\"in_features {in_features} is not divisible by the ConvRot group {group_size}\"\n        )","sourceCodeStart":123,"sourceCodeEnd":159,"githubUrl":"https://github.com/unslothai/unsloth/blob/203007d19051dcd2ae33876786d117c99f6b0368/studio/backend/core/inference/diffusion_convrot.py#L123-L159","documentation":"rotate_convrot_activation() applies x @ H blockwise over the last dimension, which requires the feature dimension to be an exact multiple of the Hadamard group size. A non-divisible feature count cannot be reshaped into blocks and raises before matmul.","triggerScenarios":"Calling rotate_convrot_activation(x, h, group_size) where x.shape[-1] % group_size != 0 — e.g. 70 features with group 16, or a Linear whose in_features was padded to a multiple of 8 (common for quantization) but not to the ConvRot group.","commonSituations":"Mixing quantization padding (pad to 256/multiple-of-8) with a ConvRot group of 64 on models with unusual hidden sizes; applying the online rotation to a tensor that skipped the apply_small_m_padding step; mismatched group size between checkpoint metadata and runtime config.","solutions":["Ensure in_features of every rotated layer is a multiple of group_size (pad the Linear first, e.g. apply_small_m_padding before/after per the documented ordering)","Choose a smaller power-of-four group (16 or 4) that divides the feature count","Exclude the offending layer from the rotation fqn list when rebuilding the checkpoint"],"exampleFix":"# before\nrotate_convrot_activation(x, h, group_size=64)  # x[..., 70]: 70 % 64 != 0\n\n# after\n# pad Linear in_features to a multiple of the group first\napply_small_m_padding(transformer, multiple=64)\nrotate_convrot_activation(x, h, group_size=64)","handlingStrategy":"validation","validationCode":"def activation_rotatable(x, group_size: int) -> bool:\n    return x.shape[-1] % group_size == 0","typeGuard":null,"tryCatchPattern":"try:\n    y = rotate_convrot_activation(x, h, group_size)\nexcept ValueError as e:\n    raise ModelShapeError(str(e)) from e  # config bug: pad features or shrink the group","preventionTips":["Pad feature dimensions to a multiple of the ConvRot group before the offline build","Keep checkpoint-recorded group sizes and runtime config in sync","Fail the whole build on the first non-divisible layer instead of skipping it — a half-rotated model renders wrong silently"],"tags":["convrot","shape-mismatch","tensor","quantization"],"backgroundTag":null,"analyzedSha":"203007d19051dcd2ae33876786d117c99f6b0368","analyzedAt":"2026-08-15T02:48:39.846Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}