huggingface/transformers · error · ValueError

`axis_key` for `quanto` backend has to be one of [`0`, `-1`]

Error message

`axis_key` for `quanto` backend has to be one of [`0`, `-1`] but got {self.axis_key}

What it means

ValueError in QuantoQuantizedLayer.__init__ validating axis_key: quanto's per-channel quantization supports key tensors quantized along axis 0 (per-token/head groups) or -1 (per-channel along the hidden dim) only. Any other axis (1, 2, positive axes other than 0) is rejected because optimum.quanto's MaxOptimizer/quantize_weight cannot handle it for the KV use-case.

Source

Thrown at src/transformers/cache_utils.py:808

        # We need to import quanto here to avoid circular imports due to optimum/quanto/models/transformers_models.py
        if not is_optimum_quanto_available():
            raise ImportError(
                "You need to install optimum-quanto in order to use KV cache quantization with optimum-quanto "
                "backend. Please install it via  with `pip install optimum-quanto`"
            )
        elif is_quanto_greater("0.2.5", accept_dev=True):
            from optimum.quanto import MaxOptimizer, qint2, qint4
        else:
            raise ImportError(
                "You need optimum-quanto package version to be greater or equal than 0.2.5 to use `QuantoQuantizedLayer`. "
            )

        if self.nbits not in [2, 4]:
            raise ValueError(f"`nbits` for `quanto` backend has to be one of [`2`, `4`] but got {self.nbits}")

        if self.axis_key not in [0, -1]:
            raise ValueError(f"`axis_key` for `quanto` backend has to be one of [`0`, `-1`] but got {self.axis_key}")

        if self.axis_value not in [0, -1]:
            raise ValueError(
                f"`axis_value` for `quanto` backend has to be one of [`0`, `-1`] but got {self.axis_value}"
            )

        self.qtype = qint4 if self.nbits == 4 else qint2
        self.optimizer = MaxOptimizer()  # hardcode as it's the only one for per-channel quantization

    def _quantize(self, tensor, axis):
        from optimum.quanto import quantize_weight

        scale, zeropoint = self.optimizer(tensor, self.qtype, axis, self.q_group_size)
        qtensor = quantize_weight(tensor, self.qtype, axis, scale, zeropoint, self.q_group_size)
        return qtensor

    def _dequantize(self, qtensor):
        return qtensor.dequantize()

View on GitHub (pinned to a597f97485)

Solutions

Set axis_key to 0 or -1 for the quanto backend
If migrating from HQQ, translate axes: HQQ 0 -> quanto 0, HQQ 1 -> quanto -1 (channel axis)
Leave axis_key at the config default unless you have a measured reason to change it

Example fix

# before
config = QuantoQuantizedCacheConfig(nbits=4, axis_key=1, backend='quanto')

# after
config = QuantoQuantizedCacheConfig(nbits=4, axis_key=0, backend='quanto')

Defensive patterns

Strategy: validation

Validate before calling

assert config.backend != 'quanto' or config.axis_key in (0, -1), 'quanto axis_key must be 0 or -1'

Type guard

def is_valid_quanto_axis(axis) -> bool:
    return axis in (0, -1)

Prevention

Remember the backend axis vocabularies: quanto {0, -1} vs HQQ {0, 1}
Translate axes explicitly when porting configs between backends
Leave axis values at defaults unless benchmarks justify a change

When it happens

Trigger: QuantoQuantizedCacheConfig(axis_key=1) (or 2) with backend='quanto'; copying axis values from HQQ configs, which use a different convention (HQQ allows 0/1); passing default axis values of another cache class.

Common situations: Switching backends while keeping axis settings; porting from torch-ao quantized cache configs whose axis semantics differ; experimentation with per-head quantization axes.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/665e859be5fb80b3. Report an issue: GitHub.