huggingface/transformers · error · ValueError

`nbits` for `quanto` backend has to be one of [`2`, `4`] but

Error message

`nbits` for `quanto` backend has to be one of [`2`, `4`] but got {self.nbits}

What it means

ValueError in QuantoQuantizedLayer.__init__ validating nbits: the optimum-quanto backend only supports 2-bit and 4-bit KV-cache quantization (qint2/qint4 are the only qtypes wired up). Any other value — 8, 3, 1, 16 — is rejected at cache construction time before any tensor is quantized.

Source

Thrown at src/transformers/cache_utils.py:805

            q_group_size=q_group_size,
            residual_length=residual_length,
        )

        # We need to import quanto here to avoid circular imports due to optimum/quanto/models/transformers_models.py
        if not is_optimum_quanto_available():
            raise ImportError(
                "You need to install optimum-quanto in order to use KV cache quantization with optimum-quanto "
                "backend. Please install it via  with `pip install optimum-quanto`"
            )
        elif is_quanto_greater("0.2.5", accept_dev=True):
            from optimum.quanto import MaxOptimizer, qint2, qint4
        else:
            raise ImportError(
                "You need optimum-quanto package version to be greater or equal than 0.2.5 to use `QuantoQuantizedLayer`. "
            )

        if self.nbits not in [2, 4]:
            raise ValueError(f"`nbits` for `quanto` backend has to be one of [`2`, `4`] but got {self.nbits}")

        if self.axis_key not in [0, -1]:
            raise ValueError(f"`axis_key` for `quanto` backend has to be one of [`0`, `-1`] but got {self.axis_key}")

        if self.axis_value not in [0, -1]:
            raise ValueError(
                f"`axis_value` for `quanto` backend has to be one of [`0`, `-1`] but got {self.axis_value}"
            )

        self.qtype = qint4 if self.nbits == 4 else qint2
        self.optimizer = MaxOptimizer()  # hardcode as it's the only one for per-channel quantization

    def _quantize(self, tensor, axis):
        from optimum.quanto import quantize_weight

        scale, zeropoint = self.optimizer(tensor, self.qtype, axis, self.q_group_size)
        qtensor = quantize_weight(tensor, self.qtype, axis, scale, zeropoint, self.q_group_size)
        return qtensor

View on GitHub (pinned to a597f97485)

Solutions

  1. Use nbits=2 or nbits=4 with backend='quanto'
  2. If you need 8-bit or 3-bit, switch backend to 'hqq' (supports 1/2/3/4/8)
  3. Cast nbits to int if it comes from a config file as a string

Example fix

# before
config = QuantoQuantizedCacheConfig(nbits=8, backend='quanto')

# after
config = QuantoQuantizedCacheConfig(nbits=4, backend='quanto')
# or use HQQ for 8-bit:
config = HqqQuantizedCacheConfig(nbits=8, backend='hqq')
Defensive patterns

Strategy: validation

Validate before calling

assert config.backend == 'quanto' and int(config.nbits) in (2, 4), 'quanto backend supports only 2/4 bits'

Type guard

def is_valid_quanto_nbits(nbits) -> bool:
    return isinstance(nbits, int) and nbits in (2, 4)

Prevention

When it happens

Trigger: QuantoQuantizedCacheConfig(nbits=8, backend='quanto'), nbits=3, or copying an HQQ-style config (which allows 1/2/3/4/8) and switching backend='quanto' without adjusting nbits; also passing nbits as a string ('4') raises the same error.

Common situations: Migrating configs between HQQ and quanto backends; assuming 8-bit is supported because other transformers quantizers offer it; YAML/JSON configs where nbits defaults to something other than 2/4.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/76e10fdf09f42ef6. Report an issue: GitHub.