huggingface/transformers · error · ValueError

The model vocabulary size is {vocabulary_size}, but the foll

Error message

The model vocabulary size is {vocabulary_size}, but the following tokens were being biased: {invalid_biases}

What it means

Thrown by SequenceBiasLogitsProcessor._prepare_bias_variables on the first __call__ when sequence_bias contains token ids >= the model's vocabulary size (scores.shape[-1]). Biased token ids index directly into the logits tensor, so out-of-range ids would corrupt memory or fail silently; the processor checks them lazily against the actual runtime vocabulary size.

Source

Thrown at src/transformers/generation/logits_process.py:1331

                torch.tensor(sequence_bias, device=input_ids.device),
                torch.tensor(0.0, device=input_ids.device),
            )

        # 5 - apply the bias to the scores
        scores_processed = scores + bias
        return scores_processed

    def _prepare_bias_variables(self, scores: torch.FloatTensor):
        vocabulary_size = scores.shape[-1]

        # Check biased tokens out of bounds
        invalid_biases = []
        for sequence_ids in self.sequence_bias:
            for token_id in sequence_ids:
                if token_id >= vocabulary_size:
                    invalid_biases.append(token_id)
        if len(invalid_biases) > 0:
            raise ValueError(
                f"The model vocabulary size is {vocabulary_size}, but the following tokens were being biased: "
                f"{invalid_biases}"
            )

        # Precompute the bias tensors to be applied. Sequences of length 1 are kept separately, as they can be applied
        # with simpler logic.
        self.length_1_bias = torch.zeros((vocabulary_size,), dtype=torch.float, device=scores.device)
        # Extract single-token sequences and their biases
        single_token_ids = []
        single_token_biases = []
        for sequence_ids, bias in self.sequence_bias.items():
            if len(sequence_ids) == 1:
                single_token_ids.append(sequence_ids[0])
                single_token_biases.append(bias)

        if single_token_ids:  # Only if we have any single-token sequences
            self.length_1_bias[single_token_ids] = torch.tensor(single_token_biases, device=scores.device)
        self.prepared_bias_variables = True

View on GitHub (pinned to a597f97485)

Solutions

Verify every biased id with the same tokenizer used for generation: tokenizer.convert_tokens_to_ids(token) and confirm it is not the unk id / out of range
Check ids against model.config.vocab_size (or len(tokenizer)) before building sequence_bias
Recompute the bias dict whenever you change the model or tokenizer
Drop or remap the offending ids listed in the error message

Example fix

# before
sequence_bias = {(123456,): 5.0}  # id beyond vocab -> ValueError at first step
out = model.generate(**inputs, sequence_bias=sequence_bias)

# after
vocab_size = model.config.vocab_size
target_id = tokenizer.convert_tokens_to_ids('Paris')
sequence_bias = {(target_id,): 5.0} if 0 <= target_id < vocab_size else {}
out = model.generate(**inputs, sequence_bias=sequence_bias)

Defensive patterns

Strategy: validation

Validate before calling

vocab_size = model.config.vocab_size
valid_bias = {
    ids: b for ids, b in sequence_bias.items()
    if all(0 <= tid < vocab_size for tid in ids)
}
# inspect dropped ids: set(sequence_bias) - set(valid_bias)

Type guard

def is_valid_sequence_bias(sequence_bias, vocab_size) -> bool:
    return all(
        isinstance(ids, tuple) and all(0 <= tid < vocab_size for tid in ids)
        for ids in sequence_bias
    )

Try / catch

try:
    out = model.generate(**inputs, sequence_bias=sequence_bias)
except ValueError as e:
    if 'vocabulary size' in str(e):
        # recompute ids with the current tokenizer and retry once
        raise
    raise

Prevention

Always derive biased ids via tokenizer.convert_tokens_to_ids with the model's own tokenizer
Check ids against model.config.vocab_size before generate()
Rebuild the bias dict whenever model or tokenizer version changes
The check runs lazily at the first generation step, so validate eagerly to fail fast

When it happens

Trigger: Passing sequence_bias={(123456,): 5.0} to a model whose logits width is smaller; model.generate(sequence_bias=...) built with token ids from a different tokenizer; using raw ids computed against a larger vocab model then switching models.

Common situations: Swapping tokenizer/model versions where the vocab shrank or ids shifted; hard-coded token ids copied from another project; tokenizing a phrase with one tokenizer and biasing generation of a model with another; multi-token sequences whose ids are valid but a stale id in the tuple is not.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/0e42e58036ff6332. Report an issue: GitHub.