huggingface/transformers · error · ValueError

Invalid max_matching_ngram_size or num_output_tokens

Error message

Invalid max_matching_ngram_size or num_output_tokens

What it means

PromptLookupCandidateGenerator (ngram prompt lookup, used by num_assistant_tokens_schedule / prompt_lookup generation) requires both max_matching_ngram_size and num_output_tokens (candidate length) to be strictly positive. Zero or negative values make the generator unable to slice n-grams or emit candidates, so the constructor rejects them.

Source

Thrown at src/transformers/generation/candidate_generator.py:1055

    def __init__(
        self,
        eos_token_id: torch.Tensor | None = None,
        num_output_tokens: int = 10,
        max_matching_ngram_size: int = 2,
        max_length: int = 20,
        logits_processor: Optional["LogitsProcessorList"] = None,
        vocab_size: int | None = None,
    ):
        self.num_output_tokens = num_output_tokens
        self.max_matching_ngram_size = max_matching_ngram_size
        self.max_length = max_length
        self.eos_token_id = eos_token_id
        self.logits_processor = logits_processor
        self.vocab_size = vocab_size

        if self.max_matching_ngram_size <= 0 or self.num_output_tokens <= 0:
            raise ValueError("Invalid max_matching_ngram_size or num_output_tokens")

    def get_candidates(self, input_ids: torch.LongTensor, **kwargs) -> tuple[torch.LongTensor, torch.FloatTensor]:
        """
        Fetches the candidates to be tried for the current input.

        Args:
            input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
                Indices of input sequence tokens in the vocabulary. [What are input IDs?](../glossary#input-ids)

        Return:
            `torch.LongTensor` of shape `(num_candidates, candidate_length)`: The candidate sequences to be tried.
        """
        bsz, input_length = input_ids.shape

        # Don't generate more than `max_length - 1` candidates since the target model generates one extra token.
        if self.max_length == input_length + 1:
            return input_ids, None

View on GitHub (pinned to a597f97485)

Solutions

1. Set both values to >= 1 (typical: max_matching_ngram_size=2-4, num_output_tokens/prompt_lookup_num_tokens=10)
2. To disable prompt lookup, omit the argument / set prompt_lookup_num_tokens=None rather than 0
3. Clamp programmatically computed values: max(1, n)

Example fix

# before
out = model.generate(inputs, prompt_lookup_num_tokens=0, do_sample=False)
# after
out = model.generate(inputs, prompt_lookup_num_tokens=10, do_sample=False)

def valid_lookup_params(max_matching_ngram_size: int, num_output_tokens: int) -> bool:
    return max_matching_ngram_size > 0 and num_output_tokens > 0

def is_positive_int(v) -> bool:
    return isinstance(v, int) and not isinstance(v, bool) and v > 0

When it happens

Trigger: model.generate(..., prompt_lookup_num_tokens=0) or negative; or constructing PromptLookupCandidateGenerator(..., max_matching_ngram_size=0, num_output_tokens=0). Also num_output_tokens computed as 0 from generation_config values.

Common situations: Passing 0 intending to 'disable' the feature (use None or omit instead), off-by-one configs copied from examples, or programmatically derived candidate counts that can reach 0.

Related errors

• `early_stopping` must be a boolean or 'never', but is {}.
• `max_new_tokens` must be greater than 0, but is {}.
• Invalid `cache_implementation` ({}). Choose one of: {}
• {auto_class} is not a valid auto class.
• {} is an abstract class. Only classes inheriting this class

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/a52a32679577d194. Report an issue: GitHub.