huggingface/transformers · error · ValueError

Setting `assistant_ensemble_weight` requires candidate logit

Error message

Setting `assistant_ensemble_weight` requires candidate logits from the assistant model. It is not supported with prompt lookup decoding.

What it means

ValueError while selecting the candidate generator for assisted generation: assistant_ensemble_weight requires logits from a real assistant model to ensemble candidate scores, but prompt_lookup_num_tokens switches to PromptLookupCandidateGenerator (n-gram copying from the prompt), which produces no assistant logits. The two options are mutually exclusive.

Source

Thrown at src/transformers/generation/utils.py:1023

        assistant_tokenizer: Optional["PreTrainedTokenizerBase"] = None,
    ) -> CandidateGenerator:
        """
        Returns the candidate generator to be used in `assisted_generation`
        """
        different_tokenizers = all(v is not None for v in (assistant_model, target_tokenizer, assistant_tokenizer))

        if generation_config.assistant_early_exit is not None:
            candidate_generator = EarlyExitCandidateGenerator(
                input_ids=input_ids,
                assistant_model=self,
                generation_config=generation_config,
                model_kwargs=model_kwargs,
                inputs_tensor=inputs_tensor,
                logits_processor=logits_processor,
            )
        elif generation_config.prompt_lookup_num_tokens is not None:
            if generation_config.assistant_ensemble_weight is not None:
                raise ValueError(
                    "Setting `assistant_ensemble_weight` requires candidate logits from the assistant model. "
                    "It is not supported with prompt lookup decoding."
                )
            candidate_generator = PromptLookupCandidateGenerator(
                eos_token_id=generation_config._eos_token_tensor,
                num_output_tokens=generation_config.prompt_lookup_num_tokens,
                max_matching_ngram_size=generation_config.max_matching_ngram_size or 2,
                max_length=generation_config.max_length,
                logits_processor=logits_processor,
                vocab_size=self.config.get_text_config().vocab_size,
            )
        elif generation_config.use_mtp:
            candidate_generator = MTPCandidateGenerator(
                main_model=self,
                generation_config=generation_config,
                logits_processor=logits_processor,
                model_kwargs=model_kwargs,
            )

View on GitHub (pinned to a597f97485)

Solutions

1. Remove assistant_ensemble_weight (set it to None) when using prompt_lookup_num_tokens.
2. Or drop prompt_lookup_num_tokens and pass a real assistant_model if you need ensemble weighting.
3. Inspect model.generation_config for both fields before generate and sanitize.

Example fix

# before
out = model.generate(**inputs, prompt_lookup_num_tokens=10, assistant_ensemble_weight=0.5)

# after
out = model.generate(**inputs, prompt_lookup_num_tokens=10)

gc = model.generation_config
if gc.prompt_lookup_num_tokens is not None and gc.assistant_ensemble_weight is not None:
    gc.assistant_ensemble_weight = None  # PLD and ensemble weighting are mutually exclusive

When it happens

Trigger: model.generate(..., assistant_model=None, prompt_lookup_num_tokens=10, assistant_ensemble_weight=0.5); a generation_config that sets both keys; enabling PLD in a config that already carries an ensemble weight from a distilled-assistant setup.

Common situations: Copy-pasting speculative-decoding configs; frameworks toggling prompt lookup for speed while leaving assistant_ensemble_weight set; defaults in generation_config.json of an assistant-enabled checkpoint.

Related errors

• Expected class name to start with Gemma4 or Gemma3n. Got {se
• Invalid max_matching_ngram_size or num_output_tokens
• Expected assistant_model to be a Gemma4AssistantForCausalLM
• `model_outputs` cannot be None, and they need to contain `hi
• Could not find `num_mtp_layers` in the model config. This mo

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/e4968f0f01e34c2e. Report an issue: GitHub.