huggingface/transformers · error · ValueError

`assistant_ensemble_weight` must be in the open interval `(0

Error message

`assistant_ensemble_weight` must be in the open interval `(0.0, 1.0)`, but is {}. Use `None` for standard (lossless) speculative decoding.

What it means

assistant_ensemble_weight blends target and assistant distributions in lossy speculative decoding (self-speculative ensemble). validate() requires it strictly inside the open interval (0.0, 1.0) or None; 0, 1, or values outside would degenerate or invert the ensemble and are rejected.

Source

Thrown at src/transformers/generation/configuration_utils.py:672

        Args:
            strict (bool): If True, raise an exception for any issues found. If False, only log issues.
            user_set_attributes (set[str], *optional*): Names of attributes the caller explicitly provided. When
                supplied, "minor issue" warnings about conflicting flag combinations (e.g. sampling-only flags set
                while `do_sample=False`) only fire if the conflicting flag is in this set -- avoiding noisy warnings
                when the value was inherited from a model's default `generation_config.json`. When `None`, all set
                attributes are considered user-set (backward-compatible behavior for direct `validate()` calls).
        """
        minor_issues = {}  # format: {attribute_name: issue_description}

        # 1. Validation of individual attributes
        # 1.1. Decoding attributes
        if self.early_stopping not in {None, True, False, "never"}:
            raise ValueError(f"`early_stopping` must be a boolean or 'never', but is {self.early_stopping}.")
        if self.max_new_tokens is not None and self.max_new_tokens <= 0:
            raise ValueError(f"`max_new_tokens` must be greater than 0, but is {self.max_new_tokens}.")
        if self.assistant_ensemble_weight is not None and not (0.0 < self.assistant_ensemble_weight < 1.0):
            raise ValueError(
                f"`assistant_ensemble_weight` must be in the open interval `(0.0, 1.0)`, "
                f"but is {self.assistant_ensemble_weight}. Use `None` for standard (lossless) speculative decoding."
            )
        if self.pad_token_id is not None and self.pad_token_id < 0:
            minor_issues["pad_token_id"] = (
                f"`pad_token_id` should be positive but got {self.pad_token_id}. This will cause errors when batch "
                "generating, if there is padding. Please set `pad_token_id` explicitly as "
                "`model.generation_config.pad_token_id=PAD_TOKEN_ID` to avoid errors in generation"
            )
        # 1.2. Cache attributes
        # "paged" re-routes to continuous batching and so it is a valid cache implementation. But we do not want to test
        # it with the `generate` as the other would be, so we we cannot add it to ALL_CACHE_IMPLEMENTATIONS
        valid_cache_implementations = ALL_CACHE_IMPLEMENTATIONS + ("paged",)
        if self.cache_implementation is not None and self.cache_implementation not in valid_cache_implementations:
            raise ValueError(
                f"Invalid `cache_implementation` ({self.cache_implementation}). Choose one of: "
                f"{valid_cache_implementations}"
            )

View on GitHub (pinned to a597f97485)

Solutions

1. Set assistant_ensemble_weight=None to use standard lossless speculative decoding
2. Otherwise pick a value strictly between 0 and 1, e.g. 0.5
3. Remove the key from generation_config.json rather than writing 0/1 to disable it

Example fix

# before
cfg = GenerationConfig(assistant_ensemble_weight=1.0)
# after
cfg = GenerationConfig(assistant_ensemble_weight=None)

def valid_assistant_ensemble_weight(v) -> bool:
    return v is None or (isinstance(v, (int, float)) and 0.0 < v < 1.0)

def is_valid_ensemble_weight(v) -> bool:
    return v is None or (isinstance(v, float) and 0.0 < v < 1.0)

When it happens

Trigger: GenerationConfig(assistant_ensemble_weight=0.0 or 1.0 or 1.5), or reading a generation_config.json where the weight was written as 0/1; validation fires when generate() runs with the config.

Common situations: Users probing boundary values (0 = 'off', 1 = 'full assistant') without realizing both are invalid; configs edited to disable the feature instead of removing the key.

Related errors

• Expected assistant_model to be a Gemma4AssistantForCausalLM
• `model_outputs` cannot be None, and they need to contain `hi
• Could not find `num_mtp_layers` in the model config. This mo
• `early_stopping` must be a boolean or 'never', but is {}.
• `max_new_tokens` must be greater than 0, but is {}.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/cb7926451615a70f. Report an issue: GitHub.