huggingface/transformers · error · ValueError

Invalid value for `do_sample`: expected a boolean, got {type

Error message

Invalid value for `do_sample`: expected a boolean, got {type(generation_config.do_sample).__name__}

What it means

During assisted generation where the assistant model uses a DIFFERENT tokenizer, the code selects the candidate generator by branching on `generation_config.do_sample is True` / `is False`. These identity checks only match real Python booleans, so any other type (int, string, None, numpy.bool_) falls into the else branch and raises. The error names the offending type so you can find where the non-boolean value came from.

Source

Thrown at src/transformers/generation/utils.py:1109

                    inputs_tensor=inputs_tensor,
                    logits_processor=logits_processor,
                    target_tokenizer=target_tokenizer,
                    assistant_tokenizer=assistant_tokenizer,
                    atm_translator=atm_translator,
                )
            elif generation_config.do_sample is False:
                candidate_generator = AssistedCandidateGeneratorDifferentTokenizers(
                    input_ids=input_ids,
                    assistant_model=assistant_model,
                    generation_config=generation_config,
                    model_kwargs=model_kwargs,
                    inputs_tensor=inputs_tensor,
                    logits_processor=logits_processor,
                    target_tokenizer=target_tokenizer,
                    assistant_tokenizer=assistant_tokenizer,
                )
            else:
                raise ValueError(
                    f"Invalid value for `do_sample`: expected a boolean, got {type(generation_config.do_sample).__name__}"
                )
        else:
            candidate_generator = AssistedCandidateGenerator(
                input_ids=input_ids,
                assistant_model=assistant_model,
                generation_config=generation_config,
                model_kwargs=model_kwargs,
                inputs_tensor=inputs_tensor,
                logits_processor=logits_processor,
            )
        return candidate_generator

    def _get_logits_processor(
        self: "GenerativePreTrainedModel",
        generation_config: GenerationConfig,
        input_ids_seq_length: int | None = None,
        encoder_input_ids: torch.LongTensor | None = None,

View on GitHub (pinned to a597f97485)

Solutions

1. Set `do_sample` to a real Python boolean: `generation_config.do_sample = bool(generation_config.do_sample)` (or `True`/`False`) before calling generate.
2. Check where the value was set: inspect `model.generation_config.do_sample`, any `GenerationConfig` passed to `generate`, and any saved `generation_config.json` for non-boolean values.
3. If loading configs from JSON/YAML, normalize types after load: `cfg.do_sample = bool(cfg.do_sample)`.
4. If you intended sampling behavior, verify `do_sample=True` plus temperature/top_p are set after fixing the type.

Example fix

// before
generation_config.do_sample = 1  # int, fails identity check `is True`/`is False`
out = model.generate(**inputs, assistant_model=assistant, assistant_tokenizer=assistant_tok, generation_config=generation_config)

// after
generation_config.do_sample = True  # real Python bool
out = model.generate(**inputs, assistant_model=assistant, assistant_tokenizer=assistant_tok, generation_config=generation_config)

gc = model.generation_config
if gc.do_sample is not None and not isinstance(gc.do_sample, bool):
    gc.do_sample = bool(gc.do_sample)

def is_bool_do_sample(gc) -> bool:
    return gc.do_sample is None or isinstance(gc.do_sample, bool)

try:
    out = model.generate(**inputs, assistant_model=assistant, assistant_tokenizer=a_tok)
except ValueError as e:
    if "Invalid value for `do_sample`" in str(e):
        model.generation_config.do_sample = bool(model.generation_config.do_sample)
        out = model.generate(**inputs, assistant_model=assistant, assistant_tokenizer=a_tok)
    else:
        raise

When it happens

Trigger: Calling `model.generate(..., assistant_model=assistant, assistant_tokenizer=...)` (or any setup where main/assistant vocab sizes differ) while `do_sample` is not a strict Python `bool`: `do_sample=1`, `do_sample=0`, `do_sample="true"`, `do_sample=None`, or a `numpy.bool_` loaded from a config file or constructed programmatically.

Common situations: GenerationConfig saved to/from JSON with integer 1/0 instead of true/false; configs built by tooling or YAML loaders that coerce booleans; passing numpy booleans from data pipelines; hand-written `GenerationConfig(do_sample=1)`.

Related errors

• {} is an abstract class. Only classes inheriting this class
• {} is an abstract class. Only classes inheriting this class
• num_return_sequences has to be 1 when doing assisted generat
• assisted generation is not supported with stateful models, s
• The main model and the assistant don't have compatible encod

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/7749d979502fc899. Report an issue: GitHub.