huggingface/transformers · error · ValueError

`num_hidden_layers` ({self.num_hidden_layers}) must be equal

Error message

`num_hidden_layers` ({self.num_hidden_layers}) must be equal to the number of `{layer_types}` ({len(layers)})

What it means

ValueError from the same layer-type validator, raised when every entry is valid but len(layer_types) (or len(mlp_layer_types)) does not equal num_hidden_layers. Each decoder layer needs exactly one entry describing its type, so a mismatch means the per-layer spec cannot be mapped onto the stack.

Source

Thrown at src/transformers/configuration_utils.py:541

    def validate_layer_type(self):
        """Check that `mlp_layer_types` and `layer_types` is correctly defined."""
        for allowed_types, layer_types in zip(
            [ALLOWED_ATTN_LAYER_TYPES, ALLOWED_MLP_LAYER_TYPES], ["layer_types", "mlp_layer_types"]
        ):
            layers = getattr(self, layer_types, None)
            if not (layers is not None and hasattr(self, "num_hidden_layers")):
                return
            if self.is_custom_code():
                # Custom code may have legacy layer types that need to be remapped
                if (remapped := remap_legacy_layer_types(layers)) != layers:
                    # Only try setattr if layers changed in case layer_types is a read-only property
                    setattr(self, layer_types, remapped)
                layers = remapped
            if not all(layer_type in allowed_types for layer_type in layers):
                raise ValueError(f"The `{layer_types}` entries must be in {allowed_types} but got {layers}")
            elif self.num_hidden_layers is not None and self.num_hidden_layers != len(layers):
                raise ValueError(
                    f"`num_hidden_layers` ({self.num_hidden_layers}) must be equal to the number of `{layer_types}` "
                    f"({len(layers)})"
                )

    @property
    def rope_scaling(self):
        return self.rope_parameters

    @rope_scaling.setter
    def rope_scaling(self, value):
        self.rope_parameters = value

    def save_pretrained(self, save_directory: str | os.PathLike, push_to_hub: bool = False, **kwargs):
        """
        Save a configuration object to the directory `save_directory`, so that it can be re-loaded using the
        [`~PreTrainedConfig.from_pretrained`] class method.

        Args:

View on GitHub (pinned to a597f97485)

Solutions

1. Regenerate layer_types to have exactly num_hidden_layers entries, e.g. (['self_attn'] * n_layers) or the model's repeating pattern tiled to depth
2. Or set num_hidden_layers = len(layer_types) if the layer list is the source of truth
3. Add an assertion len(cfg.layer_types) == cfg.num_hidden_layers right after building configs programmatically

Example fix

# before
cfg.num_hidden_layers = 32
cfg.layer_types = ['self_attn'] * 12
# after
cfg.num_hidden_layers = 32
cfg.layer_types = ['self_attn'] * 32

assert len(layer_types) == num_hidden_layers, f'need exactly {num_hidden_layers} layer_types entries, got {len(layer_types)}'

def layer_count_matches(layer_types: list[str], num_hidden_layers: int) -> bool:
    return len(layer_types) == num_hidden_layers

try:
    cfg.validate_layer_types()
except ValueError as e:
    if 'must be equal to the number of' in str(e):
        cfg.layer_types = (cfg.layer_types * cfg.num_hidden_layers)[:cfg.num_hidden_layers]
    else:
        raise

When it happens

Trigger: num_hidden_layers=32 with layer_types of length 12 (e.g. only the repeating block specified); changing num_hidden_layers for a depth variant without regenerating layer_types; configs where mlp_layer_types was forgotten entirely for some layers.

Common situations: Depth-pruning or depth-extending experiments that edit num_hidden_layers only; layer lists built from a pattern with the wrong repetition count; merging configs from different depths.

Related errors

• The `{layer_types}` entries must be in {allowed_types} but g
• The embed_dim ({self.embed_dim}) is not a multiple of the nu
• The `layer_types` entries must be in {ALLOWED_LAYER_TYPES}
• `num_hidden_layers` ({num_hidden_layers}) must be equal to t
• Layer type '{layer_idx}' not found in config.layer_types: {l

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/4e9341d72b2ba020. Report an issue: GitHub.