huggingface/transformers · error · ValueError

`per_layer_config` keys must be integer layer indices in the

Error message

`per_layer_config` keys must be integer layer indices in the range [0, {num_hidden_layers}); got {invalid_layer_indices}.

What it means

_validate_layer_indices checks every key of per_layer_config (the heterogeneity layer-override map) against config.num_hidden_layers: each key must be an integer in [0, num_hidden_layers). Out-of-range indices (negative or >= layer count) are collected and reported in the error, since there is no layer to apply them to.

Source

Thrown at src/transformers/integrations/heterogeneity/configuration_utils.py:73

        if not all(isinstance(item, str) for item in skip):
            raise TypeError("`skip` must contain only strings.")

        if skip:
            normalized["skip"] = sorted(skip)

    return normalized


def _validate_layer_indices(config: PreTrainedConfig, per_layer_overrides: dict[int, dict[str, Any]]) -> None:
    if not per_layer_overrides:
        return

    num_hidden_layers = config.num_hidden_layers
    invalid_layer_indices = [
        layer_idx for layer_idx in per_layer_overrides if layer_idx < 0 or layer_idx >= num_hidden_layers
    ]
    if invalid_layer_indices:
        raise ValueError(
            f"`per_layer_config` keys must be integer layer indices in the range [0, {num_hidden_layers}); "
            f"got {invalid_layer_indices}."
        )


def _validate_sliding_window_and_attention_chunk_size(
    config: PreTrainedConfig, per_layer_overrides: dict[int, dict[str, Any]]
) -> None:
    problematic_indices = []
    for layer_idx in range(config.num_hidden_layers):
        layer_overrides = per_layer_overrides.get(layer_idx, {})

        sliding_window = layer_overrides.get(
            "sliding_window", config._getattr_without_heterogeneous_validation("sliding_window", None)
        )
        attention_chunk_size = layer_overrides.get(
            "attention_chunk_size",
            config._getattr_without_heterogeneous_validation("attention_chunk_size", None),

View on GitHub (pinned to a597f97485)

Solutions

1. Check model.config.num_hidden_layers and clamp/clip your per_layer_config keys into [0, num_hidden_layers)
2. Drop overrides for layers that no longer exist when switching model sizes
3. If keys came out as strings from JSON, convert them with int(k)

Example fix

# before: model has 28 layers
config.per_layer_config = {0: {...}, 31: {...}}  # ValueError

# after
n = config.num_hidden_layers
config.per_layer_config = {k: v for k, v in overrides.items() if 0 <= int(k) < n}

n = config.num_hidden_layers
per_layer_config = {int(k): v for k, v in per_layer_config.items()}
assert all(0 <= k < n for k in per_layer_config), (
    f"keys must be in [0, {n}); got {sorted(k for k in per_layer_config if not 0 <= k < n)}"
)
config.per_layer_config = per_layer_config

When it happens

Trigger: Setting per_layer_config = {31: {...}} on a model with num_hidden_layers=28, or using -1 as a Python-style 'last layer' index; also mixing in non-integer keys from a JSON round-trip (e.g. strings that compare oddly).

Common situations: Porting a per-layer config between model sizes (a 32-layer recipe applied to a 28-layer model); off-by-one layer counts after changing num_hidden_layers; assuming negative indexing works like Python lists.

Related errors

• `skip` must be an iterable of strings.
• `skip` must contain only strings.
• The following attributes are missing: {sorted(missing_requir
• The following layers have the mutually exclusive `sliding_wi
• Layer type '{layer_idx}' requested, but config.layer_types i

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/440e737b27bcebe7. Report an issue: GitHub.