huggingface/transformers · error · ValueError

Layer type '{layer_idx}' is not homogeneous across layers (l

Error message

Layer type '{layer_idx}' is not homogeneous across layers (layer {idx} differs). Use an integer index to access a specific layer's config.

What it means

When `per_layer_config` is indexed by a layer-type string on a heterogeneous config, transformers verifies that every layer of that type carries identical per-layer overrides, so a single shared config can be returned. If any layer of the requested type has different overrides (e.g. a different head dim or rope setting injected per layer), the value is ambiguous and this ValueError is raised, directing you to integer indexing.

Source

Thrown at src/transformers/integrations/heterogeneity/configuration_utils.py:239

            if (layer_types := getattr(self._config, "layer_types", None)) is None:
                raise ValueError(f"Layer type '{layer_idx}' requested, but config.layer_types is not defined. ")

            if layer_idx not in layer_types:
                raise ValueError(
                    f"Layer type '{layer_idx}' not found in config.layer_types: {layer_types}. "
                    f"Available layer types: {set(layer_types)}"
                )

            # Config is actually homogeneous so just return the global config
            if not self._config.is_heterogeneous:
                return self._config

            # Ensure that all layers of the requested type have the same overrides
            layer_overrides = self._config._heterogeneity_spec.per_layer_overrides
            reference_overrides = layer_overrides.get(layer_types.index(layer_idx), {})
            for idx, layer_type in enumerate(layer_types):
                if layer_type == layer_idx and layer_overrides.get(idx, {}) != reference_overrides:
                    raise ValueError(
                        f"Layer type '{layer_idx}' is not homogeneous across layers (layer {idx} differs). "
                        f"Use an integer index to access a specific layer's config."
                    )

            return _get_layer_config(self._config, reference_overrides)

        # Return a list of configs for a slice of layers
        if isinstance(layer_idx, slice):
            return [self[i] for i in range(*layer_idx.indices(len(self)))]

        if layer_idx < 0:
            layer_idx += len(self)
        if layer_idx < 0 or layer_idx >= len(self):
            raise IndexError("list index out of range")

        # Config is actually homogeneous so just return the global config
        if not self._config.is_heterogeneous:
            return self._config

View on GitHub (pinned to a597f97485)

Solutions

Index the specific layer you care about with an integer: `config.per_layer_config[3]`.
Iterate all layers and filter by type yourself: `[c for i, c in enumerate(config.per_layer_config) if config.layer_types[i] == "sliding_attention"]`.
If the layers of that type are supposed to be identical, fix the construction of `per_layer_overrides` so every layer of the type gets the same override dict.

Example fix

# before
shared = config.per_layer_config["sliding_attention"]  # layers of this type differ

# after
layer_cfgs = [
    config.per_layer_config[i]
    for i, t in enumerate(config.layer_types)
    if t == "sliding_attention"
]

Defensive patterns

Strategy: validation

Validate before calling

def layer_type_is_homogeneous(config, layer_type: str) -> bool:
    types = config.layer_types
    ov = config._heterogeneity_spec.per_layer_overrides
    ref = ov.get(types.index(layer_type), {})
    return all(ov.get(i, {}) == ref for i, t in enumerate(types) if t == layer_type)

if layer_type_is_homogeneous(config, "sliding_attention"):
    shared = config.per_layer_config["sliding_attention"]

Type guard

def can_use_string_index(config, layer_type: str) -> bool:
    types = getattr(config, "layer_types", None)
    if not types or layer_type not in types:
        return False
    return not config.is_heterogeneous or layer_type_is_homogeneous(config, layer_type)

Try / catch

try:
    shared = config.per_layer_config[layer_type]
except ValueError:
    shared = config.per_layer_config[config.layer_types.index(layer_type)]  # first layer of that type

Prevention

Treat string indexing as an optimization, not a guarantee; keep an integer-index fallback.
When building per-layer overrides programmatically, assert all layers of a type share one override dict.
Write helpers that iterate layers by type instead of assuming uniformity.

When it happens

Trigger: Calling `config.per_layer_config["sliding_attention"]` on a config where `is_heterogeneous` is True and `config._heterogeneity_spec.per_layer_overrides` maps two layers both typed "sliding_attention" to different override dicts (e.g. layer 3 has {"num_key_value_heads": 4} but layer 9 has {}).

Common situations: Custom heterogeneous checkpoints built with per-layer overrides that only partially cover the layers of one type; programmatically generating `per_layer_overrides` with off-by-one or conditional logic so layers of the same type diverge; assuming a layer type implies uniform layer config.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/648cfa5761d2a4ba. Report an issue: GitHub.