huggingface/transformers · error · ValueError

Layer type ' ' is not homogeneous across layers (layer…

Error message

Layer type '{layer_idx}' is not homogeneous across layers (layer {idx} differs). Use an integer index to access a specific layer's config.

What it means

When `per_layer_config` is indexed by a layer-type string on a heterogeneous config, transformers verifies that every layer of that type carries identical per-layer overrides, so a single shared config can be returned. If any layer of the requested type has different overrides (e.g. a different head dim or rope setting injected per layer), the value is ambiguous and this ValueError is raised, directing you to integer indexing.

Solutions

Index the specific layer you care about with an integer: `config.per_layer_config[3]`.
Iterate all layers and filter by type yourself: `[c for i, c in enumerate(config.per_layer_config) if config.layer_types[i] == "sliding_attention"]`.
If the layers of that type are supposed to be identical, fix the construction of `per_layer_overrides` so every layer of the type gets the same override dict.

Example fix

# before
shared = config.per_layer_config["sliding_attention"]  # layers of this type differ

# after
layer_cfgs = [
    config.per_layer_config[i]
    for i, t in enumerate(config.layer_types)
    if t == "sliding_attention"
]

Defensive patterns

Strategy: validation

Validate before calling

def layer_type_is_homogeneous(config, layer_type: str) -> bool:
    types = config.layer_types
    ov = config._heterogeneity_spec.per_layer_overrides
    ref = ov.get(types.index(layer_type), {})
    return all(ov.get(i, {}) == ref for i, t in enumerate(types) if t == layer_type)

if layer_type_is_homogeneous(config, "sliding_attention"):
    shared = config.per_layer_config["sliding_attention"]

Type guard

def can_use_string_index(config, layer_type: str) -> bool:
    types = getattr(config, "layer_types", None)
    if not types or layer_type not in types:
        return False
    return not config.is_heterogeneous or layer_type_is_homogeneous(config, layer_type)

Try / catch

try:
    shared = config.per_layer_config[layer_type]
except ValueError:
    shared = config.per_layer_config[config.layer_types.index(layer_type)]  # first layer of that type

Prevention

Treat string indexing as an optimization, not a guarantee; keep an integer-index fallback.
When building per-layer overrides programmatically, assert all layers of a type share one override dict.
Write helpers that iterate layers by type instead of assuming uniformity.

When it happens

Trigger: Calling `config.per_layer_config["sliding_attention"]` on a config where `is_heterogeneous` is True and `config._heterogeneity_spec.per_layer_overrides` maps two layers both typed "sliding_attention" to different override dicts (e.g. layer 3 has {"num_key_value_heads": 4} but layer 9 has {}).

Common situations: Custom heterogeneous checkpoints built with per-layer overrides that only partially cover the layers of one type; programmatically generating `per_layer_overrides` with off-by-one or conditional logic so layers of the same type diverge; assuming a layer type implies uniform layer config.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/648cfa5761d2a4ba. Report an issue: GitHub.

Appendix: source

Thrown at src/transformers/integrations/heterogeneity/configuration_utils.py:239

            if (layer_types := getattr(self._config, "layer_types", None)) is None:
                raise ValueError(f"Layer type '{layer_idx}' requested, but config.layer_types is not defined. ")

            if layer_idx not in layer_types:
                raise ValueError(
                    f"Layer type '{layer_idx}' not found in config.layer_types: {layer_types}. "
                    f"Available layer types: {set(layer_types)}"
                )

            # Config is actually homogeneous so just return the global config
            if not self._config.is_heterogeneous:
                return self._config

            # Ensure that all layers of the requested type have the same overrides
            layer_overrides = self._config._heterogeneity_spec.per_layer_overrides
            reference_overrides = layer_overrides.get(layer_types.index(layer_idx), {})
            for idx, layer_type in enumerate(layer_types):
                if layer_type == layer_idx and layer_overrides.get(idx, {}) != reference_overrides:
                    raise ValueError(
                        f"Layer type '{layer_idx}' is not homogeneous across layers (layer {idx} differs). "
                        f"Use an integer index to access a specific layer's config."
                    )

            return _get_layer_config(self._config, reference_overrides)

        # Return a list of configs for a slice of layers
        if isinstance(layer_idx, slice):
            return [self[i] for i in range(*layer_idx.indices(len(self)))]

        if layer_idx < 0:
            layer_idx += len(self)
        if layer_idx < 0 or layer_idx >= len(self):
            raise IndexError("list index out of range")

        # Config is actually homogeneous so just return the global config
        if not self._config.is_heterogeneous:
            return self._config

View on GitHub (pinned to a597f97485)