huggingface/transformers · error · ValueError

The following layers have the mutually exclusive…

Error message

The following layers have the mutually exclusive `sliding_window` and `attention_chunk_size` both defined: {problematic_indices}. To fix this, either remove a conflicting attribute from the global config,or set it to `None` in `per_layer_config` for the problematic layers.

What it means

For heterogeneous configs, sliding_window and attention_chunk_size are mutually exclusive per layer. _validate_sliding_window_and_attention_chunk_size merges the per-layer override with the global config value (via _getattr_without_heterogeneous_validation) for every layer; any layer where both end up non-None is listed in the error, with instructions to remove one globally or explicitly null it out per layer.

Solutions

For the listed layers, set the conflicting attribute to None in per_layer_config: {layer: {"sliding_window": None}}
Or remove the unused attribute from the global config so only one is defined
Audit every problematic index listed in the message — each needs the conflict resolved

Example fix

# before: global config has sliding_window=1024 and attention_chunk_size=128
config.per_layer_config = {0: {"attention_chunk_size": 128}}

# after: null the global attr for that layer
config.per_layer_config = {0: {"attention_chunk_size": 128, "sliding_window": None}}

Defensive patterns

Strategy: validation

Validate before calling

def check_exclusive(config, per_layer):
    bad = []
    for i in range(config.num_hidden_layers):
        ov = per_layer.get(i, {})
        sw = ov.get("sliding_window", getattr(config, "sliding_window", None))
        acs = ov.get("attention_chunk_size", getattr(config, "attention_chunk_size", None))
        if sw is not None and acs is not None:
            bad.append(i)
    return bad

assert not check_exclusive(config, per_layer_config), f"conflicting layers: {check_exclusive(config, per_layer_config)}"

Prevention

In hybrid models, explicitly set the non-applicable attention attribute to None per layer
Treat sliding_window and attention_chunk_size as mutually exclusive when authoring configs

When it happens

Trigger: A global config that defines both sliding_window and attention_chunk_size (even if one is meant for a different layer subset) combined with per_layer_config that doesn't override either to None; or a per-layer override setting one while the global config supplies the other.

Common situations: Hybrid-attention models (e.g. Gemma-3/ModernBERT-style full+sliding layers or chunked-attention models) where both knobs appear in the config; inheriting a base config with sliding_window set and adding attention_chunk_size for a few layers without nulling the former.

Related errors

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/ea0d9f668c92c16b. Report an issue: GitHub.

Appendix: source

Thrown at src/transformers/integrations/heterogeneity/configuration_utils.py:98

    config: PreTrainedConfig, per_layer_overrides: dict[int, dict[str, Any]]
) -> None:
    problematic_indices = []
    for layer_idx in range(config.num_hidden_layers):
        layer_overrides = per_layer_overrides.get(layer_idx, {})

        sliding_window = layer_overrides.get(
            "sliding_window", config._getattr_without_heterogeneous_validation("sliding_window", None)
        )
        attention_chunk_size = layer_overrides.get(
            "attention_chunk_size",
            config._getattr_without_heterogeneous_validation("attention_chunk_size", None),
        )

        if sliding_window is not None and attention_chunk_size is not None:
            problematic_indices.append(layer_idx)

    if problematic_indices:
        raise ValueError(
            f"The following layers have the mutually exclusive `sliding_window` and `attention_chunk_size` both defined: "
            f"{problematic_indices}. To fix this, either remove a conflicting attribute from the global config,"
            f"or set it to `None` in `per_layer_config` for the problematic layers."
        )


def _get_per_layer_attributes(per_layer_overrides: dict[int, dict[str, Any]]) -> set[str]:
    per_layer_attributes: set[str] = set()
    for layer_overrides in per_layer_overrides.values():
        per_layer_attributes.update(layer_overrides)

    per_layer_attributes.discard("skip")
    return per_layer_attributes


def _modify_config_and_create_heterogeneity_spec(
    config: PreTrainedConfig, per_layer_overrides: dict[int, dict[str, Any]]
) -> _HeterogeneitySpec:

View on GitHub (pinned to a597f97485)