huggingface/transformers · error · ValueError

The following layers have the mutually exclusive `sliding_wi

Error message

The following layers have the mutually exclusive `sliding_window` and `attention_chunk_size` both defined: {problematic_indices}. To fix this, either remove a conflicting attribute from the global config,or set it to `None` in `per_layer_config` for the problematic layers.

What it means

For heterogeneous configs, sliding_window and attention_chunk_size are mutually exclusive per layer. _validate_sliding_window_and_attention_chunk_size merges the per-layer override with the global config value (via _getattr_without_heterogeneous_validation) for every layer; any layer where both end up non-None is listed in the error, with instructions to remove one globally or explicitly null it out per layer.

Source

Thrown at src/transformers/integrations/heterogeneity/configuration_utils.py:98

    config: PreTrainedConfig, per_layer_overrides: dict[int, dict[str, Any]]
) -> None:
    problematic_indices = []
    for layer_idx in range(config.num_hidden_layers):
        layer_overrides = per_layer_overrides.get(layer_idx, {})

        sliding_window = layer_overrides.get(
            "sliding_window", config._getattr_without_heterogeneous_validation("sliding_window", None)
        )
        attention_chunk_size = layer_overrides.get(
            "attention_chunk_size",
            config._getattr_without_heterogeneous_validation("attention_chunk_size", None),
        )

        if sliding_window is not None and attention_chunk_size is not None:
            problematic_indices.append(layer_idx)

    if problematic_indices:
        raise ValueError(
            f"The following layers have the mutually exclusive `sliding_window` and `attention_chunk_size` both defined: "
            f"{problematic_indices}. To fix this, either remove a conflicting attribute from the global config,"
            f"or set it to `None` in `per_layer_config` for the problematic layers."
        )


def _get_per_layer_attributes(per_layer_overrides: dict[int, dict[str, Any]]) -> set[str]:
    per_layer_attributes: set[str] = set()
    for layer_overrides in per_layer_overrides.values():
        per_layer_attributes.update(layer_overrides)

    per_layer_attributes.discard("skip")
    return per_layer_attributes


def _modify_config_and_create_heterogeneity_spec(
    config: PreTrainedConfig, per_layer_overrides: dict[int, dict[str, Any]]
) -> _HeterogeneitySpec:

View on GitHub (pinned to a597f97485)

Solutions

1. For the listed layers, set the conflicting attribute to None in per_layer_config: {layer: {"sliding_window": None}}
2. Or remove the unused attribute from the global config so only one is defined
3. Audit every problematic index listed in the message — each needs the conflict resolved

Example fix

# before: global config has sliding_window=1024 and attention_chunk_size=128
config.per_layer_config = {0: {"attention_chunk_size": 128}}

# after: null the global attr for that layer
config.per_layer_config = {0: {"attention_chunk_size": 128, "sliding_window": None}}

def check_exclusive(config, per_layer):
    bad = []
    for i in range(config.num_hidden_layers):
        ov = per_layer.get(i, {})
        sw = ov.get("sliding_window", getattr(config, "sliding_window", None))
        acs = ov.get("attention_chunk_size", getattr(config, "attention_chunk_size", None))
        if sw is not None and acs is not None:
            bad.append(i)
    return bad

assert not check_exclusive(config, per_layer_config), f"conflicting layers: {check_exclusive(config, per_layer_config)}"

When it happens

Trigger: A global config that defines both sliding_window and attention_chunk_size (even if one is meant for a different layer subset) combined with per_layer_config that doesn't override either to None; or a per-layer override setting one while the global config supplies the other.

Common situations: Hybrid-attention models (e.g. Gemma-3/ModernBERT-style full+sliding layers or chunked-attention models) where both knobs appear in the config; inheriting a base config with sliding_window set and adding attention_chunk_size for a few layers without nulling the former.

Related errors

• The `output_attentions` attribute is not supported when usin
• Sliding window attention layers do not support block table
• `flex_attention` does not support `dropout`. Please use it w
• `skip` must be an iterable of strings.
• `skip` must contain only strings.

AI-assisted analysis of huggingface/transformers@a597f97485 (2026-08-14). Data as JSON: /api/errors/ea0d9f668c92c16b. Report an issue: GitHub.