xai-org/x-algorithm · error · ValueError

attn_logit_cap_method {method!r} is not supported by JaxAtte

Error message

attn_logit_cap_method {method!r} is not supported by JaxAttention.

What it means

JaxAttention supports only two attention logit capping methods in _cap_attention_logits: 'tanh' (cap * tanh(logits / cap)) and 'soft_sign' (logits / (1 + |logits| / cap)). Any other value of attn_logit_cap_method reaches the trailing raise and is rejected with the offending value.

Source

Thrown at phoenix/xrex/models/attention.py:235

    query_input: jax.Array,
    key_input: jax.Array,
    pairwise_fn: Callable[..., Any] = jnp.multiply,
    dtype: Any = jnp.bfloat16,
):
    mask = pairwise_fn(jnp.expand_dims(query_input, axis=-1), jnp.expand_dims(key_input, axis=-2))
    mask = jnp.expand_dims(mask, axis=-3)
    return mask.astype(dtype)


def _cap_attention_logits(logits: jax.Array, cap: float, method: str) -> jax.Array:
    if not cap or cap <= 0.0 or method == "none":
        return logits
    if method == "tanh":
        cap_arr = jnp.array(cap, dtype=logits.dtype)
        return cap_arr * jnp.tanh(logits / cap_arr)
    if method == "soft_sign":
        return logits / (1.0 + jnp.abs(logits) / cap)
    raise ValueError(f"attn_logit_cap_method {method!r} is not supported by JaxAttention.")


class JaxAttention(Attention):
    def call_attn(
        self,
        query: jax.Array,
        key: Optional[jax.Array],
        value: Optional[jax.Array],
        segment_ids: Optional[jax.Array],
        segment_ids_k: Optional[jax.Array],
        temp: Optional[jax.Array],
        **kwargs,
    ):
        mask = kwargs.get("masks", None)
        b, t, h, d = query.shape
        _, _, kv_h, _ = key.shape
        assert h % kv_h == 0, f"query_heads {h} must be a multiple of kv_heads {kv_h}"
        assert self.sharding_context is not None

View on GitHub (pinned to 24c60942c5)

Solutions

1. Use 'tanh' or 'soft_sign' as attn_logit_cap_method.
2. To disable capping, clear the cap setting rather than passing a sentinel method name.
3. If you need a new method, add an explicit branch in _cap_attention_logits before the raise.

Example fix

# before
config.attn_logit_cap_method = "relu"

# after
config.attn_logit_cap_method = "tanh"

assert config.attn_logit_cap_method in {"tanh", "soft_sign"}, config.attn_logit_cap_method

def is_valid_cap_method(m: str) -> bool:
    return m in {"tanh", "soft_sign"}

When it happens

Trigger: Setting config.attn_logit_cap_method to something like 'relu', 'sigmoid', 'none', or a typo like 'tanh ' while using the jax_attn implementation; call_attn invokes _cap_attention_logits when a cap is configured.

Common situations: Copying a config from a codebase (e.g. Gemma-style) that accepts more cap methods; typos; version drift where a method was removed or renamed.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• Invalid attention implementation: {self.config.attn_impl}
• Rope type {self.config.rope_type} is not supported
• FlashAttention2 has been removed. Please use fa_version='3'
• Unknown loss_type: {loss_type}
• Invalid argument {arg!r}, not a key=value replacement and no

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/e7517b5690b50b34. Report an issue: GitHub.