google-research/timesfm · error · ValueError

Layer norm: {config.attention_norm} not supported.

Error message

Layer norm: {config.attention_norm} not supported.

What it means

The Torch transformer block only supports RMS normalization for attention layers. At construction time, if config.attention_norm is anything other than "rms", the block raises this ValueError because no pre/post-attention LayerNorm module can be created. The f-string interpolation in the source actually shows the raw {config.attention_norm} placeholder text when raised.

Source

Thrown at src/timesfm/torch/transformer.py:318

    )

    x = x.reshape(b, n_patches, self.in_features)
    out = self.out(x)
    return out, decode_cache


class Transformer(nn.Module):
  """Classic Transformer used in TimesFM."""

  def __init__(self, config: configs.TransformerConfig):
    super().__init__()
    self.config = config

    if config.attention_norm == "rms":
      self.pre_attn_ln = RMSNorm(num_features=config.model_dims)
      self.post_attn_ln = RMSNorm(num_features=config.model_dims)
    else:
      raise ValueError(f"Layer norm: {config.attention_norm} not supported.")

    self.attn = MultiHeadAttention(
      num_heads=config.num_heads,
      in_features=config.model_dims,
      use_per_dim_scale=True,
      use_rotary_position_embeddings=config.use_rotary_position_embeddings,
      qk_norm=config.qk_norm,
      fuse_qkv=config.fuse_qkv,
    )

    if config.feedforward_norm == "rms":
      self.pre_ff_ln = RMSNorm(num_features=config.model_dims)
      self.post_ff_ln = RMSNorm(num_features=config.model_dims)
    else:
      raise ValueError(f"Layer norm: {config.feedforward_norm} not supported.")

    self.ff0 = nn.Linear(
      in_features=config.model_dims,

View on GitHub (pinned to 331c6d33cb)

Solutions

1. Set config.attention_norm = "rms" in the TimesFM config before constructing the model.
2. If the config was loaded from a checkpoint/JSON, inspect and fix the attention_norm field to "rms".
3. If you need LayerNorm, extend the __init__ in src/timesfm/torch/transformer.py to add an nn.LayerNorm branch.

Example fix

// before
config = TimesFmConfig(model_dims=1280, attention_norm="layer")
model = TimesFmTorch(config)  # raises ValueError
// after
config = TimesFmConfig(model_dims=1280, attention_norm="rms")
model = TimesFmTorch(config)  # ok

if config.attention_norm != "rms":
    raise ValueError(f"attention_norm must be 'rms', got {config.attention_norm!r}")

def has_valid_attention_norm(config) -> bool:
    return getattr(config, "attention_norm", None) == "rms"

try:
    model = TimesFmTorch(config)
except ValueError as e:
    if "attention_norm" in str(e):
        config.attention_norm = "rms"
        model = TimesFmTorch(config)
    else:
        raise

When it happens

Trigger: Constructing a Torch transformer block (e.g. TimesFmTorch or its stack of blocks) with config.attention_norm set to anything other than "rms" (e.g. "layer", "layernorm", or a None value).

Common situations: Porting configs from other models (which use standard LayerNorm) into TimesFM, hand-editing config dicts/JSON, or copying a config from the JAX/Flax implementation that allows other norm types.

Related errors

• Layer norm: {config.feedforward_norm} not supported.
• Activation: {config.ff_activation} not supported.
• Unsupported array shape: {x.shape}
• train_dynamic_numerical_covariates and test_dynamic_numerica
• train_dynamic_categorical_covariates and test_dynamic_catego

AI-assisted analysis of google-research/timesfm@331c6d33cb (2026-08-29). Data as JSON: /api/errors/35d11ecc0b14519a. Report an issue: GitHub.