google-research/timesfm · error · ValueError

Memory dimension ({self.in_features}) must be divisible by '

Error message

Memory dimension ({self.in_features}) must be divisible by 'num_heads' heads ({self.num_heads}).

What it means

Multi-head attention splits the memory/key-value dimension evenly across heads: each head gets in_features // num_heads features. If in_features % num_heads != 0 the split is impossible, so __init__ raises ValueError. In the reference config, model_dims=1280 with num_heads=16.

Source

Thrown at src/timesfm/torch/transformer.py:194

    *,
    use_per_dim_scale: bool = True,
    use_rotary_position_embeddings: bool = True,
    use_bias: bool = False,
    attention_fn: Callable[..., torch.Tensor] = _torch_dot_product_attention,
    qk_norm: str = "rms",
    fuse_qkv: bool = False,
  ):
    super().__init__()
    self.num_heads = num_heads
    self.in_features = in_features
    self.head_dim = in_features // num_heads
    self.use_bias = use_bias
    self.attention_fn = attention_fn
    self.qk_norm = qk_norm
    self.fuse_qkv = fuse_qkv

    if self.in_features % self.num_heads != 0:
      raise ValueError(
        f"Memory dimension ({self.in_features}) must be divisible by "
        f"'num_heads' heads ({self.num_heads})."
      )

    if self.fuse_qkv:
      self.qkv_proj = nn.Linear(self.in_features, 3 * self.in_features, bias=use_bias)
    else:
      self.query = nn.Linear(self.in_features, self.in_features, bias=use_bias)
      self.key = nn.Linear(self.in_features, self.in_features, bias=use_bias)
      self.value = nn.Linear(self.in_features, self.in_features, bias=use_bias)
    self.out = nn.Linear(self.in_features, self.in_features, bias=use_bias)

    if self.qk_norm == "rms":
      self.query_ln = RMSNorm(self.head_dim)
      self.key_ln = RMSNorm(self.head_dim)
    else:
      self.query_ln = nn.Identity()
      self.key_ln = nn.Identity()

View on GitHub (pinned to 331c6d33cb)

Solutions

Choose num_heads that divides in_features exactly (model_dims=1280 works with 8, 16, 20, 32 heads).
When changing model_dims, pick a highly composite number for head-count flexibility.
Validate at config time: assert model_dims % num_heads == 0 before building.
Match the reference config (model_dims=1280, num_heads=16) unless there is a specific need.

Example fix

// before
TransformerConfig(model_dims=1000, num_heads=16)  # ValueError: 1000 % 16 != 0
// after
TransformerConfig(model_dims=1024, num_heads=16)  # 1024 / 16 = 64 per head

Defensive patterns

Strategy: validation

Validate before calling

if model_dims % num_heads != 0:
    raise ValueError(f"model_dims={model_dims} not divisible by num_heads={num_heads}")

Try / catch

try:
    attn = AttentionLayer(in_features=model_dims, num_heads=num_heads, ...)
except ValueError as e:
    if "divisible" in str(e):
        model_dims = (model_dims // num_heads + 1) * num_heads
        attn = AttentionLayer(in_features=model_dims, num_heads=num_heads, ...)
    else:
        raise

Prevention

Pick num_heads as a divisor of model_dims; prefer highly composite dims (1280, 1024).
Assert model_dims % num_heads == 0 wherever TransformerConfig is built.
Start from the reference config (1280 dims, 16 heads) and change one knob at a time.
Document the divisibility constraint next to custom config factories.

When it happens

Trigger: Constructing the attention layer (via TransformerConfig) where in_features is not divisible by num_heads — e.g. model_dims=1000 with num_heads=16, or an arbitrary experimental num_heads like 100.

Common situations: Customizing model_dims or num_heads for smaller/faster models with incompatible values; reusing a num_heads from another config; typos in num_heads.

Related errors

AI-assisted analysis of google-research/timesfm@331c6d33cb (2026-08-29). Data as JSON: /api/errors/c0cf6f2ea70912e6. Report an issue: GitHub.