{"record":{"id":"bbdcf212d1c19bf7","repo":"google-research/timesfm","slug":"memory-dimension-self-qkv-features-must-be-div","errorCode":null,"errorMessage":"Memory dimension ({self.qkv_features}) must be divisible by 'num_heads' heads ({self.num_heads}).","messagePattern":"Memory dimension \\((.+?)\\) must be divisible by 'num_heads' heads \\((.+?)\\)\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/timesfm/flax/transformer.py","lineNumber":161,"sourceCode":"    use_rotary_position_embeddings: bool = True,\n    use_bias: bool = False,\n    deterministic: bool | None = None,\n    attention_fn: Callable[..., Array] = nnx.dot_product_attention,\n    qk_norm: str = \"rms\",\n    rngs=nnx.Rngs(42),\n  ):\n    self.num_heads = num_heads\n    self.in_features = in_features\n    self.qkv_features = in_features\n    self.out_features = in_features\n    self.in_kv_features = in_features\n    self.deterministic = deterministic\n    self.use_bias = use_bias\n    self.attention_fn = attention_fn\n    self.qk_norm = qk_norm\n\n    if self.qkv_features % self.num_heads != 0:\n      raise ValueError(\n        f\"Memory dimension ({self.qkv_features}) must be divisible by \"\n        f\"'num_heads' heads ({self.num_heads}).\"\n      )\n    self.head_dim = self.qkv_features // self.num_heads\n\n    linear_general = functools.partial(\n      LinearGeneral,\n      out_features=(self.num_heads, self.head_dim),\n      use_bias=self.use_bias,\n    )\n    # project inputs_q to multi-headed q/k/v\n    # dimensions are then [batch..., length, n_heads, n_features_per_head]\n    self.query = linear_general(self.in_features, rngs=rngs)\n    self.key = linear_general(self.in_kv_features, rngs=rngs)\n    self.value = linear_general(self.in_kv_features, rngs=rngs)\n\n    if self.qk_norm == \"rms\":\n      self.query_ln = RMSNorm(self.head_dim)","sourceCodeStart":143,"sourceCodeEnd":179,"githubUrl":"https://github.com/google-research/timesfm/blob/331c6d33cb1ac2611de3056d0ac7164aab6301eb/src/timesfm/flax/transformer.py#L143-L179","documentation":"Multi-head attention splits the qkv/memory feature dimension into num_heads chunks of size head_dim, which requires qkv_features % num_heads == 0. The constructor validates this at build time and raises ValueError otherwise.","triggerScenarios":"Constructing MultiHeadAttention where qkv_features (often config.model_dims) is not divisible by num_heads, e.g. model_dims=100 with num_heads=4, or model_dims=128 with num_heads=7.","commonSituations":"Changing num_heads or model_dims independently in a config; adapting configs between model sizes where the two values were coupled; picking a head count that 'looks reasonable' without checking divisibility.","solutions":["Choose num_heads that divides qkv_features evenly (e.g. model_dims=128 works with 1,2,4,8,16 heads).","Adjust qkv_features to the nearest multiple of num_heads if head count is fixed.","Add a config-level assertion/check early (at config load) to fail fast with a clear message."],"exampleFix":"// before\nattn = MultiHeadAttention(num_heads=7, qkv_features=128)  # ValueError\n// after\nattn = MultiHeadAttention(num_heads=8, qkv_features=128)  # head_dim = 16","handlingStrategy":"validation","validationCode":"if config.model_dims % config.num_heads != 0:\n    raise ValueError(f\"model_dims ({config.model_dims}) must be divisible by num_heads ({config.num_heads})\")","typeGuard":null,"tryCatchPattern":"try:\n    attn = MultiHeadAttention(num_heads=config.num_heads, qkv_features=config.model_dims)\nexcept ValueError as e:\n    if \"divisible by\" in str(e):\n        config.num_heads = max(h for h in (1,2,4,8,16) if config.model_dims % h == 0)\n        attn = MultiHeadAttention(num_heads=config.num_heads, qkv_features=config.model_dims)\n    else:\n        raise","preventionTips":["Choose num_heads from divisors of model_dims.","Validate the pairs (model_dims, num_heads) at config load.","Cross-check head counts when porting configs between model sizes."],"tags":["config","shape","attention","valueerror"],"backgroundTag":"dimension-not-divisible","analyzedSha":"331c6d33cb1ac2611de3056d0ac7164aab6301eb","analyzedAt":"2026-08-29T01:04:23.138Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}