{"record":{"id":"aca79ead265fa492","repo":"google-research/timesfm","slug":"the-embedding-dims-of-the-rotary-position-embeddin-aca79e","errorCode":null,"errorMessage":"The embedding dims of the rotary position embeddingmust match the hidden dimension of the inputs.","messagePattern":"The embedding dims of the rotary position embeddingmust match the hidden dimension of the inputs\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/timesfm/torch/transformer.py","lineNumber":77,"sourceCode":"  def __init__(\n    self,\n    embedding_dims: int,\n    min_timescale: float = 1.0,\n    max_timescale: float = 10000.0,\n  ):\n    super().__init__()\n    self.embedding_dims = embedding_dims\n    self.min_timescale = min_timescale\n    self.max_timescale = max_timescale\n\n  def forward(\n    self,\n    inputs: torch.Tensor,\n    position: torch.Tensor | None = None,\n  ):\n    \"\"\"Generates a JTensor of sinusoids with different frequencies.\"\"\"\n    if self.embedding_dims != inputs.shape[-1]:\n      raise ValueError(\n        \"The embedding dims of the rotary position embedding\"\n        \"must match the hidden dimension of the inputs.\"\n      )\n    half_embedding_dim = self.embedding_dims // 2\n    fraction = (\n      2\n      * torch.arange(0, half_embedding_dim, device=inputs.device)\n      / self.embedding_dims\n    )\n    timescale = (\n      self.min_timescale * (self.max_timescale / self.min_timescale) ** fraction\n    ).to(inputs.device)\n    if position is None:\n      seq_length = inputs.shape[1]\n      position = torch.arange(seq_length, dtype=torch.float32, device=inputs.device)[\n        None, :\n      ]\n","sourceCodeStart":59,"sourceCodeEnd":95,"githubUrl":"https://github.com/google-research/timesfm/blob/331c6d33cb1ac2611de3056d0ac7164aab6301eb/src/timesfm/torch/transformer.py#L59-L95","documentation":"The rotary position embedding applies element-wise sinusoids that must align with the last dimension of the input tensor. If self.embedding_dims != inputs.shape[-1] the multiply would broadcast incorrectly, so forward() raises ValueError. This is an internal-shape invariant of the transformer.","triggerScenarios":"Calling the rotary embedding forward with an input whose feature dimension differs from embedding_dims — typically when TransformerConfig.model_dims was changed inconsistently, or a reshaped/projected tensor with the wrong last-axis size is passed (e.g. per-head tensors whose last dim is head_dim).","commonSituations":"Modifying model_dims without rebuilding all layers consistently; feeding concatenated/split-head tensors to RoPE directly; custom attention code calling the RoPE module with transposed or reshaped inputs.","solutions":["Ensure the input's last dimension equals the rotary embedding_dims (match TransformerConfig.model_dims).","If model_dims changed, rebuild the whole model/config so every layer uses the same dimension.","Apply RoPE before splitting into heads, or reshape back to (..., seq, model_dims) first; only use per-head RoPE if per-head dim equals embedding_dims.","Check any custom projection before RoPE: its output features must equal embedding_dims."],"exampleFix":"// before\nx = x.reshape(B, T, num_heads, head_dim)  # last dim = head_dim != model_dims\nx = rotary(x)  # ValueError\n// after\nx = x.reshape(B, T, model_dims)\nx = rotary(x)  # OK","handlingStrategy":"type-guard","validationCode":"def can_apply_rotary(rotary, x):\n    return x.shape[-1] == rotary.embedding_dims\n\nif can_apply_rotary(rotary, x):\n    x = rotary(x)","typeGuard":"def rotary_compatible(rotary, x) -> bool:\n    return x.ndim in (3, 4) and x.shape[-1] == rotary.embedding_dims","tryCatchPattern":"try:\n    x = rotary(x)\nexcept ValueError:\n    x = x.reshape(*x.shape[:-2], -1)  # merge heads back to model_dims\n    x = rotary(x)","preventionTips":["Apply rotary embeddings before splitting into attention heads.","Keep model_dims consistent across all transformer layers and the rotary module.","Reshape tensors back to (..., seq, model_dims) before RoPE.","Add shape asserts in custom attention code paths."],"tags":["value-error","shape-mismatch","transformer","torch"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"331c6d33cb1ac2611de3056d0ac7164aab6301eb","analyzedAt":"2026-08-29T01:04:23.138Z","schemaVersion":2},"datasetVersion":"2026-08-29T02:17:18.158Z"}