google-research/timesfm · error · ValueError

The embedding dims of the rotary position embeddingmust matc

Error message

The embedding dims of the rotary position embeddingmust match the hidden dimension of the inputs.

What it means

Rotary position embeddings rotate pairs of input channels, so the rotary module's embedding_dims must equal the last (feature) dimension of the inputs tensor. The check runs in __call__, so it only fires at forward time when the configured rotary dims and actual hidden size diverge.

Source

Thrown at src/timesfm/flax/transformer.py:87

  def __init__(
    self,
    embedding_dims: int,
    min_timescale: int = 1,
    max_timescale: int = 10000,
  ):
    self.embedding_dims = embedding_dims
    self.min_timescale = min_timescale
    self.max_timescale = max_timescale

  def __call__(
    self,
    inputs: Float[Array, "b ... d"],
    position: Array | None = None,
  ):
    """Generates a JTensor of sinusoids with different frequencies."""
    if self.embedding_dims != inputs.shape[-1]:
      raise ValueError(
        "The embedding dims of the rotary position embedding"
        "must match the hidden dimension of the inputs."
      )
    half_embedding_dim = self.embedding_dims // 2
    fraction = 2 * jnp.arange(0, half_embedding_dim) / self.embedding_dims
    timescale = (
      self.min_timescale * (self.max_timescale / self.min_timescale) ** fraction
    )
    if position is None:
      seq_length = inputs.shape[1]
      position = jnp.arange(seq_length, dtype=jnp.float32)[None, :]
    if len(inputs.shape) == 4:
      position = position[..., None, None]
      timescale = timescale[None, None, None, :]
    elif len(inputs.shape) == 3:
      position = position[..., None]
      timescale = timescale[None, None, :]
    else:

View on GitHub (pinned to 331c6d33cb)

Solutions

Align the rotary embedding's embedding_dims with the input feature dimension (inputs.shape[-1]).
If model_dims changed, re-instantiate the transformer/rotary module with the new dims rather than reusing the old one.
Inspect inputs.shape[-1] at the call site and compare to the config's model_dims to find where the divergence originates.

Example fix

// before
rotary = RotaryEmbedding(embedding_dims=64)
y = rotary(x)  # x.shape[-1] == 128 -> ValueError
// after
rotary = RotaryEmbedding(embedding_dims=x.shape[-1])  # 128
y = rotary(x)

Defensive patterns

Strategy: validation

Validate before calling

assert rotary.embedding_dims == inputs.shape[-1], (
    f"rotary dims {rotary.embedding_dims} != input feature dim {inputs.shape[-1]}")

Type guard

def rotary_dims_match(rotary, inputs) -> bool:
    d = getattr(inputs, "shape", (None,))[-1]
    return d == rotary.embedding_dims

Try / catch

try:
    out = rotary(x)
except ValueError as e:
    if "rotary position embedding" in str(e):
        rotary = RotaryEmbedding(embedding_dims=x.shape[-1])
        out = rotary(x)
    else:
        raise

Prevention

Build the rotary module from the same config field (model_dims) used to size the input layers.
Re-instantiate all modules whenever model_dims changes.
Assert shape consistency in a forward-pass smoke test.

When it happens

Trigger: Calling the rotary embedding with inputs whose last axis does not match self.embedding_dims, e.g. model_dims changed in config but the rotary embedding was built with the old dims, or feeding a tensor with the wrong feature count.

Common situations: Mismatches after changing model_dims in a transformer config without rebuilding layers, loading a checkpoint with different head/hidden dims, or passing an intermediate tensor of the wrong width into the rotary module directly.

Related errors

AI-assisted analysis of google-research/timesfm@331c6d33cb (2026-08-29). Data as JSON: /api/errors/0a8a2008a0d59fc5. Report an issue: GitHub.