apache/beam · error · TypeError

Batch {batch!r} does not have expected shape: {self.shape!r}

Error message

Batch {batch!r} does not have expected shape: {self.shape!r}

What it means

For each declared dimension (except the N placeholder dimension), NumpyArray.type_check requires the batch array's actual shape to match. A batch with the wrong number of elements per batch or wrong element dimensions fails.

Source

Thrown at sdks/python/apache_beam/typehints/batch.py:258

# https://numpy.org/doc/stable/reference/typing.html for now they don't allow
# specifying shape, seems to be coming after
# https://www.python.org/dev/peps/pep-0646/
class NumpyTypeHint():
  class NumpyTypeConstraint(typehints.TypeConstraint):
    def __init__(self, dtype, shape=()):
      self.dtype = np.dtype(dtype)
      self.shape = shape

    def type_check(self, batch):
      if not isinstance(batch, np.ndarray):
        raise TypeError(f"Batch {batch!r} is not an instance of ndarray")
      if not np.issubdtype(batch.dtype, self.dtype):
        raise TypeError(
            f"Batch {batch!r} does not have expected dtype: {self.dtype!r}")

      for dim in range(len(self.shape)):
        if not self.shape[dim] == N and not batch.shape[dim] == self.shape[dim]:
          raise TypeError(
              f"Batch {batch!r} does not have expected shape: {self.shape!r}")

    def _consistent_with_check_(self, sub):
      # TODO Check sub against batch type, and element type
      return True

    def __key(self):
      return (self.dtype, self.shape)

    def __eq__(self, other) -> bool:
      if isinstance(other, NumpyTypeHint.NumpyTypeConstraint):
        return self.__key() == other.__key()

      return NotImplemented

    def __hash__(self) -> int:
      return hash(self.__key())

View on GitHub (pinned to 12126d8942)

Solutions

1. Fix element construction so each element has the declared shape
2. Adjust the NumpyArray shape declaration to match real element dims
3. Pad/truncate elements to a uniform shape before batching

Example fix

// before
np.asarray(elements)  # elements of varying length, declared shape (3,)
// after
np.asarray([e[:3] for e in elements], dtype=np.int64)  # enforce shape (3,)

import numpy as np
# declared shape (3,) => every batch row must be length 3
assert arr.ndim == len(declared_shape) and all(s == d for s, d in zip(arr.shape, declared_shape) if d != N)

def has_shape(arr, declared_shape, N):
    return all(sd == N or ad == sd for ad, sd in zip(arr.shape, declared_shape))

When it happens

Trigger: Declaring NumpyArray[np.int64, (3,)] (fixed element size 3) but producing batches whose non-N dims are (4,) or whose element rows are size 2.

Common situations: Variable-length elements batched into an array with a fixed declared shape; ragged data that can't fit the declared per-element shape.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

• Failed to align batch type's batch dimension with element ty
• Element type is not a dtype
• batch type must be np.ndarray or beam.typehints.batch.NumpyA
• batch type and element type must have equivalent dtypes (bat
• Batch {batch!r} is not an instance of ndarray

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a3a60a1af21e0184. Report an issue: GitHub.