apache/beam · error · TypeError

Batch {batch!r} does not have expected dtype: {self.dtype!r}

Error message

Batch {batch!r} does not have expected dtype: {self.dtype!r}

What it means

Runtime check inside NumpyTypeConstraint.type_check: the value is an ndarray but its dtype differs from the dtype encoded in the NumpyArray[...] typehint (e.g. float64 data under a float32 constraint), so the batch is rejected.

Source

Thrown at sdks/python/apache_beam/typehints/batch.py:253

  def estimate_byte_size(self, batch):
    return batch.nbytes


# numpy is starting to add typehints, which we should support
# https://numpy.org/doc/stable/reference/typing.html for now they don't allow
# specifying shape, seems to be coming after
# https://www.python.org/dev/peps/pep-0646/
class NumpyTypeHint():
  class NumpyTypeConstraint(typehints.TypeConstraint):
    def __init__(self, dtype, shape=()):
      self.dtype = np.dtype(dtype)
      self.shape = shape

    def type_check(self, batch):
      if not isinstance(batch, np.ndarray):
        raise TypeError(f"Batch {batch!r} is not an instance of ndarray")
      if not np.issubdtype(batch.dtype, self.dtype):
        raise TypeError(
            f"Batch {batch!r} does not have expected dtype: {self.dtype!r}")

      for dim in range(len(self.shape)):
        if not self.shape[dim] == N and not batch.shape[dim] == self.shape[dim]:
          raise TypeError(
              f"Batch {batch!r} does not have expected shape: {self.shape!r}")

    def _consistent_with_check_(self, sub):
      # TODO Check sub against batch type, and element type
      return True

    def __key(self):
      return (self.dtype, self.shape)

    def __eq__(self, other) -> bool:
      if isinstance(other, NumpyTypeHint.NumpyTypeConstraint):
        return self.__key() == other.__key()

View on GitHub (pinned to 12126d8942)

Solutions

Cast the batch to the declared dtype: arr.astype(expected_dtype)
Create arrays with an explicit dtype=np.int64 (or declared dtype)
Align the declared NumpyArray dtype with the dtype your data actually produces

Example fix

// before
return np.asarray(elements)
// after
return np.asarray(elements, dtype=np.int64)

Defensive patterns

Strategy: validation

Validate before calling

import numpy as np
assert np.issubdtype(arr.dtype, np.int64), f'got {arr.dtype}'

Type guard

import numpy as np
def has_dtype(arr, dtype):
    return isinstance(arr, np.ndarray) and np.issubdtype(arr.dtype, dtype)

Try / catch

try:
    check_batch(arr)
except TypeError:
    arr = arr.astype(expected_dtype)

Prevention

Create arrays with explicit dtype= parameters
Cast batches with astype before emitting
Verify arr.dtype after numpy inference from Python lists

When it happens

Trigger: Producing np.float64 arrays while the pipeline type hint is NumpyArray[np.int64, shape]; numpy inferring a wider dtype (int32->int64, float) from mixed data.

Common situations: Arrays built from Python lists getting numpy's default dtype; upcast arrays from arithmetic; platform-dependent default int dtype.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e697486899894264. Report an issue: GitHub.