apache/beam · error · ValueError

PartitionFn yielded a

Error message

PartitionFn yielded a '{type(partition).__name__}' when it should only yield integers

What it means

Raised inside Partition's DoFn process() when the user-supplied PartitionFn.partition_for() returns something that is not an integral number (and booleans are explicitly rejected). Beam requires each partition index to be an int so it can route the element to a tagged output. The f-string includes the actual returned type name to aid debugging.

Solutions

  1. Ensure partition_for returns a plain Python int: return int(index).
  2. Replace float arithmetic with integer arithmetic (use // with int operands, not floats).
  3. Add explicit bool handling: comparisons like x > 0 yield bool; convert to int and use distinct branch logic.
  4. Cover all input cases in partition_for so no path returns None.

Example fix

// before
class MyFn(beam.PartitionFn):
  def partition_for(self, elem, n):
    return hash(elem.key) % n / 1.0  # float
// after
class MyFn(beam.PartitionFn):
  def partition_for(self, elem, n):
    return int(hash(elem.key) % n)
Defensive patterns

Strategy: validation

Validate before calling

idx = partitionfn.partition_for(elem, n)
assert isinstance(idx, int) and not isinstance(idx, bool), f'partition_for returned {type(idx).__name__}'

Type guard

def is_valid_partition(v):
    return isinstance(v, int) and not isinstance(v, bool)

Try / catch

try:
    pc | beam.Partition(fn, n)
except ValueError as e:
    log.error('PartitionFn returned non-integer: %s', e)

Prevention

When it happens

Trigger: A custom PartitionFn whose partition_for returns a float (e.g. element % 2.0), a numpy integer-like type not registered as numbers.Integral (rare), a bool (True/False), or returns None/str on some code path.

Common situations: Using floor division that yields a float (x // 2.0), returning a numpy.int64 from numpy-based logic in environments where it is accepted (accepted) vs a numpy.float64 (rejected), accidentally returning True/False from a comparison-based partitioner, or a partition function with an unhandled branch returning None.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/392ac098a0b83f2a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/transforms/core.py:3832

  When apply()d, a Partition() PTransform requires the following:

  Args:
    partitionfn: a PartitionFn, or a callable with the signature described in
      CallableWrapperPartitionFn.
    n: number of output partitions.

  The result of this PTransform is a simple list of the output PCollections
  representing each of n partitions, in order.
  """
  class ApplyPartitionFnFn(DoFn):
    """A DoFn that applies a PartitionFn."""
    def process(self, element, partitionfn, n, *args, **kwargs):
      partition = partitionfn.partition_for(element, n, *args, **kwargs)
      import numbers
      if isinstance(partition,
                    bool) or not isinstance(partition, numbers.Integral):
        raise ValueError(
            f"PartitionFn yielded a '{type(partition).__name__}' "
            "when it should only yield integers")
      if not 0 <= int(partition) < n:
        raise ValueError(
            'PartitionFn specified out-of-bounds partition index: '
            '%d not in [0, %d)' % (partition, n))
      # Each input is directed into the output that corresponds to the
      # selected partition.
      yield pvalue.TaggedOutput(str(partition), element)

  def make_fn(self, fn, has_side_inputs):
    return fn if isinstance(fn, PartitionFn) else CallableWrapperPartitionFn(fn)

  def expand(self, pcoll):
    n = int(self.args[0])
    args, kwargs = util.insert_values_in_args(
        self.args, self.kwargs, self.side_inputs)
    return pcoll | ParDo(self.ApplyPartitionFnFn(), self.fn, *args, **

View on GitHub (pinned to 12126d8942)