apache/beam · error · ValueError

PartitionFn specified out-of-bounds partition index

Error message

PartitionFn specified out-of-bounds partition index: %d not in [0, %d)

What it means

Raised inside Partition's DoFn after the type check: the PartitionFn returned an integer, but its value is outside the valid range [0, n), where n is the number of output partitions. Beam routes each element to a tagged output named str(partition), so out-of-range indices cannot be delivered. The error reports the offending index and the valid bound.

Solutions

Always compute the index modulo n: return value % n (with a non-negative base).
Match hard-coded indices to the partition count passed to beam.Partition().
Clamp or validate: index = min(max(0, computed), n - 1) if clamping is semantically OK.
Pass the partition count into the PartitionFn configuration rather than duplicating constants.

Example fix

// before
my_partition = pc | beam.Partition(lambda x, n: x.bucket, 3)  # bucket can be 0..9
// after
my_partition = pc | beam.Partition(lambda x, n: x.bucket % n, 3)

Defensive patterns

Strategy: validation

Validate before calling

idx = partitionfn.partition_for(elem, n)
assert 0 <= int(idx) < n, f'partition {idx} out of range [0, {n})'

Try / catch

try:
    pc | beam.Partition(fn, n)
except ValueError as e:
    log.error('Out-of-bounds partition: %s', e)

Prevention

Always modulo by n: value % n
Derive partition count from one shared constant
Unit-test partition_for with edge elements
Avoid hard-coded indices

When it happens

Trigger: partition_for returns hash(x) % m where m != n; using a hard-coded partition index like return 3 while constructing beam.Partition(2); computing an index from data whose cardinality exceeds the configured partition count; negative indices from modulo of negative hashes in other languages.

Common situations: Changing the number of partitions in the pipeline (beam.Partition(fn, 3)) without updating a hard-coded PartitionFn; using element values directly as indices; stale constants after refactoring partition count.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/cb5a3b17eafdc844. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/transforms/core.py:3836

    partitionfn: a PartitionFn, or a callable with the signature described in
      CallableWrapperPartitionFn.
    n: number of output partitions.

  The result of this PTransform is a simple list of the output PCollections
  representing each of n partitions, in order.
  """
  class ApplyPartitionFnFn(DoFn):
    """A DoFn that applies a PartitionFn."""
    def process(self, element, partitionfn, n, *args, **kwargs):
      partition = partitionfn.partition_for(element, n, *args, **kwargs)
      import numbers
      if isinstance(partition,
                    bool) or not isinstance(partition, numbers.Integral):
        raise ValueError(
            f"PartitionFn yielded a '{type(partition).__name__}' "
            "when it should only yield integers")
      if not 0 <= int(partition) < n:
        raise ValueError(
            'PartitionFn specified out-of-bounds partition index: '
            '%d not in [0, %d)' % (partition, n))
      # Each input is directed into the output that corresponds to the
      # selected partition.
      yield pvalue.TaggedOutput(str(partition), element)

  def make_fn(self, fn, has_side_inputs):
    return fn if isinstance(fn, PartitionFn) else CallableWrapperPartitionFn(fn)

  def expand(self, pcoll):
    n = int(self.args[0])
    args, kwargs = util.insert_values_in_args(
        self.args, self.kwargs, self.side_inputs)
    return pcoll | ParDo(self.ApplyPartitionFnFn(), self.fn, *args, **
                         kwargs).with_outputs(*[str(t) for t in range(n)])


class Windowing(object):

View on GitHub (pinned to 12126d8942)