apache/beam · error · OverflowError

{element}

Error message

{element}

What it means

Raised by CySumInt64.add_input in cy_combiners.py when an element added to the int64 sum accumulator, after conversion with int(), falls outside the signed 64-bit range [INT64_MIN, INT64_MAX]. An OverflowError carrying the offending element is thrown to protect the CyC-backed accumulator from overflowing.

Source

Thrown at sdks/python/apache_beam/transforms/cy_combiners.py:95

    self.value += n

  def merge(self, accumulators):
    for accumulator in accumulators:
      self.value += accumulator.value

  def extract_output(self):
    return self.value


class SumInt64Accumulator(object):
  def __init__(self):
    self.value = 0

  def add_input(self, element):
    global INT64_MAX, INT64_MIN  # pylint: disable=global-variable-not-assigned
    element = int(element)
    if not INT64_MIN <= element <= INT64_MAX:
      raise OverflowError(element)
    self.value += element

  def add_input_n(self, element, n):
    global INT64_MAX, INT64_MIN  # pylint: disable=global-variable-not-assigned
    element = int(element)
    if not INT64_MIN <= element <= INT64_MAX:
      raise OverflowError(element)
    self.value += element * n

  def merge(self, accumulators):
    for accumulator in accumulators:
      self.value += accumulator.value

  def extract_output(self):
    if not INT64_MIN <= self.value <= INT64_MAX:
      self.value %= 2**64
      if self.value >= INT64_MAX:
        self.value -= 2**64

View on GitHub (pinned to 12126d8942)

Solutions

1. Clamp or validate input values to fit in int64 before combining.
2. Use the plain Python (arbitrary-precision) sum combiner instead of the Cython int64 variant.
3. Aggregate in stages (e.g. per-key partial sums) or switch the sink type to a wider representation if supported.
4. Check for data errors producing absurd magnitudes (e.g. unit or scaling bugs).

Example fix

// before
elements | beam.CombineGlobally(beam.combiners.SumCombineFn())  # element 2**63
// after
validated = elements | beam.Filter(lambda x: -2**63 < x < 2**63-1)

INT64_MIN, INT64_MAX = -2**63, 2**63 - 1
def fits_int64(x):
    return INT64_MIN <= int(x) <= INT64_MAX
pcoll = pcoll | beam.Filter(fits_int64)

def is_int64(x):
    try:
        return -2**63 <= int(x) <= 2**63 - 1
    except (TypeError, ValueError):
        return False

try:
    total = pcoll | beam.CombineGlobally(beam.combiners.SumCombineFn())
except OverflowError as e:
    logging.error('int64 sum overflow for element %s', e)
    raise

When it happens

Trigger: Calling add_input (directly or via beam.Combine with sum over int64) with a value like 2**63 or -2**63-1, or a float/object that int()-converts to such a magnitude.

Common situations: Aggregating huge counters or pre-summed values; accidentally passing floats like 1e30; summing in a pipeline whose sink declares INT64 while data was assumed unbounded Python ints.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Can not encode {} as a 64-bit integer
• Could not find coder for URN " + urn
• Unknown type of encoding context
• Unknown type of decoding context
• Timing number 0b" + timingNumber.toString(2) + " has more th

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e55ecb45b3770901. Report an issue: GitHub.