apache/beam · error · OverflowError

Cannot estimate size for integer value '{value}'. Value is o

Error message

Cannot estimate size for integer value '{value}'. Value is out of the range for VarIntCoder (64-bit signed integer). Original error: {e}

What it means

VarIntCoderImpl.estimate_size computes the encoded size via get_varint_size, which requires the value to fit a 64-bit signed integer. On overflow it re-raises OverflowError explaining that the size cannot be estimated. This mirrors the encode-side limit: a value outside int64 cannot be VarInt-encoded at all.

Source

Thrown at sdks/python/apache_beam/coders/coder_impl.py:1078

    ivalue = value  # type cast
    if 0 <= ivalue < len(small_ints):
      return small_ints[ivalue]
    return StreamCoderImpl.encode(self, value)

  def decode(self, encoded):
    if len(encoded) == 1:
      i = ord(encoded)
      if 0 <= i < 128:
        return i
    return StreamCoderImpl.decode(self, encoded)

  def estimate_size(self, value, nested=False):
    # type: (Any, bool) -> int
    # Note that VarInts are encoded the same way regardless of nesting.
    try:
      return get_varint_size(value)
    except OverflowError as e:
      raise OverflowError(
          f"Cannot estimate size for integer value '{value}'. "
          f"Value is out of the range for VarIntCoder (64-bit signed integer). "
          f"Original error: {e}") from e


class VarInt32CoderImpl(StreamCoderImpl):
  """For internal use only; no backwards-compatibility guarantees.

  A coder for int32 objects."""
  def encode_to_stream(self, value, out, nested):
    # type: (int, create_OutputStream, bool) -> None
    out.write_var_int32(value)

  def decode_from_stream(self, in_stream, nested):
    # type: (create_InputStream, bool) -> int
    return in_stream.read_var_int32()

  def encode(self, value):

View on GitHub (pinned to 12126d8942)

Solutions

1. Validate/clamp the int into the 64-bit signed range before it reaches the coder
2. Rescale units (ns to ms/us) so the value fits int64
3. Switch huge numbers to a string/bytes-based coder where size estimation is well-defined
4. Fix the upstream computation that produces out-of-range ints

Example fix

// before
size = VarIntCoder().estimate_size(bignum)  # OverflowError
// after
INT64_MAX = 2**63 - 1
if bignum > INT64_MAX or bignum < -(2**63):
    bignum = max(-(2**63), min(INT64_MAX, bignum))  # clamp
size = VarIntCoder().estimate_size(bignum)

INT64_MIN, INT64_MAX = -(2**63), 2**63 - 1
assert INT64_MIN <= value <= INT64_MAX, f'cannot estimate VarInt size for {value}'

def fits_int64(v: int) -> bool:
    return -(2**63) <= v <= 2**63 - 1

try:
    size = coder.estimate_size(value)
except OverflowError as e:
    log.warning('Size estimation failed: %s', e)
    size = coder.estimate_size(clamp_to_int64(value))

When it happens

Trigger: Calling VarIntCoder().estimate_size(value) (directly or via pipeline size estimation of an int PCollection) with value outside -(2**63) to 2**63-1.

Common situations: Same root cause as encode overflow: nanosecond timestamps, giant counters, or arbitrary-precision Python ints reaching size estimation before encode fails; often seen in runner-side size estimation of elements.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Integer value '{value}' is out of the encodable range for Va
• Encountered unexpected value for null indicator: '%s'
• Number of components does not match number of coders.
• Cannot read state-written iterable without state reader.
• Invalid PaneInfoEncoding: %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a119190161f633a6. Report an issue: GitHub.