apache/beam · error · RuntimeError

Cannot safely index records from

Error message

Cannot safely index records from {len(path_indices)} files of size {readable_file.metadata.size_in_bytes} as their product is greater than 2^63.

What it means

_ReadFromPandasFileDoFn.process refuses to split records from files when the product of the file count and the per-file index stride would exceed 2^63, which would overflow the integer index space used for restriction tracking. The stride (indices_per_file) is computed so len(path_indices) * indices_per_file stays below 2^63, and any file larger than that stride cannot be safely indexed.

Solutions

1. Reduce the number of input files so the computed indices_per_file (10**int(log10(2^63 // len(files)))) grows larger than each file's byte size
2. Split large files into smaller files so each file's size_in_bytes is below indices_per_file
3. Read the data with non-dataframe Beam IO (e.g. beam.io.ReadFromText) and convert to dataframes after splitting
4. Downgrade or patch Beam if the restriction-sizing heuristic is unsuitable for your workload

Example fix

// before
beam.dataframe.io.read_csv('gs://bucket/data/part-*')  # 500 files of ~1GB each
// after
# fewer, smaller files, e.g. 100 files of ~500MB
beam.dataframe.io.read_csv('gs://bucket/data_repartitioned/part-*')

import math
n = len(files)
stride = 10 ** int(math.log(2**63 // n, 10))
oversized = [f for f in files if os.path.getsize(f) > stride]
if oversized:
    raise ValueError(f'{len(oversized)} files exceed safe index stride {stride}; split them or reduce file count')

When it happens

Trigger: Calling apache_beam.dataframe.io read_csv/read_json/read_excel (via beam.dataframe.io.read) with many input files whose size_in_bytes exceeds 10**floor(log10(2^63 // len(files))); e.g. hundreds of files each larger than the computed stride.

Common situations: Reading very large CSV/parquet-style files with beam dataframes; splitting a huge dataset across many files so the automatic file count rises and the per-file stride shrinks below actual file size.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Accessing locals with @ is not yet supported…
• align_axis must be one of ('index', 0, 'columns', 1). got
• align( )
• Assigning an index is not yet supported. Consider using…
• axis must be one of (0, 1, 'index', 'columns'), got

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8a8bd6071cc4a63b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/dataframe/io.py:645

  def restriction_size(self, readable_file, restriction):
    return restriction.size()

  def create_tracker(self, restriction):
    tracker = beam.io.restriction_trackers.OffsetRestrictionTracker(restriction)
    if self.splitter:
      return tracker
    else:
      return beam.io.restriction_trackers.UnsplittableRestrictionTracker(
          tracker)

  def process(
      self, readable_file, path_indices, tracker=beam.DoFn.RestrictionParam()):
    reader = self.reader
    if isinstance(reader, str):
      reader = getattr(pd, self.reader)
    indices_per_file = 10**int(math.log(2**63 // len(path_indices), 10))
    if readable_file.metadata.size_in_bytes > indices_per_file:
      raise RuntimeError(
          f'Cannot safely index records from {len(path_indices)} files '
          f'of size {readable_file.metadata.size_in_bytes} '
          f'as their product is greater than 2^63.')
    start_index = (
        tracker.current_restriction().start +
        path_indices[readable_file.metadata.path] * indices_per_file)
    with readable_file.open() as handle:
      if self.incremental:
        # TODO(robertwb): We could consider trying to get progress for
        # non-incremental sources that are read linearly, as long as they
        # don't try to seek.  This could be deceptive as progress would
        # advance to 100% the instant the (large) read was done, discounting
        # any downstream processing.
        handle = _TruncatingFileHandle(
            handle,
            tracker,
            splitter=self.splitter or
            _DelimSplitter(b'\n', _DEFAULT_BYTES_CHUNKSIZE))

View on GitHub (pinned to 12126d8942)