{"record":{"id":"8a8bd6071cc4a63b","repo":"apache/beam","slug":"cannot-safely-index-records-from-len-path-indices-files-of","errorCode":null,"errorMessage":"Cannot safely index records from {len(path_indices)} files of size {readable_file.metadata.size_in_bytes} as their product is greater than 2^63.","messagePattern":"Cannot safely index records from (.+?) files of size (.+?) as their product is greater than 2\\^63\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/dataframe/io.py","lineNumber":645,"sourceCode":"  def restriction_size(self, readable_file, restriction):\n    return restriction.size()\n\n  def create_tracker(self, restriction):\n    tracker = beam.io.restriction_trackers.OffsetRestrictionTracker(restriction)\n    if self.splitter:\n      return tracker\n    else:\n      return beam.io.restriction_trackers.UnsplittableRestrictionTracker(\n          tracker)\n\n  def process(\n      self, readable_file, path_indices, tracker=beam.DoFn.RestrictionParam()):\n    reader = self.reader\n    if isinstance(reader, str):\n      reader = getattr(pd, self.reader)\n    indices_per_file = 10**int(math.log(2**63 // len(path_indices), 10))\n    if readable_file.metadata.size_in_bytes > indices_per_file:\n      raise RuntimeError(\n          f'Cannot safely index records from {len(path_indices)} files '\n          f'of size {readable_file.metadata.size_in_bytes} '\n          f'as their product is greater than 2^63.')\n    start_index = (\n        tracker.current_restriction().start +\n        path_indices[readable_file.metadata.path] * indices_per_file)\n    with readable_file.open() as handle:\n      if self.incremental:\n        # TODO(robertwb): We could consider trying to get progress for\n        # non-incremental sources that are read linearly, as long as they\n        # don't try to seek.  This could be deceptive as progress would\n        # advance to 100% the instant the (large) read was done, discounting\n        # any downstream processing.\n        handle = _TruncatingFileHandle(\n            handle,\n            tracker,\n            splitter=self.splitter or\n            _DelimSplitter(b'\\n', _DEFAULT_BYTES_CHUNKSIZE))","sourceCodeStart":627,"sourceCodeEnd":663,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/dataframe/io.py#L627-L663","documentation":"_ReadFromPandasFileDoFn.process refuses to split records from files when the product of the file count and the per-file index stride would exceed 2^63, which would overflow the integer index space used for restriction tracking. The stride (indices_per_file) is computed so len(path_indices) * indices_per_file stays below 2^63, and any file larger than that stride cannot be safely indexed.","triggerScenarios":"Calling apache_beam.dataframe.io read_csv/read_json/read_excel (via beam.dataframe.io.read) with many input files whose size_in_bytes exceeds 10**floor(log10(2^63 // len(files))); e.g. hundreds of files each larger than the computed stride.","commonSituations":"Reading very large CSV/parquet-style files with beam dataframes; splitting a huge dataset across many files so the automatic file count rises and the per-file stride shrinks below actual file size.","solutions":["Reduce the number of input files so the computed indices_per_file (10**int(log10(2^63 // len(files)))) grows larger than each file's byte size","Split large files into smaller files so each file's size_in_bytes is below indices_per_file","Read the data with non-dataframe Beam IO (e.g. beam.io.ReadFromText) and convert to dataframes after splitting","Downgrade or patch Beam if the restriction-sizing heuristic is unsuitable for your workload"],"exampleFix":"// before\nbeam.dataframe.io.read_csv('gs://bucket/data/part-*')  # 500 files of ~1GB each\n// after\n# fewer, smaller files, e.g. 100 files of ~500MB\nbeam.dataframe.io.read_csv('gs://bucket/data_repartitioned/part-*')","handlingStrategy":"validation","validationCode":"import math\nn = len(files)\nstride = 10 ** int(math.log(2**63 // n, 10))\noversized = [f for f in files if os.path.getsize(f) > stride]\nif oversized:\n    raise ValueError(f'{len(oversized)} files exceed safe index stride {stride}; split them or reduce file count')","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep input file sizes well under 10**floor(log10(2^63 / file_count)) bytes","Repartition large datasets into many modestly sized files before reading with beam.dataframe.io","Estimate the stride with the same formula (10**int(log10(2^63 // n))) during pipeline planning"],"tags":["python","apache-beam","dataframe","integer-overflow"],"backgroundTag":"value-out-of-range","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}