apache/beam · error · IOError

No such file or directory

Error message

No such file or directory: %s

What it means

FileBasedSourceVerifier._read_with_retry uses the Beam FileSystems API to match the given file path; if the match returns no metadata entries the path resolved to nothing, so it raises IOError 'No such file or directory'. This happens before any retry/read logic, meaning the file simply does not exist at the specified location.

Solutions

1. Verify the file_path exists (ls / gsutil ls) and correct the path
2. Ensure the pipeline writing the output has completed and files are flushed before verifying
3. Check the glob pattern — a too-narrow pattern may match nothing
4. Increase sleep_secs on the verifier so it waits for output readiness

Example fix

# before
FileBasedOutputVerifier('gs://wrong-bucket/out/*')
// after
FileBasedOutputVerifier('gs://my-bucket/correct/output-dir/*')

from apache_beam.io.filesystems import FileSystems
assert FileSystems.match([file_path]) and FileSystems.match([file_path])[0].metadata_list

try:
    verifier.match()
except IOError as e:
    _LOGGER.warning('Output not found yet (%s); retrying after sleep', e)
    time.sleep(sleep_secs)
    verifier.match()

When it happens

Trigger: Calling _matches (during verify()) with a file_path glob that matched zero files — path misspelled, file not yet written, wrong bucket/directory, or wrong filesystem scheme.

Common situations: Verifier runs before the pipeline finished writing output; glob pattern wrong for the temp/output location; reading from GCS with a mistyped bucket name.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Artifacts not found at location
• Could not find the provided transforms config source
• ENOENT
• err (re-raised OSError from os.makedirs)
• f"Error loading providers from

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/384bb53c00dd26cc. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/testing/pipeline_verifiers.py:120

      else:
        raise ValueError(
            'Sleep seconds, if received, must be int. '
            'But received: %r, %s' % (sleep_secs, type(sleep_secs)))
    else:
      self.sleep_secs = None

    self.file_path = file_path
    self.expected_checksum = expected_checksum

  @retry.with_exponential_backoff(
      num_retries=MAX_RETRIES, retry_filter=retry_on_io_error_and_server_error)
  def _read_with_retry(self):
    """Read path with retry if I/O failed"""
    read_lines = []
    match_result = FileSystems.match([self.file_path])[0]
    matched_path = [f.path for f in match_result.metadata_list]
    if not matched_path:
      raise IOError('No such file or directory: %s' % self.file_path)

    _LOGGER.info(
        'Find %d files in %s: \n%s',
        len(matched_path),
        self.file_path,
        '\n'.join(matched_path))
    for path in matched_path:
      with FileSystems.open(path, 'r') as f:
        for line in f:
          read_lines.append(line)
    return read_lines

  def _matches(self, _):
    if self.sleep_secs:
      # Wait to have output file ready on FS
      _LOGGER.info('Wait %d seconds...', self.sleep_secs)
      time.sleep(self.sleep_secs)

View on GitHub (pinned to 12126d8942)