What causes "File not found: %s" in apache/beam?

Guard with fs.exists(url) before calling metadata/size/last_updated.

apache/beam · error · BeamIOError

File not found

Error message

File not found: %s

What it means

HadoopFileSystem.metadata fetches file status via the libhdfs client with strict=False, which returns None instead of raising for missing paths; when status is None the method raises BeamIOError 'File not found: <url>'. size() and last_updated() call metadata(), so both propagate this error.

Solutions

Guard with fs.exists(url) before calling metadata/size/last_updated.
Correct the URL/path — verify with hdfs dfs -ls <path> on the same cluster.
Check you are pointing at the right HDFS cluster/NameNode configuration.
Handle BeamIOError in callers and fall back to defaults or re-list the directory to find actual files.
If files may vanish transiently, retry with backoff before failing the pipeline.

Example fix

# before
size = fs.size('hdfs://nn/data/file')
# after
if fs.exists('hdfs://nn/data/file'):
    size = fs.size('hdfs://nn/data/file')

Defensive patterns

Strategy: validation

Validate before calling

if not fs.exists(url):
    return None  # or raise a clearer error
meta = fs.metadata(url)

Try / catch

from apache_beam.io.filesystem import BeamIOError
try:
    meta = fs.metadata(url)
except BeamIOError as e:
    if 'File not found' in str(e):
        meta = None  # handle absence explicitly
    else:
        raise

Prevention

Always gate size/last_updated calls behind fs.exists().
Verify paths and cluster configuration when files unexpectedly vanish.
Handle TTL-based deletion of upstream data.
Fall back to listing the parent directory to discover real file names.

When it happens

Trigger: Calling fs.metadata(url), fs.size(url), or fs.last_updated(url) with a URL that does not exist in HDFS — deleted file, wrong path, or a path only present on a different cluster/namespace.

Common situations: Checking file sizes for input sizing where upstream writes failed or files were TTL-deleted; typos in pipeline options pointing at input paths; racing with a cleanup job that removed the file.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/64ed60b045133ad0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:438

        file_checksum[_FILE_CHECKSUM_BYTES],
    )

  def metadata(self, url):
    """Fetch metadata fields of a file on the FileSystem.

    Args:
      url: string url of a file.

    Returns:
      :class:`~apache_beam.io.filesystem.FileMetadata`.

    Raises:
      ``BeamIOError``: if url doesn't exist.
    """
    _, path = self._parse_url(url)
    status = self._hdfs_client.status(path, strict=False)
    if status is None:
      raise BeamIOError('File not found: %s' % url)
    return FileMetadata(
        url, status[_FILE_STATUS_LENGTH], status[_FILE_STATUS_UPDATED] / 1000.0)

  def delete(self, urls):
    exceptions = {}
    for url in urls:
      try:
        _, path = self._parse_url(url)
        self._hdfs_client.delete(path, recursive=True)
      except Exception as e:  # pylint: disable=broad-except
        exceptions[url] = e

    if exceptions:
      raise BeamIOError("Delete operation failed", exceptions)

View on GitHub (pinned to 12126d8942)