{"record":{"id":"64ed60b045133ad0","repo":"apache/beam","slug":"file-not-found-s","errorCode":null,"errorMessage":"File not found: %s","messagePattern":"File not found: (.+?)","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/hadoopfilesystem.py","lineNumber":438,"sourceCode":"        file_checksum[_FILE_CHECKSUM_BYTES],\n    )\n\n  def metadata(self, url):\n    \"\"\"Fetch metadata fields of a file on the FileSystem.\n\n    Args:\n      url: string url of a file.\n\n    Returns:\n      :class:`~apache_beam.io.filesystem.FileMetadata`.\n\n    Raises:\n      ``BeamIOError``: if url doesn't exist.\n    \"\"\"\n    _, path = self._parse_url(url)\n    status = self._hdfs_client.status(path, strict=False)\n    if status is None:\n      raise BeamIOError('File not found: %s' % url)\n    return FileMetadata(\n        url, status[_FILE_STATUS_LENGTH], status[_FILE_STATUS_UPDATED] / 1000.0)\n\n  def delete(self, urls):\n    exceptions = {}\n    for url in urls:\n      try:\n        _, path = self._parse_url(url)\n        self._hdfs_client.delete(path, recursive=True)\n      except Exception as e:  # pylint: disable=broad-except\n        exceptions[url] = e\n\n    if exceptions:\n      raise BeamIOError(\"Delete operation failed\", exceptions)\n","sourceCodeStart":420,"sourceCodeEnd":453,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/hadoopfilesystem.py#L420-L453","documentation":"HadoopFileSystem.metadata fetches file status via the libhdfs client with strict=False, which returns None instead of raising for missing paths; when status is None the method raises BeamIOError 'File not found: <url>'. size() and last_updated() call metadata(), so both propagate this error.","triggerScenarios":"Calling fs.metadata(url), fs.size(url), or fs.last_updated(url) with a URL that does not exist in HDFS — deleted file, wrong path, or a path only present on a different cluster/namespace.","commonSituations":"Checking file sizes for input sizing where upstream writes failed or files were TTL-deleted; typos in pipeline options pointing at input paths; racing with a cleanup job that removed the file.","solutions":["Guard with fs.exists(url) before calling metadata/size/last_updated.","Correct the URL/path — verify with hdfs dfs -ls <path> on the same cluster.","Check you are pointing at the right HDFS cluster/NameNode configuration.","Handle BeamIOError in callers and fall back to defaults or re-list the directory to find actual files.","If files may vanish transiently, retry with backoff before failing the pipeline."],"exampleFix":"# before\nsize = fs.size('hdfs://nn/data/file')\n# after\nif fs.exists('hdfs://nn/data/file'):\n    size = fs.size('hdfs://nn/data/file')","handlingStrategy":"validation","validationCode":"if not fs.exists(url):\n    return None  # or raise a clearer error\nmeta = fs.metadata(url)","typeGuard":null,"tryCatchPattern":"from apache_beam.io.filesystem import BeamIOError\ntry:\n    meta = fs.metadata(url)\nexcept BeamIOError as e:\n    if 'File not found' in str(e):\n        meta = None  # handle absence explicitly\n    else:\n        raise","preventionTips":["Always gate size/last_updated calls behind fs.exists().","Verify paths and cluster configuration when files unexpectedly vanish.","Handle TTL-based deletion of upstream data.","Fall back to listing the parent directory to discover real file names."],"tags":["python","apache-beam","hdfs","file-metadata"],"backgroundTag":"file-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}