{"record":{"id":"9086736e9436066c","repo":"apache/beam","slug":"path-already-exists-s","errorCode":null,"errorMessage":"Path already exists: %s","messagePattern":"Path already exists: (.+?)","errorType":"exception","errorClass":"BeamIOError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/hadoopfilesystem.py","lineNumber":86,"sourceCode":"    self._hdfs_client = hdfs_client\n    self._path = path\n    self._size = self._hdfs_client.status(path)[_FILE_STATUS_LENGTH]\n\n  @property\n  def size(self):\n    return self._size\n\n  def get_range(self, start, end):\n    with self._hdfs_client.read(self._path, offset=start,\n                                length=end - start) as reader:\n      return reader.read()\n\n\nclass HdfsUploader(filesystemio.Uploader):\n  def __init__(self, hdfs_client, path):\n    self._hdfs_client = hdfs_client\n    if self._hdfs_client.status(path, strict=False) is not None:\n      raise BeamIOError('Path already exists: %s' % path)\n\n    self._handle_context = self._hdfs_client.write(path)\n    self._handle = self._handle_context.__enter__()\n\n  def put(self, data):\n    # hdfs uses an async writer which first add data to a queue. To avoid buffer\n    # gets reused upstream a deepcopy is required here.\n    self._handle.write(bytes(data))\n\n  def finish(self):\n    self._handle.__exit__(None, None, None)\n    self._handle = None\n    self._handle_context = None\n\n\nclass HadoopFileSystem(FileSystem):\n  \"\"\"``FileSystem`` implementation that supports HDFS.\n","sourceCodeStart":68,"sourceCodeEnd":104,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/hadoopfilesystem.py#L68-L104","documentation":"HdfsUploader.__init__ checks the target path on HDFS via the hdfs client before opening a write handle. If the path already exists on the remote filesystem, it raises BeamIOError 'Path already exists: %s' to prevent overwriting existing data during upload.","triggerScenarios":"Calling put/upload operations that construct an HdfsUploader for a path where hdfs_client.status(path, strict=False) returns non-None, e.g. writing to an HDFS location that already contains a file or directory of the same name.","commonSituations":"Re-running a pipeline that writes to the same output path without cleanup; concurrent writers racing to the same destination; stale files left from earlier failed jobs.","solutions":["Delete or rename the existing HDFS path before uploading (e.g. hdfs dfs -rm -r /path or client.delete(path)).","Write to a unique path per run (timestamped or UUID-suffixed directory).","Check existence first with client.status(path) and skip or pick another name.","Enable pipeline-level overwrite/unique-output options if the framework provides them."],"exampleFix":"// before\nuploader = HdfsUploader(hdfs_client, '/user/output/data.txt')\n// after\nif hdfs_client.status('/user/output/data.txt', strict=False):\n    hdfs_client.delete('/user/output/data.txt')\nuploader = HdfsUploader(hdfs_client, '/user/output/data.txt')","handlingStrategy":"validation","validationCode":"def path_exists(client, path):\n    return client.status(path, strict=False) is not None\nif path_exists(hdfs_client, dest):\n    hdfs_client.delete(dest)","typeGuard":"None","tryCatchPattern":"try:\n    uploader = HdfsUploader(hdfs_client, dest)\nexcept BeamIOError as e:\n    if 'Path already exists' in str(e):\n        hdfs_client.delete(dest)\n        uploader = HdfsUploader(hdfs_client, dest)\n    else:\n        raise","preventionTips":["Use timestamped/unique output paths per run","Clean up or rename prior outputs before reruns","Check client.status(path, strict=False) before writing","Avoid multiple concurrent writers to the same path"],"tags":["hdfs","io","conflict","apache-beam"],"backgroundTag":"file-already-exists","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}