What causes "Path already exists: %s" in apache/beam?

Delete or rename the existing HDFS path before uploading (e.g. hdfs dfs -rm -r /path or client.delete(path)).

apache/beam · error · BeamIOError

Path already exists

Error message

Path already exists: %s

What it means

HdfsUploader.__init__ checks the target path on HDFS via the hdfs client before opening a write handle. If the path already exists on the remote filesystem, it raises BeamIOError 'Path already exists: %s' to prevent overwriting existing data during upload.

Solutions

Delete or rename the existing HDFS path before uploading (e.g. hdfs dfs -rm -r /path or client.delete(path)).
Write to a unique path per run (timestamped or UUID-suffixed directory).
Check existence first with client.status(path) and skip or pick another name.
Enable pipeline-level overwrite/unique-output options if the framework provides them.

Example fix

// before
uploader = HdfsUploader(hdfs_client, '/user/output/data.txt')
// after
if hdfs_client.status('/user/output/data.txt', strict=False):
    hdfs_client.delete('/user/output/data.txt')
uploader = HdfsUploader(hdfs_client, '/user/output/data.txt')

Defensive patterns

Strategy: validation

Validate before calling

def path_exists(client, path):
    return client.status(path, strict=False) is not None
if path_exists(hdfs_client, dest):
    hdfs_client.delete(dest)

Type guard

None

Try / catch

try:
    uploader = HdfsUploader(hdfs_client, dest)
except BeamIOError as e:
    if 'Path already exists' in str(e):
        hdfs_client.delete(dest)
        uploader = HdfsUploader(hdfs_client, dest)
    else:
        raise

Prevention

Use timestamped/unique output paths per run
Clean up or rename prior outputs before reruns
Check client.status(path, strict=False) before writing
Avoid multiple concurrent writers to the same path

When it happens

Trigger: Calling put/upload operations that construct an HdfsUploader for a path where hdfs_client.status(path, strict=False) returns non-None, e.g. writing to an HDFS location that already contains a file or directory of the same name.

Common situations: Re-running a pipeline that writes to the same output path without cleanup; concurrent writers racing to the same destination; stale files left from earlier failed jobs.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9086736e9436066c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/hadoopfilesystem.py:86

    self._hdfs_client = hdfs_client
    self._path = path
    self._size = self._hdfs_client.status(path)[_FILE_STATUS_LENGTH]

  @property
  def size(self):
    return self._size

  def get_range(self, start, end):
    with self._hdfs_client.read(self._path, offset=start,
                                length=end - start) as reader:
      return reader.read()


class HdfsUploader(filesystemio.Uploader):
  def __init__(self, hdfs_client, path):
    self._hdfs_client = hdfs_client
    if self._hdfs_client.status(path, strict=False) is not None:
      raise BeamIOError('Path already exists: %s' % path)

    self._handle_context = self._hdfs_client.write(path)
    self._handle = self._handle_context.__enter__()

  def put(self, data):
    # hdfs uses an async writer which first add data to a queue. To avoid buffer
    # gets reused upstream a deepcopy is required here.
    self._handle.write(bytes(data))

  def finish(self):
    self._handle.__exit__(None, None, None)
    self._handle = None
    self._handle_context = None


class HadoopFileSystem(FileSystem):
  """``FileSystem`` implementation that supports HDFS.

View on GitHub (pinned to 12126d8942)