apache/hadoop · error · FileAlreadyExistsException

Object %s already exists.

Error message

Object %s already exists.

What it means

getWriteGeneration(resourceId, overwrite) implements create-without-overwrite semantics using GCS object generations: for a non-existing object it returns 0, for an existing object with overwrite=true it returns the current generation to satisfy, and for an existing object with overwrite=false it throws FileAlreadyExistsException. The generationId is then used as a precondition on the write, so concurrent creators race safely and the loser gets this exception.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorage.java:137

   * @param resourceId object for which generation info is requested
   * @param overwrite  whether existing object should be overwritten
   * @return the generation of the object
   * @throws IOException if the object already exists and cannot be overwritten
   */
  private long getWriteGeneration(StorageResourceId resourceId, boolean overwrite)
      throws IOException {
    LOG.trace("getWriteGeneration({}, {})", resourceId, overwrite);
    GoogleCloudStorageItemInfo info = getItemInfo(resourceId);
    if (!info.exists()) {
      return 0L;
    }
    if (info.exists() && overwrite) {
      long generation = info.getContentGeneration();
      checkState(generation != 0, "Generation should not be 0 for an existing item");
      return generation;
    }

    throw new FileAlreadyExistsException(String.format("Object %s already exists.", resourceId));
  }

  void close() {
    try {
      storage.close();
    } catch (Exception e) {
      LOG.warn("Error occurred while closing the storage client", e);
    }
  }

  GoogleCloudStorageItemInfo getItemInfo(StorageResourceId resourceId) throws IOException {
    LOG.trace("getItemInfo({})", resourceId);

    // Handle ROOT case first.
    if (resourceId.isRoot()) {
      return GoogleCloudStorageItemInfo.ROOT_INFO;
    }
    GoogleCloudStorageItemInfo itemInfo = null;

View on GitHub (pinned to 2add963021)

Solutions

1. Delete or move the existing object before re-creating it (fs.delete(path, false) when exists)
2. Write with overwrite=true if clobbering is acceptable: fs.create(path, true)
3. Use unique output paths per attempt (task attempt IDs) as the built-in committers do
4. On MapReduce/Spark, ensure the previous job's _temporary tree was cleaned before rerun

Example fix

// before
FSDataOutputStream out = fs.create(path, false); // throws FileAlreadyExistsException

// after
if (fs.exists(path)) {
  fs.delete(path, false);
}
FSDataOutputStream out = fs.create(path, false);

// Check before exclusive create
if (fs.exists(path)) {
  fs.delete(path, false); // or choose a unique output name
}
FSDataOutputStream out = fs.create(path, false);

boolean isAlreadyExists(IOException e) {
  return e instanceof FileAlreadyExistsException;
}

try {
  out = fs.create(path, false);
} catch (FileAlreadyExistsException e) {
  // another writer won the create race — pick a new name or verify and skip
  path = uniqueAttemptPath(path);
  out = fs.create(path, false);
}

When it happens

Trigger: Calling create() on a GCS path that already exists while overwrite is false (fs.create(path, false), or committers that create output objects exclusively); two writers targeting the same gs://bucket/object simultaneously; job retry re-creating the same task output.

Common situations: Re-running a job without cleaning previous _temporary/output directories; custom OutputCommitter or direct-storage writers using overwrite=false; concurrent tasks computing the same deterministic output path.

Related errors

• Error accessing Bucket %s
• Error accessing %s
• Bucket doesn't match for source '%s' and destination '%s'!
• Listing '%s' failed
• Deleting resource %s failed.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c656296d6ae6a339. Report an issue: GitHub.