apache/hadoop · error · AbfsRestOperationException

PathConflict

PathConflict

Error message

The specified path, or an element of the path, exists and its resource type is invalid for this operation.

What it means

In conditionalCreateOverwriteFile on a non-HNS account, the client first probes the path with GetPathStatus; if something already exists there and overwrite=false, it throws AbfsRestOperationException with HTTP 409, AzureServiceErrorCode.PathConflict, and the service-style message 'The specified path, or an element of the path, exists and its resource type is invalid for this operation.' The client synthesizes the same conflict the DFS endpoint would return, before even attempting the create.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:518

      final ContextEncryptionAdapter contextEncryptionAdapter,
      final TracingContext tracingContext) throws AzureBlobFileSystemException {
    AbfsRestOperation op;
    if (isFileCreation) {
      if (getAbfsConfiguration().getIsCreateIdempotencyEnabled()) {
        AbfsRestOperation statusOp = null;
        try {
          // Check if the file already exists by calling GetPathStatus
          statusOp = getPathStatus(path, tracingContext, null, false);
        } catch (AbfsRestOperationException ex) {
          // If the path does not exist, continue with file creation
          // For other errors, rethrow the exception
          if (ex.getStatusCode() != HTTP_NOT_FOUND) {
            throw ex;
          }
        }
        // If the file exists and overwrite is not allowed, throw conflict
        if (statusOp != null && statusOp.hasResult() && !overwrite) {
          throw new AbfsRestOperationException(
              HTTP_CONFLICT,
              AzureServiceErrorCode.PATH_CONFLICT.getErrorCode(),
              PATH_EXISTS,
              null);
        } else {
          // Proceed with file creation (force overwrite = true)
          op = createFile(path, true, permissions, isAppendBlob, eTag,
              contextEncryptionAdapter, tracingContext);
        }
      } else {
        op = createFile(path, overwrite, permissions, isAppendBlob, eTag,
            contextEncryptionAdapter, tracingContext);
      }
    } else {
      // Create a directory with the specified parameters
      op = createDirectory(path, permissions, isAppendBlob, eTag,
          contextEncryptionAdapter, tracingContext);
    }

View on GitHub (pinned to 2add963021)

Solutions

1. Pass overwrite=true when clobbering prior output is acceptable: fs.create(path, true)
2. Delete or rename the pre-existing path before creating
3. Catch the 409/PathConflict and implement idempotent 'already done' logic for restarted jobs

Example fix

// before: refuses to clobber -> AbfsRestOperationException 409 PathConflict
try (FSDataOutputStream out = fs.create(outPath, false)) { ... }

// after: explicit overwrite
try (FSDataOutputStream out = fs.create(outPath, true)) { ... }

if (fs.exists(path)) {
  if (!overwriteAllowed) throw new IllegalStateException("output already exists: " + path);
  fs.delete(path, false);
}
try (FSDataOutputStream out = fs.create(path, true)) { ... }

try {
  fs.create(path, false);
} catch (AbfsRestOperationException e) {
  if (e.getStatusCode() == 409
      && AzureServiceErrorCode.PATH_CONFLICT.getErrorCode().equals(e.getErrorCode())) {
    // target occupied: decide overwrite vs fail
  } else { throw e; }
}

When it happens

Trigger: fs.create(path, false) (overwrite=false) when getPathStatus succeeds for that path on a flat-namespace account — any file or blob already occupies the target.

Common situations: Job reruns without cleaning prior output; leftover temp files from a failed committer; the overwrite flag not propagated (FSDataOutputStream created through APIs defaulting to overwrite=false).

Related errors

• Cannot create file {} because parent folder does not exist.
• Parallel access to the create path detected. Failing request
• PathNotFound
• Both global key and encryption context are set, only one all
• Parallel access to the create path detected. Failing request

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3635a79f92429bf1. Report an issue: GitHub.