apache/hadoop · error · AbfsRestOperationException

PathNotFound

PathNotFound

Error message

openFileForWrite must be used with files and not directories

What it means

openFileForWrite backs FS.create(), FS.append(), and FS.openFile(...WRITE). It first issues GetPathStatus; if the existing path is a directory it aborts with an AbfsRestOperationException whose status code and error code are borrowed from AzureServiceErrorCode.PATH_NOT_FOUND (HTTP 404, PathNotFound). The 404 code is misleading - the path exists, it is just the wrong resource type, as the message says.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java:1107

  public OutputStream openFileForWrite(final Path path,
      final FileSystem.Statistics statistics, final boolean overwrite,
      TracingContext tracingContext) throws IOException {
    try (AbfsPerfInfo perfInfo = startTracking("openFileForWrite", "getPathStatus")) {
      LOG.debug("openFileForWrite filesystem: {} path: {} overwrite: {}",
              getClient().getFileSystem(),
              path,
              overwrite);

      String relativePath = getRelativePath(path);
      AbfsClient writeClient = getClientHandler().getIngressClient();

      final AbfsRestOperation op = getClient()
          .getPathStatus(relativePath, false, tracingContext, null);
      perfInfo.registerResult(op.getResult());

      if (getClient().checkIsDir(op.getResult())) {
        throw new AbfsRestOperationException(
              AzureServiceErrorCode.PATH_NOT_FOUND.getStatusCode(),
              AzureServiceErrorCode.PATH_NOT_FOUND.getErrorCode(),
              "openFileForWrite must be used with files and not directories",
              null);
      }

      final long contentLength = extractContentLength(op.getResult());
      final long offset = overwrite ? 0 : contentLength;

      perfInfo.registerSuccess(true);

      boolean isAppendBlob = false;
      if (isAppendBlobKey(path.toString())) {
        isAppendBlob = true;
      }

      AbfsLease lease = maybeCreateLease(relativePath, tracingContext);
      final String eTag = extractEtagHeader(op.getResult());

View on GitHub (pinned to 2add963021)

Solutions

1. Remove or rename the conflicting directory before creating the file: fs.delete(dirPath, true) or rename it, then retry create.
2. Pre-check with fs.getFileStatus(path).isFile() when the path is expected to exist, and fail with a clear upstream message.
3. Fix caller-side path construction: no trailing slash on file paths, distinct names for directories and files at the same level.

Example fix

// before
try (FSDataOutputStream out = fs.create outputPath) { ... }

// after
if (fs.exists(outputPath) && !fs.getFileStatus(outputPath).isFile()) {
  throw new IOException("Output path is a directory: " + outputPath);
}
try (FSDataOutputStream out = fs.create(outputPath, true)) { ... }

if (fs.exists(path)) {
  FileStatus st = fs.getFileStatus(path);
  if (st.isDirectory()) {
    throw new IOException("Refusing to open for write, path is a directory: " + path);
  }
}

try { out = fs.create(path, overwrite); } catch (AbfsRestOperationException e) {
  if ("PathNotFound".equals(e.getErrorCode()) && fs.exists(path) && fs.getFileStatus(path).isDirectory()) {
    // real cause: directory at file path - not a 404
  }
}

When it happens

Trigger: fs.create(path, overwrite) or fs.append(path), or openFile with WRITE/CREATE options, on a path that currently exists as a directory (previously created by mkdir). Races where another actor mkdirs the path between check and write also produce it.

Common situations: Output committers/partition writers colliding with partition directory names (e.g. trying to write file 'date=2024' when 'date=2024/' was created as a directory); re-running jobs that created directories on prior attempts; path-building bugs appending '/' or reusing directory names as file names.

Related errors

• File doesn't have encryptionContext.
• Cannot create file {} because parent folder does not exist.
• PathConflict
• Parallel access to the create path detected. Failing request
• <path> already exists as a directory

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/56a52a687a946172. Report an issue: GitHub.