What causes "{path}: No such file or directory!" in apache/hadoop?

Verify the path exists first with fs.exists(path) (which uses the same listing) and fix the URL/bucket/prefix typos

apache/hadoop · error · FileNotFoundException

{path}: No such file or directory!

Error message

{path}: No such file or directory!

What it means

Thrown by AliyunOSSFileSystem.getFileStatus(): no object metadata exists for the key (meta == null), and a listing under that prefix returned neither object summaries nor common prefixes through the final (non-truncated) page. The connector therefore concludes nothing exists at the path and throws FileNotFoundException(path + ": No such file or directory!").

Source

Thrown at hadoop-tools/hadoop-aliyun/src/main/java/org/apache/hadoop/fs/aliyun/oss/AliyunOSSFileSystem.java:291

    ObjectMetadata meta = store.getObjectMetadata(key);
    // If key not found and key does not end with "/"
    if (meta == null && !key.endsWith("/")) {
      // In case of 'dir + "/"'
      key += "/";
      meta = store.getObjectMetadata(key);
    }
    if (meta == null) {
      OSSListRequest listRequest = store.createListObjectsRequest(key,
          maxKeys, null, null, false);
      OSSListResult listing = store.listObjects(listRequest);
      do {
        if (CollectionUtils.isNotEmpty(listing.getObjectSummaries()) ||
            CollectionUtils.isNotEmpty(listing.getCommonPrefixes())) {
          return new OSSFileStatus(0, true, 1, 0, 0, qualifiedPath, username);
        } else if (listing.isTruncated()) {
          listing = store.continueListObjects(listRequest, listing);
        } else {
          throw new FileNotFoundException(
              path + ": No such file or directory!");
        }
      } while (true);
    } else if (objectRepresentsDirectory(key, meta.getContentLength())) {
      return new OSSFileStatus(0, true, 1, 0, meta.getLastModified().getTime(),
          qualifiedPath, username);
    } else {
      return new OSSFileStatus(meta.getContentLength(), false, 1,
          getDefaultBlockSize(path), meta.getLastModified().getTime(),
          qualifiedPath, username);
    }
  }

  @Override
  public String getScheme() {
    return "oss";
  }

View on GitHub (pinned to 2add963021)

Solutions

Verify the path exists first with fs.exists(path) (which uses the same listing) and fix the URL/bucket/prefix typos
If reading job output, wait for the job/commit to finish before opening the file
Confirm fs.oss.endpoint and bucket name are correct by listing the parent directory

Example fix

// before
FSDataInputStream in = fs.open(new Path("oss://bucket/missing"));

// after
Path p = new Path("oss://bucket/missing");
if (!fs.exists(p)) {
  throw new FileNotFoundException("Output not committed yet: " + p);
}
FSDataInputStream in = fs.open(p);

Defensive patterns

Strategy: try-catch

Validate before calling

if (!fs.exists(path)) {
  throw new FileNotFoundException("Path absent (not yet committed or wrong prefix): " + path);
}

Try / catch

catch (FileNotFoundException e) { /* expected during racing reads: poll exists() or wait for job completion, do not retry open() blindly */ }

Prevention

Gate reads on job/commit completion before opening output paths
Validate bucket and endpoint config with a listStatus of the parent before first read
Use exists() (same listing semantics) for cheap pre-checks

When it happens

Trigger: Calling getFileStatus, open, rename source, delete, or listStatus on a path whose key has no object and no children; reading output before the writer committed (committers often make files visible only at commit); typo'd bucket/prefix in the URL.

Common situations: Race where a reader queries a path a concurrent job has not yet created; wrong fs.oss.endpoint or bucket so listings target the wrong store; keys created outside Hadoop with different prefix conventions; FileOutputCommitter v1 semantics hiding output until task commit.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/783366edc5999fc1. Report an issue: GitHub.