What causes "key + ": No such file or directory."" in apache/hadoop?

Guard with exists()/getFileStatus() before getFileChecksum when the file may be gone

apache/hadoop · error · FileNotFoundException

key + ": No such file or directory."

Error message

key + ": No such file or directory."

What it means

BosNativeFileSystemStore.getFileChecksum(key) HEADs the object to read its x-bce-crc32c metadata; a FileNotFoundException from that HEAD is rethrown as FileNotFoundException('<key>: No such file or directory.'). A present object without CRC32C metadata returns null instead — only a missing key produces this error.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BosNativeFileSystemStore.java:351

   *
   * @param key the object key
   * @return the file checksum, or null if unavailable
   * @throws IOException if the key does not exist or an I/O
   *                     error occurs
   */
  public FileChecksum getFileChecksum(String key)
      throws IOException {
    try {
      ObjectMetadata metadata =
          bosClientProxy.getObjectMetadata(
              bucketName, key);
      if (metadata.getxBceCrc32c() == null) {
        return null;
      }
      return new BosCRC32CCheckSum(
          metadata.getxBceCrc32c());
    } catch (FileNotFoundException e) {
      throw new FileNotFoundException(
          key + ": No such file or directory.");
    }
  }

  /**
   * Checks whether the given key represents a directory.
   *
   * @param key the object key
   * @return true if the key is a directory, false if it is a
   *         file or does not exist
   * @throws IOException if an I/O error occurs
   */
  public abstract boolean isDirectory(String key)
      throws IOException;

  /**
   * Retrieves the metadata for the specified key.
   *

View on GitHub (pinned to 2add963021)

Solutions

Guard with exists()/getFileStatus() before getFileChecksum when the file may be gone
For distcp races, pass -skipcrccheck or retry the copy once the source/target is stable
If the file definitely exists, verify the path-to-key mapping: bucket endpoint, fs.bos prefix config, and URI encoding of special characters

Example fix

// before
FileChecksum c = fs.getFileChecksum(path);

// after
if (!fs.exists(path)) {
  throw new FileNotFoundException(path + " already deleted");
}
FileChecksum c = fs.getFileChecksum(path);

Defensive patterns

Strategy: validation

Validate before calling

if (!fs.exists(path)) {
  // handle absent file explicitly instead of calling getFileChecksum
  return Optional.empty();
}
return Optional.of(fs.getFileChecksum(path));

Try / catch

catch (FileNotFoundException e) {
  // expected when files are deleted concurrently: skip or retry once
  LOG.debug("file vanished before checksum: {}", path);
}

Prevention

Guard checksum calls with exists() when deletes are possible
Use -skipcrccheck in distcp for racy sources
Encode/normalize keys once at the boundary to avoid HEAD on a wrong key

When it happens

Trigger: Calling FileSystem.getFileChecksum(path) on a key that does not exist: file deleted between listing and checksum call, or the path translates to a different key than intended (prefix/endpoint mismatch, double URI encoding).

Common situations: distcp with checksum comparison racing a concurrent delete or an in-flight write; Hive/Impala verification steps; keys containing characters that get re-encoded ('+', '%', spaces) so the HEAD targets a different key.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1e5661d45e8e3b46. Report an issue: GitHub.