apache/hadoop · error · FileNotFoundException

Can't open %s because it is a directory

Error message

Can't open %s because it is a directory

What it means

RawFileSystem.open stats the path first; when the target is a directory it throws FileNotFoundException with 'Can't open <path> because it is a directory'. Reading a prefix as a file has no meaning on object storage, so the open is refused before any range read starts.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/RawFileSystem.java:137

    return scheme;
  }

  @VisibleForTesting
  String bucket() {
    return bucket;
  }

  @Override
  public void setConf(Configuration conf) {
    super.setConf(conf);
  }

  @Override
  public FSDataInputStream open(Path path, int bufferSize) throws IOException {
    LOG.debug("Opening '{}' for reading.", path);
    RawFileStatus status = innerFileStatus(path);
    if (status.isDirectory()) {
      throw new FileNotFoundException(
          String.format("Can't open %s because it is a directory", path));
    }

    // Parse the range size from the hadoop conf.
    long rangeSize = getConf().getLong(
        ConfKeys.FS_OBJECT_STREAM_RANGE_SIZE,
        ConfKeys.FS_OBJECT_STREAM_RANGE_SIZE_DEFAULT);
    Preconditions.checkArgument(rangeSize > 0, "Object storage range size must be positive.");

    FSInputStream fsIn = new ObjectMultiRangeInputStream(taskThreadPool, storage, path,
        status.getLen(), rangeSize, status.checksum());
    return new FSDataInputStream(fsIn);
  }

  public FSDataInputStream open(Path path, byte[] expectedChecksum, Range range) {
    return new FSDataInputStream(
        new ObjectRangeInputStream(storage, path, range, expectedChecksum));
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Point open() at files: list the directory or glob with /dir/*
2. Guard with getFileStatus(path).isDirectory() and fail with an actionable message
3. Use the framework input format (FileInputFormat, Spark read) for directory inputs

Example fix

// before
FSDataInputStream in = fs.open(new Path("/inputs/2024")); // directory

// after
for (FileStatus f : fs.listStatus(new Path("/inputs/2024"))) {
  if (f.isFile()) {
    try (FSDataInputStream in = fs.open(f.getPath())) {
      ... // read the file
    }
  }
}

FileStatus st = fs.getFileStatus(p);
if (st.isDirectory()) {
  throw new IOException("Refusing to open a directory: " + p);
}

static boolean isReadableFile(FileSystem fs, Path p) throws IOException {
  return fs.exists(p) && fs.getFileStatus(p).isFile();
}

try {
  return fs.open(p);
} catch (FileNotFoundException e) {
  if (e.getMessage() != null && e.getMessage().contains("because it is a directory")) {
    // expand the directory: list and open files instead
  } else {
    throw e;
  }
}

When it happens

Trigger: fs.open(dir) where dir exists only as a directory; passing a partition directory instead of files inside it; a missing glob so the input Path stays the directory itself.

Common situations: Input configuration points at the directory rather than files; a file reader API fed a directory path; job code that uses open() where the framework's input format was expected.

Related errors

• Directory {} is not empty.
• '{}' is a directory
• new destination is an existed directory
• {} is a directory
• Unable to recover task %s, output: %s

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/363ef43d9815fdc4. Report an issue: GitHub.