juicedata/juicefs · error · IOException

The directory item limit of " + controlDir + "

Error message

The directory item limit of " + controlDir +
              " is exceeded: limit=" + maxDirItems + " items=" + maps

What it means

TestDFSIO.createControlFile checks the requested number of maps against the NameNode directory-item limit dfs.namenode.fs-limits.max-directory-items (default 1M) and throws IOException with this message when maps would exceed the number of entries the control directory can hold. The benchmark creates one control file per map task in controlDir, so the NameNode limit would be violated.

Source

Thrown at sdk/java/src/main/java/io/juicefs/bench/TestDFSIO.java:410

  private Path getDataDir(Configuration conf) {
    return new Path(getBaseDir(conf), "io_data");
  }


  @SuppressWarnings("deprecation")
  private void createControlFile(FileSystem fs,
                                 long nrBytes, // in bytes
                                 int maps
  ) throws IOException {
    LOG.info("creating control file: " + nrBytes + " bytes, " + maps + " files");
    final int maxDirItems = config.getInt("dfs.namenode.fs-limits.max-directory-items", 1024 * 1024);
    Path controlDir = getControlDir(config);

    if (maps > maxDirItems) {
      final String message = "The directory item limit of " + controlDir +
              " is exceeded: limit=" + maxDirItems + " items=" + maps;
      throw new IOException(message);
    }

    fs.delete(controlDir, true);

    for (int i = 0; i < maps; i++) {
      String name = getFileName(i);
      Path controlFile = new Path(controlDir, "in_file_" + name);
      SequenceFile.Writer writer = null;
      try {
        writer = SequenceFile.createWriter(fs, config, controlFile,
                Text.class, LongWritable.class,
                CompressionType.NONE);
        writer.append(new Text(name), new LongWritable(nrBytes));
      } catch (Exception e) {
        throw new IOException(e.getLocalizedMessage());
      } finally {
        if (writer != null) {
          writer.close();

View on GitHub (pinned to c9a67b23e8)

Solutions

Reduce -maps to at most dfs.namenode.fs-limits.max-directory-items (default 1048576).
Raise the limit on the NameNode (set dfs.namenode.fs-limits.max-directory-items higher and restart) if you truly need more files.
Point the control directory at a filesystem/location with an adequate limit.
Split the benchmark into multiple sequential runs with fewer maps each.

Example fix

// before
TestDFSIO -write -maps 2000000 -size 1GB
// after
TestDFSIO -write -maps 100000 -size 1GB

Defensive patterns

Strategy: validation

Validate before calling

int maxDirItems = config.getInt("dfs.namenode.fs-limits.max-directory-items", 1048576);
if (maps > maxDirItems) {
  throw new IllegalArgumentException("reduce -maps to <= " + maxDirItems);
}

Try / catch

try {
  runTestDFSIO(args);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("directory item limit")) {
    throw new IllegalArgumentException("-maps exceeds max-directory-items; lower -maps or raise the limit", e);
  }
  throw e;
}

Prevention

Keep -maps well under 1,048,576 unless the cluster limit was raised.
Check dfs.namenode.fs-limits.max-directory-items on the target cluster before stress runs.
Split very large benchmarks into sequential smaller runs.

When it happens

Trigger: Launching TestDFSIO with -maps greater than the max-directory-items limit of the filesystem backing the control directory (controlDirPath), so createControlFile refuses to write per-map control files.

Common situations: Stress-testing with very high map counts (e.g. -maps 2000000) while the target FS keeps the default 1,048,576 limit; running against a cluster where the operator lowered max-directory-items; pointing controlDir at a small-config JuiceFS volume.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/e4749dc1045f9f83. Report an issue: GitHub.