{"record":{"id":"e4749dc1045f9f83","repo":"juicedata/juicefs","slug":"the-directory-item-limit-of-controldir","errorCode":null,"errorMessage":"The directory item limit of \" + controlDir +\n              \" is exceeded: limit=\" + maxDirItems + \" items=\" + maps","messagePattern":"The directory item limit of \" \\+ controlDir \\+\n              \" is exceeded: limit=\" \\+ maxDirItems \\+ \" items=\" \\+ maps","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"sdk/java/src/main/java/io/juicefs/bench/TestDFSIO.java","lineNumber":410,"sourceCode":"\n  private Path getDataDir(Configuration conf) {\n    return new Path(getBaseDir(conf), \"io_data\");\n  }\n\n\n  @SuppressWarnings(\"deprecation\")\n  private void createControlFile(FileSystem fs,\n                                 long nrBytes, // in bytes\n                                 int maps\n  ) throws IOException {\n    LOG.info(\"creating control file: \" + nrBytes + \" bytes, \" + maps + \" files\");\n    final int maxDirItems = config.getInt(\"dfs.namenode.fs-limits.max-directory-items\", 1024 * 1024);\n    Path controlDir = getControlDir(config);\n\n    if (maps > maxDirItems) {\n      final String message = \"The directory item limit of \" + controlDir +\n              \" is exceeded: limit=\" + maxDirItems + \" items=\" + maps;\n      throw new IOException(message);\n    }\n\n    fs.delete(controlDir, true);\n\n    for (int i = 0; i < maps; i++) {\n      String name = getFileName(i);\n      Path controlFile = new Path(controlDir, \"in_file_\" + name);\n      SequenceFile.Writer writer = null;\n      try {\n        writer = SequenceFile.createWriter(fs, config, controlFile,\n                Text.class, LongWritable.class,\n                CompressionType.NONE);\n        writer.append(new Text(name), new LongWritable(nrBytes));\n      } catch (Exception e) {\n        throw new IOException(e.getLocalizedMessage());\n      } finally {\n        if (writer != null) {\n          writer.close();","sourceCodeStart":392,"sourceCodeEnd":428,"githubUrl":"https://github.com/juicedata/juicefs/blob/c9a67b23e8e08ec23ec331aa6f1675e2319e921c/sdk/java/src/main/java/io/juicefs/bench/TestDFSIO.java#L392-L428","documentation":"TestDFSIO.createControlFile checks the requested number of maps against the NameNode directory-item limit dfs.namenode.fs-limits.max-directory-items (default 1M) and throws IOException with this message when maps would exceed the number of entries the control directory can hold. The benchmark creates one control file per map task in controlDir, so the NameNode limit would be violated.","triggerScenarios":"Launching TestDFSIO with -maps greater than the max-directory-items limit of the filesystem backing the control directory (controlDirPath), so createControlFile refuses to write per-map control files.","commonSituations":"Stress-testing with very high map counts (e.g. -maps 2000000) while the target FS keeps the default 1,048,576 limit; running against a cluster where the operator lowered max-directory-items; pointing controlDir at a small-config JuiceFS volume.","solutions":["Reduce -maps to at most dfs.namenode.fs-limits.max-directory-items (default 1048576).","Raise the limit on the NameNode (set dfs.namenode.fs-limits.max-directory-items higher and restart) if you truly need more files.","Point the control directory at a filesystem/location with an adequate limit.","Split the benchmark into multiple sequential runs with fewer maps each."],"exampleFix":"// before\nTestDFSIO -write -maps 2000000 -size 1GB\n// after\nTestDFSIO -write -maps 100000 -size 1GB","handlingStrategy":"validation","validationCode":"int maxDirItems = config.getInt(\"dfs.namenode.fs-limits.max-directory-items\", 1048576);\nif (maps > maxDirItems) {\n  throw new IllegalArgumentException(\"reduce -maps to <= \" + maxDirItems);\n}","typeGuard":null,"tryCatchPattern":"try {\n  runTestDFSIO(args);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"directory item limit\")) {\n    throw new IllegalArgumentException(\"-maps exceeds max-directory-items; lower -maps or raise the limit\", e);\n  }\n  throw e;\n}","preventionTips":["Keep -maps well under 1,048,576 unless the cluster limit was raised.","Check dfs.namenode.fs-limits.max-directory-items on the target cluster before stress runs.","Split very large benchmarks into sequential smaller runs."],"tags":["java","hadoop","benchmark","limits"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"c9a67b23e8e08ec23ec331aa6f1675e2319e921c","analyzedAt":"2026-09-06T17:55:48.476Z","contentChangedAt":"2026-09-06T17:55:48.476Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}