{"record":{"id":"8895f932583a918b","repo":"apache/hadoop","slug":"too-many-distribution-intervals-numintervals","errorCode":null,"errorMessage":"Too many distribution intervals {numIntervals}","messagePattern":"Too many distribution intervals (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionVisitor.java","lineNumber":92,"sourceCode":"  /**\n   * File or directory information.\n   */\n  private static class FileContext {\n    String path;\n    long fileSize;\n    int numBlocks;\n    int replication;\n  }\n\n  public FileDistributionVisitor(String filename, long maxSize, int step,\n      boolean formatOutput) throws IOException {\n    super(filename, false);\n    this.maxSize = (maxSize == 0 ? MAX_SIZE_DEFAULT : maxSize);\n    this.step = (step == 0 ? INTERVAL_DEFAULT : step);\n    this.formatOutput = formatOutput;\n    long numIntervals = this.maxSize / this.step;\n    if(numIntervals >= Integer.MAX_VALUE)\n      throw new IOException(\"Too many distribution intervals \" + numIntervals);\n    this.distribution = new int[1 + (int)(numIntervals)];\n    this.totalFiles = 0;\n    this.totalDirectories = 0;\n    this.totalBlocks = 0;\n    this.totalSpace = 0;\n    this.maxFileSize = 0;\n  }\n\n  @Override\n  void start() throws IOException {}\n\n  @Override\n  void finish() throws IOException {\n    output();\n    super.finish();\n  }\n\n  @Override","sourceCodeStart":74,"sourceCodeEnd":110,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionVisitor.java#L74-L110","documentation":"Thrown by the FileDistribution processor of the legacy OfflineImageViewer when the constructor computes numIntervals = maxSize / step and the quotient reaches Integer.MAX_VALUE, because the histogram is allocated as int[1 + numIntervals]. maxSize defaults to 0x2000000000L (128 GiB) and step to 0x200000 (2 MiB); passing 0 for either selects the default rather than zero. It is a pure argument-sanity check that fires before any fsimage byte is read.","triggerScenarios":"Running `hdfs oiv_legacy -p FileDistribution -i <fsimage> -o <out> -maxSize <bytes> -step <bytes>` where maxSize/step >= 2147483647. Typical case: leaving -maxSize at the 137438953472-byte default while passing -step 64 (2^37/2^6 = 2^31), or -step 1 with any multi-GB -maxSize. Also fires when constructing new FileDistributionVisitor(filename, maxSize, step, formatOutput) directly with such values. Note -step is parsed with Integer.parseInt (OfflineImageViewer.java:253), so it must itself fit an int.","commonSituations":"Operators lowering -step from megabytes to bytes for a fine-grained histogram without shrinking -maxSize; scripts that feed raw byte counts into both options; commands copied from examples tuned for a much smaller range.","solutions":["Raise -step until maxSize/step < 2147483647 (e.g. keep step >= 128 for the default 128 GiB range)","Lower -maxSize if only small files matter (e.g. -maxSize 1073741824 with -step 1 gives 2^30 intervals, which fits)","Omit both options to accept the 128 GiB / 2 MiB defaults (65536 intervals)","If embedding the visitor in Java, pre-compute the interval count and reject bad pairs with a clear message before constructing the visitor"],"exampleFix":"# before\nhdfs oiv_legacy -p FileDistribution -i fsimage_0000000000000060000 -o dist.txt -maxSize 137438953472 -step 64\n# IOException: Too many distribution intervals 2147483648\n\n# after\nhdfs oiv_legacy -p FileDistribution -i fsimage_0000000000000060000 -o dist.txt -maxSize 137438953472 -step 128","handlingStrategy":"validation","validationCode":"long maxSize = Long.parseLong(cmd.getOptionValue(\"maxSize\", \"0\"));\nint step = Integer.parseInt(cmd.getOptionValue(\"step\", \"0\"));\nif (maxSize == 0) maxSize = 0x2000000000L; // 128 GiB default\nif (step == 0) step = 0x200000;            // 2 MiB default\nif (maxSize / step >= Integer.MAX_VALUE) {\n  throw new IllegalArgumentException(\n      \"maxSize/step = \" + (maxSize / step) + \" must be < \" + Integer.MAX_VALUE\n      + \" - increase -step or decrease -maxSize\");\n}\nnew FileDistributionVisitor(outputFile, maxSize, step, formatOutput);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Think of -step in megabytes unless -maxSize was deliberately shrunk","Remember 0 selects the default for both -maxSize and -step, not zero","The protobuf `hdfs oiv -p FileDistribution` (FileDistributionCalculator) enforces the same maxSize/step limit - apply the same check there"],"tags":["hdfs","offline-image-viewer","fsimage","file-distribution","input-validation"],"backgroundTag":"invalid-argument-value","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}