apache/hadoop · error · IOException

Too many distribution intervals {numIntervals}

Error message

Too many distribution intervals {numIntervals}

What it means

Thrown by the FileDistribution processor of the legacy OfflineImageViewer when the constructor computes numIntervals = maxSize / step and the quotient reaches Integer.MAX_VALUE, because the histogram is allocated as int[1 + numIntervals]. maxSize defaults to 0x2000000000L (128 GiB) and step to 0x200000 (2 MiB); passing 0 for either selects the default rather than zero. It is a pure argument-sanity check that fires before any fsimage byte is read.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionVisitor.java:92

  /**
   * File or directory information.
   */
  private static class FileContext {
    String path;
    long fileSize;
    int numBlocks;
    int replication;
  }

  public FileDistributionVisitor(String filename, long maxSize, int step,
      boolean formatOutput) throws IOException {
    super(filename, false);
    this.maxSize = (maxSize == 0 ? MAX_SIZE_DEFAULT : maxSize);
    this.step = (step == 0 ? INTERVAL_DEFAULT : step);
    this.formatOutput = formatOutput;
    long numIntervals = this.maxSize / this.step;
    if(numIntervals >= Integer.MAX_VALUE)
      throw new IOException("Too many distribution intervals " + numIntervals);
    this.distribution = new int[1 + (int)(numIntervals)];
    this.totalFiles = 0;
    this.totalDirectories = 0;
    this.totalBlocks = 0;
    this.totalSpace = 0;
    this.maxFileSize = 0;
  }

  @Override
  void start() throws IOException {}

  @Override
  void finish() throws IOException {
    output();
    super.finish();
  }

  @Override

View on GitHub (pinned to 2add963021)

Solutions

1. Raise -step until maxSize/step < 2147483647 (e.g. keep step >= 128 for the default 128 GiB range)
2. Lower -maxSize if only small files matter (e.g. -maxSize 1073741824 with -step 1 gives 2^30 intervals, which fits)
3. Omit both options to accept the 128 GiB / 2 MiB defaults (65536 intervals)
4. If embedding the visitor in Java, pre-compute the interval count and reject bad pairs with a clear message before constructing the visitor

Example fix

# before
hdfs oiv_legacy -p FileDistribution -i fsimage_0000000000000060000 -o dist.txt -maxSize 137438953472 -step 64
# IOException: Too many distribution intervals 2147483648

# after
hdfs oiv_legacy -p FileDistribution -i fsimage_0000000000000060000 -o dist.txt -maxSize 137438953472 -step 128

long maxSize = Long.parseLong(cmd.getOptionValue("maxSize", "0"));
int step = Integer.parseInt(cmd.getOptionValue("step", "0"));
if (maxSize == 0) maxSize = 0x2000000000L; // 128 GiB default
if (step == 0) step = 0x200000;            // 2 MiB default
if (maxSize / step >= Integer.MAX_VALUE) {
  throw new IllegalArgumentException(
      "maxSize/step = " + (maxSize / step) + " must be < " + Integer.MAX_VALUE
      + " - increase -step or decrease -maxSize");
}
new FileDistributionVisitor(outputFile, maxSize, step, formatOutput);

When it happens

Trigger: Running `hdfs oiv_legacy -p FileDistribution -i <fsimage> -o <out> -maxSize <bytes> -step <bytes>` where maxSize/step >= 2147483647. Typical case: leaving -maxSize at the 137438953472-byte default while passing -step 64 (2^37/2^6 = 2^31), or -step 1 with any multi-GB -maxSize. Also fires when constructing new FileDistributionVisitor(filename, maxSize, step, formatOutput) directly with such values. Note -step is parsed with Integer.parseInt (OfflineImageViewer.java:253), so it must itself fit an int.

Common situations: Operators lowering -step from megabytes to bytes for a fine-grained histogram without shrinking -maxSize; scripts that feed raw byte counts into both options; commands copied from examples tuned for a much smaller range.

Related errors

• Unrecognized FSImage
• Unrecognized FSImage
• Unrecognized section {s.getName()}
• Cannot process fslayout version {imageVersion}
• Image compression codec not supported: {codecClassName}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/8895f932583a918b. Report an issue: GitHub.