apache/hadoop · error · IOException

Unrecognized FSImage

Error message

Unrecognized FSImage

What it means

FileDistributionCalculator.visit() (the FileDistribution processor of the offline image viewer) runs the same FSImageUtil.checkFileFormat() check as FSImageLoader: the input must be at least MINIMUM_FILE_LENGTH bytes and begin with the magic HDFSIMG1, otherwise IOException("Unrecognized FSImage") aborts the size-distribution histogram run before any section is read.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java:98

  FileDistributionCalculator(Configuration conf, long maxSize, int steps,
      boolean formatOutput, PrintStream out) {
    this.conf = conf;
    this.maxSize = maxSize == 0 ? MAX_SIZE_DEFAULT : maxSize;
    this.steps = steps == 0 ? INTERVAL_DEFAULT : steps;
    this.formatOutput = formatOutput;
    this.out = out;
    long numIntervals = this.maxSize / this.steps;
    // avoid OutOfMemoryError when allocating an array
    Preconditions.checkState(numIntervals <= MAX_INTERVALS,
        "Too many distribution intervals (maxSize/step): " + numIntervals +
        ", should be less than " + (MAX_INTERVALS+1) + ".");
    this.distribution = new int[1 + (int) (numIntervals)];
  }

  void visit(RandomAccessFile file) throws IOException {
    if (!FSImageUtil.checkFileFormat(file)) {
      throw new IOException("Unrecognized FSImage");
    }

    FileSummary summary = FSImageUtil.loadSummary(file);
    try (FileInputStream in = new FileInputStream(file.getFD())) {
      for (FileSummary.Section s : summary.getSectionsList()) {
        if (SectionName.fromString(s.getName()) != SectionName.INODE) {
          continue;
        }

        in.getChannel().position(s.getOffset());
        InputStream is = FSImageUtil.wrapInputStreamForCompression(conf,
            summary.getCodec(), new BufferedInputStream(new LimitInputStream(
                in, s.getLength())));
        run(is);
        output();
      }
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

1. Pass the real protobuf fsimage file: hdfs oiv -p FileDistribution -i fsimage_0000000000000123456 -o dist.txt maxSize step.
2. Confirm the header with head -c 8 <file> — it must print HDFSIMG1.
3. For 1.x-format images use hdfs oiv_legacy or regenerate the image from a 2.x+ NameNode.

Example fix

# before
hdfs oiv -p FileDistribution -i fsimage_0000000000000098765.md5 -o dist.txt 0 0
# -> Unrecognized FSImage

# after
hdfs oiv -p FileDistribution -i fsimage_0000000000000098765 -o dist.txt 0 0

import java.nio.file.*;

static boolean isProtoBufFsImage(Path p) throws IOException {
  if (Files.size(p) < 64) return false;
  try (InputStream in = Files.newInputStream(p)) {
    byte[] magic = new byte[8];
    if (in.read(magic) != 8) return false;
    return new String(magic, StandardCharsets.UTF_8).equals("HDFSIMG1");
  }
}

try {
  new FileDistributionCalculator(conf, maxSize, step, format, out).visit(file);
} catch (IOException e) {
  if ("Unrecognized FSImage".equals(e.getMessage())) {
    // wrong input file (.md5, edits, legacy or truncated image)
    throw new IOException("Not a protobuf fsimage: " + file, e);
  }
  throw e;
}

When it happens

Trigger: hdfs oiv -p FileDistribution -i <wrong file>: an fsimage.md5, an edit log, a pre-protobuf (Hadoop 1.x) fsimage, or a truncated/corrupt image copy.

Common situations: Storage-dir globs catching .md5 siblings; images truncated in transfer; running the modern FileDistribution processor on legacy-format images.

Related errors

• Unrecognized FSImage
• Unrecognized section {s.getName()}
• Too many distribution intervals {numIntervals}
• Unrecognized FSImage
• Unrecognized FSImage

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f9bb7bd0129ff462. Report an issue: GitHub.