{"record":{"id":"f9bb7bd0129ff462","repo":"apache/hadoop","slug":"unrecognized-fsimage-f9bb7b","errorCode":null,"errorMessage":"Unrecognized FSImage","messagePattern":"Unrecognized FSImage","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java","lineNumber":98,"sourceCode":"\n  FileDistributionCalculator(Configuration conf, long maxSize, int steps,\n      boolean formatOutput, PrintStream out) {\n    this.conf = conf;\n    this.maxSize = maxSize == 0 ? MAX_SIZE_DEFAULT : maxSize;\n    this.steps = steps == 0 ? INTERVAL_DEFAULT : steps;\n    this.formatOutput = formatOutput;\n    this.out = out;\n    long numIntervals = this.maxSize / this.steps;\n    // avoid OutOfMemoryError when allocating an array\n    Preconditions.checkState(numIntervals <= MAX_INTERVALS,\n        \"Too many distribution intervals (maxSize/step): \" + numIntervals +\n        \", should be less than \" + (MAX_INTERVALS+1) + \".\");\n    this.distribution = new int[1 + (int) (numIntervals)];\n  }\n\n  void visit(RandomAccessFile file) throws IOException {\n    if (!FSImageUtil.checkFileFormat(file)) {\n      throw new IOException(\"Unrecognized FSImage\");\n    }\n\n    FileSummary summary = FSImageUtil.loadSummary(file);\n    try (FileInputStream in = new FileInputStream(file.getFD())) {\n      for (FileSummary.Section s : summary.getSectionsList()) {\n        if (SectionName.fromString(s.getName()) != SectionName.INODE) {\n          continue;\n        }\n\n        in.getChannel().position(s.getOffset());\n        InputStream is = FSImageUtil.wrapInputStreamForCompression(conf,\n            summary.getCodec(), new BufferedInputStream(new LimitInputStream(\n                in, s.getLength())));\n        run(is);\n        output();\n      }\n    }\n  }","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java#L80-L116","documentation":"FileDistributionCalculator.visit() (the FileDistribution processor of the offline image viewer) runs the same FSImageUtil.checkFileFormat() check as FSImageLoader: the input must be at least MINIMUM_FILE_LENGTH bytes and begin with the magic HDFSIMG1, otherwise IOException(\"Unrecognized FSImage\") aborts the size-distribution histogram run before any section is read.","triggerScenarios":"hdfs oiv -p FileDistribution -i <wrong file>: an fsimage.md5, an edit log, a pre-protobuf (Hadoop 1.x) fsimage, or a truncated/corrupt image copy.","commonSituations":"Storage-dir globs catching .md5 siblings; images truncated in transfer; running the modern FileDistribution processor on legacy-format images.","solutions":["Pass the real protobuf fsimage file: hdfs oiv -p FileDistribution -i fsimage_0000000000000123456 -o dist.txt maxSize step.","Confirm the header with head -c 8 <file> — it must print HDFSIMG1.","For 1.x-format images use hdfs oiv_legacy or regenerate the image from a 2.x+ NameNode."],"exampleFix":"# before\nhdfs oiv -p FileDistribution -i fsimage_0000000000000098765.md5 -o dist.txt 0 0\n# -> Unrecognized FSImage\n\n# after\nhdfs oiv -p FileDistribution -i fsimage_0000000000000098765 -o dist.txt 0 0","handlingStrategy":"validation","validationCode":"import java.nio.file.*;\n\nstatic boolean isProtoBufFsImage(Path p) throws IOException {\n  if (Files.size(p) < 64) return false;\n  try (InputStream in = Files.newInputStream(p)) {\n    byte[] magic = new byte[8];\n    if (in.read(magic) != 8) return false;\n    return new String(magic, StandardCharsets.UTF_8).equals(\"HDFSIMG1\");\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  new FileDistributionCalculator(conf, maxSize, step, format, out).visit(file);\n} catch (IOException e) {\n  if (\"Unrecognized FSImage\".equals(e.getMessage())) {\n    // wrong input file (.md5, edits, legacy or truncated image)\n    throw new IOException(\"Not a protobuf fsimage: \" + file, e);\n  }\n  throw e;\n}","preventionTips":["Verify the HDFSIMG1 magic before running FileDistribution.","Select the fsimage_* file explicitly, never a glob that catches .md5 siblings.","Compare transfer sizes/checksums for images copied between hosts."],"tags":["hdfs","offline-image-viewer","oiv","file-distribution","fsimage","file-format"],"backgroundTag":"unsupported-file-format","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}