apache/hadoop · error · IOException

Unrecognized FSImage

Error message

Unrecognized FSImage

What it means

Before parsing, PBImageTextWriter.visit() (the Delimited/DetectCorruption processors of `hdfs oiv`) verifies via FSImageUtil.checkFileFormat that the file starts with the fsimage magic 'HDFSIMG1' and meets the minimum length. The check failed, so the input is not a protobuf fsimage: it may be an edit log, the .md5 sidecar, a legacy pre-2.x image, a truncated download, or an unrelated file.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageTextWriter.java:597

  /**
   * Get text output for the header line.
   */
  abstract protected String getHeader();

  /**
   * Method called at the end of output() phase after all the inodes
   * with known parentPath has been printed out. Can be used to print
   * additional data depending on the written inodes.
   */
  abstract protected void afterOutput() throws IOException;

  public void visit(String filePath) throws IOException {
    filename = new File(filePath);
    RandomAccessFile file = new RandomAccessFile(filePath, "r");
    Configuration conf = new Configuration();
    if (!FSImageUtil.checkFileFormat(file)) {
      throw new IOException("Unrecognized FSImage");
    }

    FileSummary summary = FSImageUtil.loadSummary(file);

    try (FileInputStream fin = new FileInputStream(file.getFD())) {
      InputStream is;
      ArrayList<FileSummary.Section> sections =
          Lists.newArrayList(summary.getSectionsList());
      Collections.sort(sections,
          new Comparator<FileSummary.Section>() {
            @Override
            public int compare(FsImageProto.FileSummary.Section s1,
                FsImageProto.FileSummary.Section s2) {
              FSImageFormatProtobuf.SectionName n1 =
                  FSImageFormatProtobuf.SectionName.fromString(s1.getName());
              FSImageFormatProtobuf.SectionName n2 =
                  FSImageFormatProtobuf.SectionName.fromString(s2.getName());
              if (n1 == null) {

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm the target is a protobuf fsimage: `head -c 8 <file>` must print HDFSIMG1
2. Pick the file explicitly by name (fsimage_<txid>, largest txid = newest) instead of globbing
3. Verify the transfer with the sibling fsimage.md5 (`md5sum -c`); re-download on mismatch
4. For pre-Hadoop-2 images use `hdfs oiv_legacy`

Example fix

# before: wrong file from a glob
hdfs oiv -p delimited -i /dfs/name/current/edits_* -o out.csv
# IOException: Unrecognized FSImage

# after: select and verify the real image first
head -c 8 /dfs/name/current/fsimage_0000000000000123456   # HDFSIMG1
hdfs oiv -p delimited -i /dfs/name/current/fsimage_0000000000000123456 -o out.csv

// Reject non-fsimage input before calling visit()
try (RandomAccessFile f = new RandomAccessFile(path, "r")) {
  byte[] magic = new byte[FSImageUtil.MAGIC_HEADER.length];
  f.readFully(magic);
  if (!Arrays.equals(magic, FSImageUtil.MAGIC_HEADER)) {
    throw new IllegalArgumentException(
        path + " is not a protobuf fsimage (missing HDFSIMG1 magic)");
  }
}

When it happens

Trigger: Passing edits_inprogress_*, fsimage.md5, a legacy-format image, or a partially downloaded fsimage to `hdfs oiv -i`; globbing /dfs/name/current/* and feeding the wrong match.

Common situations: Shell glob picking an edits file instead of the image; an interrupted download leaving a short file; an HTML error page saved as the 'image'; legacy image run through the PB tool.

Related errors

• Unrecognized FSImage
• Unrecognized FSImage
• Unrecognized FSImage
• Unrecognized section {s.getName()}
• Too many distribution intervals {numIntervals}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6f2d75034c4b474f. Report an issue: GitHub.