What causes "Only read ${actualNumSnapshots} entries out of ${expectedNumSnapshots}" in apache/hadoop?

Read the chained IOException cause first - it names the actual tag mismatch or stream error that stopped the loop

apache/hadoop · error · IOException

Only read ${actualNumSnapshots} <snapshot> entries out of ${

Error message

Only read ${actualNumSnapshots} <snapshot> entries out of ${expectedNumSnapshots}

What it means

Thrown by Hadoop's ReverseXML processor (OfflineImageReconstructor, invoked via `hdfs oiv -processor ReverseXML`) while rebuilding a binary fsimage from an XML dump. The <SnapshotSection> header declares <snapshotCount>N</snapshotCount>; the reconstructor loops N times calling expectTag("snapshot"), and when that read fails (wrong tag, premature end event, XMLStreamException) the failure is re-wrapped with this message. The real reason is always in the chained cause exception.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageReconstructor.java:1372

      while (true) {
        Node sd = header.removeChild(SNAPSHOT_SECTION_SNAPSHOT_TABLE_DIR);
        if (sd == null) {
          break;
        }
        Long dir;
        while ((dir = sd.removeChildLong(SNAPSHOT_SECTION_DIR)) != null) {
          // Add all snapshottable directories, one by one
          bld.addSnapshottableDir(dir);
        }
      }
      header.verifyNoRemainingKeys("SnapshotSection");
      bld.build().writeDelimitedTo(out);
      int actualNumSnapshots = 0;
      while (actualNumSnapshots < expectedNumSnapshots) {
        try {
          expectTag(SNAPSHOT_SECTION_SNAPSHOT, false);
        } catch (IOException e) {
          throw new IOException("Only read " + actualNumSnapshots +
              " <snapshot> entries out of " + expectedNumSnapshots, e);
        }
        actualNumSnapshots++;
        Node snapshot = new Node();
        loadNodeChildren(snapshot, "snapshot fields");
        FsImageProto.SnapshotSection.Snapshot.Builder s =
            FsImageProto.SnapshotSection.Snapshot.newBuilder();
        Integer snapshotId = snapshot.removeChildInt(SECTION_ID);
        if (snapshotId == null) {
          throw new IOException("<snapshot> section was missing <id>");
        }
        s.setSnapshotId(snapshotId);
        Node snapshotRoot = snapshot.removeChild(SNAPSHOT_SECTION_ROOT);
        INodeSection.INode.Builder inodeBld = processINodeXml(snapshotRoot);
        s.setRoot(inodeBld);
        s.build().writeDelimitedTo(out);
      }
      expectTagEnd(SNAPSHOT_SECTION_NAME);

View on GitHub (pinned to 2add963021)

Solutions

Read the chained IOException cause first - it names the actual tag mismatch or stream error that stopped the loop
If the XML was hand-edited, make <snapshotCount> equal the real number of <snapshot> elements (or restore the missing entries)
Regenerate the XML from the original binary fsimage with a same-version oiv (`hdfs oiv -processor XML`) instead of editing it
Stream-count <snapshot> elements vs <snapshotCount> and run xmllint --noout before ReverseXML

Example fix

<!-- before: header promises 5 snapshots, only 3 exist -->
<SnapshotSection>
  <snapshotCount>5</snapshotCount>
  <snapshottableDir>16386</snapshottableDir>
  <snapshot><id>101</id>...</snapshot>
  <snapshot><id>102</id>...</snapshot>
  <snapshot><id>103</id>...</snapshot>
</SnapshotSection>
<!-- after: count matches reality -->
<SnapshotSection>
  <snapshotCount>3</snapshotCount>
  ...

Defensive patterns

Strategy: validation

Validate before calling

# python: verify snapshotCount matches <snapshot> entries before ReverseXML
import xml.etree.ElementTree as ET

def snapshot_counts_ok(path):
    declared, actual = None, 0
    for ev, el in ET.iterparse(path, events=('end',)):
        if el.tag == 'snapshotCount' and declared is None:
            declared = int(el.text)
        elif el.tag == 'snapshot' and declared is not None:
            actual += 1
    return declared is not None and declared == actual

Try / catch

# wrap the CLI; never reuse the partial output file after a failure
import subprocess
rc = subprocess.call(['hdfs','oiv','-processor','ReverseXML','-i','dump.xml','-o','out.img'])
if rc != 0:
    # stderr carries 'Only read N <snapshot> entries...' plus the cause chain
    raise RuntimeError('fsimage reconstruction failed; inspect stderr cause chain')

Prevention

Never hand-edit snapshotCount without recounting <snapshot> elements
Regenerate XML with the same oiv version that will reconstruct it
Run xmllint --noout plus a streaming count check before ReverseXML
Treat the chained cause, not the wrapper message, as the root cause

When it happens

Trigger: Running `hdfs oiv -processor ReverseXML -i dump.xml -o fsimage.out` where <snapshotCount> exceeds the number of <snapshot> elements actually present, or where a <snapshot> entry is malformed so expectTag hits an unexpected event mid-loop.

Common situations: Disaster-recovery workflows that hand-edit fsimage XML then reconstitute it; test harnesses synthesizing <SnapshotSection> blocks; XML dumps produced by an oiv from a different Hadoop release; files truncated during transfer or by editors on huge images.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/be54ddb8d1b66e53. Report an issue: GitHub.