apache/hadoop · error · IllegalStateException

Expected {len} found {total}

Error message

Expected {len} found {total}

What it means

fs2img's BlockResolver.resolve() splits a file into synthetic blocks using blockLengths(s) and asserts the parts sum exactly to s.getLen(); a mismatch throws IllegalStateException with both numbers. The invariant matters because a generated fsimage whose blocks do not cover the exact file length would be corrupt. Either the file changed size between stat capture and block layout, or the resolver's partitioning is wrong.

Source

Thrown at hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockResolver.java:62

        .setGenStamp(genstamp);
    return b.build();
  }

  /**
   * @param s the external reference.
   * @return sequence of blocks that make up the reference.
   */
  public Iterable<BlockProto> resolve(FileStatus s) {
    List<Long> lengths = blockLengths(s);
    ArrayList<BlockProto> ret = new ArrayList<>(lengths.size());
    long tot = 0;
    for (long l : lengths) {
      tot += l;
      ret.add(buildBlock(nextId(), l));
    }
    if (tot != s.getLen()) {
      // log a warning?
      throw new IllegalStateException(
          "Expected " + s.getLen() + " found " + tot);
    }
    return ret;
  }

  /**
   * @return the next block id.
   */
  public abstract long nextId();

  /**
   * @return the maximum sequentially allocated block ID for this filesystem.
   */
  protected abstract long lastId();

  /**
   * @param status the external reference.
   * @return the lengths of the resultant blocks.

View on GitHub (pinned to 2add963021)

Solutions

1. Run fs2img against a quiesced tree: stop writers, or copy/snapshot the tree first and image the stable copy.
2. For a custom BlockResolver, emit blocks whose lengths sum exactly to s.getLen() - always include the final partial block.
3. Re-run after the tree is stable; appends racing the walk produce this transiently.

Example fix

// before: drops the final partial block
long full = s.getLen() / BLOCK_SIZE;
List<Long> ls = new ArrayList<>();
for (int i = 0; i < full; i++) { ls.add(BLOCK_SIZE); }

// after: include the remainder so lengths sum to s.getLen()
long remaining = s.getLen();
List<Long> ls = new ArrayList<>();
while (remaining > 0) {
  long l = Math.min(BLOCK_SIZE, remaining);
  ls.add(l);
  remaining -= l;
}

// For custom resolvers: unit-test that blockLengths() sums to getLen() for sizes incl. non-multiples
long sum = 0;
for (Long l : resolver.blockLengths(status)) { sum += l; }
assert sum == status.getLen() : "resolver drops " + (status.getLen() - sum) + " bytes";

try {
  for (TreePath e : new FSTreeWalk(root, conf)) { writer.accept(e); }
} catch (IllegalStateException e) {
  if (e.getMessage().startsWith("Expected ")) {
    // file changed under the walk: quiesce writers and rerun fs2img on a stable tree
  }
}

When it happens

Trigger: (a) The source tree is being written while 'hadoop fs2img' runs: a file is appended or truncated between the FileStatus being captured by the walk and blockLengths() being computed. (b) A custom BlockResolver whose blockLengths() does not cover the exact length, e.g. integer division by block size that drops the final partial block.

Common situations: Imaging a live directory such as log output still being appended; a custom resolver for a provisioned image generator that ignores the remainder block; races with compaction jobs rewriting files in place.

Related errors

• FS modified
• Incompatible layout {} (expected {})
• symlinks not supported
• Incorrect key length: expected={expectedLength} actual={len}
• The length of the feature flag section was negative at {} by

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f6403667ff031b7a. Report an issue: GitHub.