{"record":{"id":"f6403667ff031b7a","repo":"apache/hadoop","slug":"expected-len-found-total","errorCode":null,"errorMessage":"Expected {len} found {total}","messagePattern":"Expected (.+?) found (.+?)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockResolver.java","lineNumber":62,"sourceCode":"        .setGenStamp(genstamp);\n    return b.build();\n  }\n\n  /**\n   * @param s the external reference.\n   * @return sequence of blocks that make up the reference.\n   */\n  public Iterable<BlockProto> resolve(FileStatus s) {\n    List<Long> lengths = blockLengths(s);\n    ArrayList<BlockProto> ret = new ArrayList<>(lengths.size());\n    long tot = 0;\n    for (long l : lengths) {\n      tot += l;\n      ret.add(buildBlock(nextId(), l));\n    }\n    if (tot != s.getLen()) {\n      // log a warning?\n      throw new IllegalStateException(\n          \"Expected \" + s.getLen() + \" found \" + tot);\n    }\n    return ret;\n  }\n\n  /**\n   * @return the next block id.\n   */\n  public abstract long nextId();\n\n  /**\n   * @return the maximum sequentially allocated block ID for this filesystem.\n   */\n  protected abstract long lastId();\n\n  /**\n   * @param status the external reference.\n   * @return the lengths of the resultant blocks.","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-fs2img/src/main/java/org/apache/hadoop/hdfs/server/namenode/BlockResolver.java#L44-L80","documentation":"fs2img's BlockResolver.resolve() splits a file into synthetic blocks using blockLengths(s) and asserts the parts sum exactly to s.getLen(); a mismatch throws IllegalStateException with both numbers. The invariant matters because a generated fsimage whose blocks do not cover the exact file length would be corrupt. Either the file changed size between stat capture and block layout, or the resolver's partitioning is wrong.","triggerScenarios":"(a) The source tree is being written while 'hadoop fs2img' runs: a file is appended or truncated between the FileStatus being captured by the walk and blockLengths() being computed. (b) A custom BlockResolver whose blockLengths() does not cover the exact length, e.g. integer division by block size that drops the final partial block.","commonSituations":"Imaging a live directory such as log output still being appended; a custom resolver for a provisioned image generator that ignores the remainder block; races with compaction jobs rewriting files in place.","solutions":["Run fs2img against a quiesced tree: stop writers, or copy/snapshot the tree first and image the stable copy.","For a custom BlockResolver, emit blocks whose lengths sum exactly to s.getLen() - always include the final partial block.","Re-run after the tree is stable; appends racing the walk produce this transiently."],"exampleFix":"// before: drops the final partial block\nlong full = s.getLen() / BLOCK_SIZE;\nList<Long> ls = new ArrayList<>();\nfor (int i = 0; i < full; i++) { ls.add(BLOCK_SIZE); }\n\n// after: include the remainder so lengths sum to s.getLen()\nlong remaining = s.getLen();\nList<Long> ls = new ArrayList<>();\nwhile (remaining > 0) {\n  long l = Math.min(BLOCK_SIZE, remaining);\n  ls.add(l);\n  remaining -= l;\n}","handlingStrategy":"try-catch","validationCode":"// For custom resolvers: unit-test that blockLengths() sums to getLen() for sizes incl. non-multiples\nlong sum = 0;\nfor (Long l : resolver.blockLengths(status)) { sum += l; }\nassert sum == status.getLen() : \"resolver drops \" + (status.getLen() - sum) + \" bytes\";","typeGuard":null,"tryCatchPattern":"try {\n  for (TreePath e : new FSTreeWalk(root, conf)) { writer.accept(e); }\n} catch (IllegalStateException e) {\n  if (e.getMessage().startsWith(\"Expected \")) {\n    // file changed under the walk: quiesce writers and rerun fs2img on a stable tree\n  }\n}","preventionTips":["Freeze or copy the source tree before imaging; never image a directory with active writers.","For custom BlockResolver implementations, always account for the final partial block."],"tags":["fs2img","block-resolver","fsimage","length-mismatch","file-mutation-race"],"backgroundTag":"file-length-mismatch","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}