apache/hadoop · error · IOException

Length not matched: length=${length}, checksum.getLength()=$

Error message

Length not matched: length=${length}, checksum.getLength()=${checksum.getLength()}

What it means

MD5MD5CRC32FileChecksum has a fixed serialized length, and JsonUtilClient compares that length with the length field from the WebHDFS GETFILECHECKSUM response. This IOException means the advertised length differs from the length of the checksum object decoded from the bytes field, so the response is internally inconsistent.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/JsonUtilClient.java:573

    case CRC32:
      checksum = new MD5MD5CRC32GzipFileChecksum();
      break;
    case CRC32C:
      checksum = new MD5MD5CRC32CastagnoliFileChecksum();
      break;
    default:
      throw new IOException("Unknown algorithm: " + algorithm);
    }
    checksum.readFields(in);

    //check algorithm name
    if (!checksum.getAlgorithmName().equals(algorithm)) {
      throw new IOException("Algorithm not matched. Expected " + algorithm
          + ", Received " + checksum.getAlgorithmName());
    }
    //check length
    if (length != checksum.getLength()) {
      throw new IOException("Length not matched: length=" + length
          + ", checksum.getLength()=" + checksum.getLength());
    }

    return checksum;
  }

  /** Convert a Json map to a AclStatus object. */
  static AclStatus toAclStatus(final Map<?, ?> json) {
    if (json == null) {
      return null;
    }

    final Map<?, ?> m = (Map<?, ?>) json.get(AclStatus.class.getSimpleName());

    AclStatus.Builder aclStatusBuilder = new AclStatus.Builder();
    aclStatusBuilder.owner((String) m.get("owner"));
    aclStatusBuilder.group((String) m.get("group"));
    aclStatusBuilder.stickyBit((Boolean) m.get("stickyBit"));

View on GitHub (pinned to 2add963021)

Solutions

Inspect the raw GETFILECHECKSUM JSON and confirm its length matches the fixed MD5MD5CRC32 checksum size emitted by the same Hadoop version.
Remove or fix intermediaries that rewrite or partially cache the checksum response.
Use a matching Hadoop client against the real NameNode endpoint.
Fall back to computing a checksum by reading the file if the server checksum format cannot be trusted.

Example fix

// before
Map<?, ?> json = WebHdfsFileSystem.jsonParse(connection, false);

// after: validate the response before decoding
Object cs = json.get("FileChecksum");
if (cs == null || ((Map<?, ?>) cs).get("length") == null
    || ((Map<?, ?>) cs).get("bytes") == null) {
  throw new IOException("Incomplete WebHDFS checksum response: " + json);
}
return JsonUtilClient.toMD5MD5CRC32FileChecksum(json);

Defensive patterns

Strategy: try-catch

Try / catch

try {
  return fs.getFileChecksum(path);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Length not matched")) {
    throw new IOException("Invalid WebHDFS checksum length for " + path, e);
  }
  throw e;
}

Prevention

Keep checksum JSON fields length and bytes together when caching or forwarding responses.
Use matching Hadoop versions for checksum verification across a cluster.
Prefer reading the file and hashing it when an intermediary cannot preserve WebHDFS response integrity.

When it happens

Trigger: getFileChecksum through WebHDFS returns a length field that is not the expected MD5MD5CRC32 checksum length while the response is otherwise decoded. A truncated or hand-crafted bytes field, a proxy-modified length, or an incompatible response schema can produce it.

Common situations: A WebHDFS gateway constructs or patches checksum JSON incorrectly; a mock returns arbitrary lengths; an HTTP cache serves a partial or mixed response; client and server versions disagree on checksum serialization.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/2b7a4fd592b6d133. Report an issue: GitHub.