apache/hadoop · error · IOException

Incorrect key length: expected={expectedLength} actual={len}

Error message

Incorrect key length: expected={expectedLength} actual={len}

What it means

Thrown when the stream returned by TFile.Writer.prepareAppendKey(expectedLength) is closed but the number of bytes actually written differs from expectedLength (which must be -1 for unknown length). The KeyRegister verifies the count on close and raises IOException, after which the writer is in an inconsistent state and only close() is legitimate. This is a contract check: pre-declaring the key size lets TFile write the length prefix without buffering.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:459

        }
        expectedLength = len;
      }

      @Override
      public void close() throws IOException {
        if (closed == true) {
          return;
        }

        try {
          ++errorCount;
          byte[] key = currentKeyBufferOS.getBuffer();
          int len = currentKeyBufferOS.size();
          /**
           * verify length.
           */
          if (expectedLength >= 0 && expectedLength != len) {
            throw new IOException("Incorrect key length: expected="
                + expectedLength + " actual=" + len);
          }

          Utils.writeVInt(blkAppender, len);
          blkAppender.write(key, 0, len);
          if (tfileIndex.getFirstKey() == null) {
            tfileIndex.setFirstKey(key, 0, len);
          }

          if (tfileMeta.isSorted() && tfileMeta.getRecordCount()>0) {
            byte[] lastKey = lastKeyBufferOS.getBuffer();
            int lastLen = lastKeyBufferOS.size();
            if (tfileMeta.getComparator().compare(key, 0, len, lastKey, 0,
                lastLen) < 0) {
              throw new IOException("Keys are not added in sorted order");
            }
          }

View on GitHub (pinned to 2add963021)

Solutions

1. Pass -1 to prepareAppendKey when the key length is not known exactly
2. When pre-declaring a length, write exactly that many bytes (usually by serializing into a ByteArrayOutputStream first and using its size())
3. After this IOException, only call writer.close(); do not attempt further appends on the corrupted writer

Example fix

// before
DataOutputStream kout = writer.prepareAppendKey(8);
kout.write(keyBytes); // keyBytes.length != 8
kout.close();

// after
DataOutputStream kout = writer.prepareAppendKey(keyBytes.length);
kout.write(keyBytes, 0, keyBytes.length);
kout.close();

// Serialize first so the declared length always matches the bytes
byte[] keyBytes = serializeKey(record);
try (DataOutputStream kout = writer.prepareAppendKey(keyBytes.length)) {
  kout.write(keyBytes);
}
// Or, when length is unknown up front:
try (DataOutputStream kout = writer.prepareAppendKey(-1)) {
  kout.write(keyBytes);
}

catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Incorrect key length")) {
    // writer is now inconsistent: stop appending and close the writer
  }
}

When it happens

Trigger: Calling prepareAppendKey(n) with n >= 0 and writing fewer or more than n bytes to the returned DataOutputStream before closing it; serializing an object whose advertised size does not match the bytes emitted (e.g. Writable.getLength() inconsistent with write()).

Common situations: Copying an expected-length value from another record type; changing key serialization without updating the declared length; mixing up key and value lengths; defensive code that guesses a length instead of passing -1.

Related errors

• Cannot close TFile in the middle of key-value insertion.
• Keys are not added in sorted order
• Incorrect state to start a new key: {state}
• Incorrect state to start a new value: {state}
• Incorrect state to start a Meta Block: {state}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c8d68d0a5c248a11. Report an issue: GitHub.