What causes "zero length key found!" in apache/hadoop?

Confirm the file is bad by decoding it independently: `hadoop fs -text ` — if that fails too, restore or regenerate the file.

apache/hadoop · critical · IOException

zero length key found!

Error message

zero length key found!

What it means

In the block-compressed branch of SequenceFile.Reader.nextRaw(DataOutputBuffer, ValueBytes), the key length is a VInt read from the decompressed key-length block (keyLenIn). A negative value means that block stream is misaligned or damaged, and the guard throws IOException("zero length key found!"). Note the message is slightly misleading: it fires for any keyLength < 0, which always indicates corruption — a well-formed file never writes negative key lengths.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:2639

        return length;
      } else {
        //Reset syncSeen
        syncSeen = false;
        
        // Read 'key'
        if (noBufferedKeys == 0) {
          if (in.getPos() >= end) 
            return -1;

          try { 
            readBlock();
          } catch (EOFException eof) {
            return -1;
          }
        }
        int keyLength = WritableUtils.readVInt(keyLenIn);
        if (keyLength < 0) {
          throw new IOException("zero length key found!");
        }
        key.write(keyIn, keyLength);
        --noBufferedKeys;
        
        // Read raw 'value'
        seekToCurrentValue();
        int valLength = WritableUtils.readVInt(valLenIn);
        UncompressedBytes rawValue = (UncompressedBytes)val;
        rawValue.reset(valIn, valLength);
        --noBufferedValues;
        
        return (keyLength+valLength);
      }
      
    }

    /**
     * Read 'raw' keys.

View on GitHub (pinned to 2add963021)

Solutions

Confirm the file is bad by decoding it independently: `hadoop fs -text <file>` — if that fails too, restore or regenerate the file.
Make sure the exact codec class used at write time is on the reader's classpath and identically configured.
If only the tail is damaged, read up to the last complete block and stop: compare reader.getPosition() with the file length and treat EOF mid-block as end of data.
Re-run the producing job to completion so no partial blocks are written.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  while (reader.nextRaw(keyBuf, valBytes) > 0) { /* ... */ }
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("zero length key")) {
    LOG.error("Corrupt key block near pos {} in {} — regenerate file",
              reader.getPosition(), path);
    // stop processing this file; mark the input bad and continue with others
  } else {
    throw e;
  }
}

Prevention

Ship the same compression codec jars to writers and readers
Don't kill jobs mid-flush; verify outputs with `hadoop fs -text` after risky transfers
Use checksummed transfers (distcp -crc) when moving SequenceFiles

When it happens

Trigger: Reading a BLOCK-compressed SequenceFile whose key-length block yields a negative VInt: a corrupt or truncated file, a partially flushed final block from a killed writer, or a decompression codec mismatch (missing/different codec jar) that turns the block into garbage lengths.

Common situations: Jobs killed mid-flush leaving partial block tails; LZO/Snappy/Gzip codec jars absent or different versions on the reader classpath; files corrupted in transfer (distcp/checksum mismatches); SequenceFiles read past a truncated HDFS replica.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ff53727254830af9. Report an issue: GitHub.