apache/hadoop · error · RuntimeException

Value length unknown.

Error message

Value length unknown.

What it means

RuntimeException from Scanner.Entry.getValueLength() when vlen < 0. TFile stores large values in compressed chunks; when the total value length was not recorded at write time it is unknown until the whole value has been consumed. The Javadoc contract is explicit: isValueLengthKnown() must be tested true before calling getValueLength().

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:1835

         * @return The input stream.
         */
        public DataInputStream getKeyStream() {
          keyDataInputStream.reset(keyBuffer, klen);
          return keyDataInputStream;
        }

        /**
         * Get the length of the value. isValueLengthKnown() must be tested
         * true.
         * 
         * @return the length of the value.
         */
        public int getValueLength() {
          if (vlen >= 0) {
            return vlen;
          }

          throw new RuntimeException("Value length unknown.");
        }

        /**
         * Copy value into user-supplied buffer. User supplied buffer must be
         * large enough to hold the whole value. The value part of the key-value
         * pair pointed by the current cursor is not cached and can only be
         * examined once. Calling any of the following functions more than once
         * without moving the cursor will result in exception:
         * {@link #getValue(byte[])}, {@link #getValue(byte[], int)},
         * {@link #getValueStream}.
         *
         * @param buf buf.
         * @return the length of the value. Does not require
         *         isValueLengthKnown() to be true.
         * @throws IOException raised on errors performing I/O.
         * 
         */
        public int getValue(byte[] buf) throws IOException {

View on GitHub (pinned to 2add963021)

Solutions

1. Branch on entry.isValueLengthKnown(); only call getValueLength() in the true branch.
2. When unknown, get the actual length from the return value of entry.getValue(buf) (it returns the number of bytes read) or stream via getValueStream().
3. If you control the writer and always need lengths, write values with the whole-value append (Writer.append(key, value)) so lengths are advertised.

Example fix

// before
int vlen = entry.getValueLength(); // RuntimeException on chunked values
// after
if (entry.isValueLengthKnown()) {
  int vlen = entry.getValueLength();
} else {
  int vlen = entry.getValue(buf, 0); // returns actual bytes read
}

int vlen;
if (entry.isValueLengthKnown()) {
  vlen = entry.getValueLength();
} else {
  vlen = -1; // get actual size from getValue(buf) return or streaming
}

When it happens

Trigger: Calling entry.getValueLength() on an entry whose value was chunked, i.e. where checkKey() found valueBufferInputStream.isLastChunk() false and left vlen = -1. Values larger than the chunk threshold or written through the streaming append API hit this.

Common situations: Mixed-size payloads: small values work (length known) so code passes testing, then the first multi-chunk value (typically > 64KB before compression) blows up in production. Common in map-side join files and pre-HFile TFile users.

Related errors

• Cannot search in unsorted TFile
• Key length out of range: {klen}
• Buffer not enough to store the key
• Buffer too small to hold value
• Attempt to examine value multiple times.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e01b6e72da29701a. Report an issue: GitHub.