What causes "Incorrect state to start a new value: {state}" in apache/hadoop?

Close the key stream before calling prepareAppendValue(), keeping each stream in its own try-with-resources block in the right order

apache/hadoop · error · IllegalStateException

Incorrect state to start a new value: {state}

Error message

Incorrect state to start a new value: {state}

What it means

Thrown by TFile.Writer.prepareAppendValue(int) when the writer's state is not END_KEY, i.e. when the key stream from prepareAppendKey() has not been closed yet, or a previous value stream is still open. The state machine requires key-close before value-start (READY -> IN_KEY -> END_KEY -> IN_VALUE -> READY), and this check enforces it with IllegalStateException plus the current state name.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:569

    /**
     * Obtain an output stream for writing a value into TFile. This may only be
     * called right after a key appending operation (the key append stream must
     * be closed).
     * 
     * @param length
     *          The expected length of the value. If length of the value is not
     *          known, set length = -1. Otherwise, the application must write
     *          exactly as many bytes as specified here before calling close on
     *          the returned output stream. Advertising the value size up-front
     *          guarantees that the value is encoded in one chunk, and avoids
     *          intermediate chunk buffering.
     * @throws IOException raised on errors performing I/O.
     * @return DataOutputStream.
     */
    public DataOutputStream prepareAppendValue(int length) throws IOException {
      if (state != State.END_KEY) {
        throw new IllegalStateException(
            "Incorrect state to start a new value: " + state.name());
      }

      DataOutputStream ret;

      // unknown length
      if (length < 0) {
        if (valueBuffer == null) {
          valueBuffer = new byte[getChunkBufferSize(conf)];
        }
        ret = new ValueRegister(new ChunkEncoder(blkAppender, valueBuffer));
      } else {
        ret =
            new ValueRegister(new Chunk.SingleChunkEncoder(blkAppender, length));
      }

      state = State.IN_VALUE;
      return ret;

View on GitHub (pinned to 2add963021)

Solutions

Close the key stream before calling prepareAppendValue(), keeping each stream in its own try-with-resources block in the right order
Use writer.append(key, value) or writer.append(key, koff, klen, value, voff, vlen) to avoid manual state handling entirely
After any exception mid-record, close outstanding streams in finally, then close the writer; never resume appending

Example fix

// before
DataOutputStream k = writer.prepareAppendKey(-1);
k.write(key);
DataOutputStream v = writer.prepareAppendValue(-1); // IllegalStateException: IN_KEY

// after
try (DataOutputStream k = writer.prepareAppendKey(-1)) {
  k.write(key);
}
try (DataOutputStream v = writer.prepareAppendValue(-1)) {
  v.write(value);
}

Defensive patterns

Strategy: validation

Validate before calling

// Always finish the key stream before starting the value stream
try (DataOutputStream k = writer.prepareAppendKey(keyLen)) {
  k.write(keyBuf, 0, keyLen);
} // state now END_KEY
try (DataOutputStream v = writer.prepareAppendValue(valLen)) {
  v.write(valBuf, 0, valLen);
} // state now READY

Try / catch

catch (IllegalStateException e) {
  // state in the message (IN_KEY / IN_VALUE / READY) tells you which stream was open;
  // unwind: close inner streams, then writer.close()
}

Prevention

Wrap each append stream in its own try-with-resources block in strict key-then-value order
Use the append() convenience overloads unless chunked value writing is required
Never call prepareAppendValue() twice for one record

When it happens

Trigger: Calling prepareAppendValue() immediately after prepareAppendKey() without closing the key stream; calling it twice for one record; calling it on a fresh writer where no key was appended.

Common situations: Writing key and value with nested streams and missing the inner key-stream close; copy-paste of a key-writing block that drops the close; early exits between key write and value start.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e50c85be953a27c4. Report an issue: GitHub.