apache/hadoop · error · IllegalStateException

Incorrect state to start a new key: {state}

Error message

Incorrect state to start a new key: {state}

What it means

Thrown by TFile.Writer.prepareAppendKey(int) when the writer's state is anything other than READY: either a previous key/value append stream is still open, or the writer is closed/otherwise mid-transition. TFile.Writer is a strict state machine (READY -> IN_KEY -> END_KEY -> IN_VALUE -> READY), and each step's preparatory call validates the current state, surfacing violations as IllegalStateException with the offending state name in the message.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:542

    }

    /**
     * Obtain an output stream for writing a key into TFile. This may only be
     * called when there is no active Key appending stream or value appending
     * stream.
     * 
     * @param length
     *          The expected length of the key. If length of the key is not
     *          known, set length = -1. Otherwise, the application must write
     *          exactly as many bytes as specified here before calling close on
     *          the returned output stream.
     * @return The key appending output stream.
     * @throws IOException raised on errors performing I/O.
     * 
     */
    public DataOutputStream prepareAppendKey(int length) throws IOException {
      if (state != State.READY) {
        throw new IllegalStateException("Incorrect state to start a new key: "
            + state.name());
      }

      initDataBlock();
      DataOutputStream ret = new KeyRegister(length);
      state = State.IN_KEY;
      return ret;
    }

    /**
     * Obtain an output stream for writing a value into TFile. This may only be
     * called right after a key appending operation (the key append stream must
     * be closed).
     * 
     * @param length
     *          The expected length of the value. If length of the value is not
     *          known, set length = -1. Otherwise, the application must write
     *          exactly as many bytes as specified here before calling close on

View on GitHub (pinned to 2add963021)

Solutions

1. Always close the stream returned by prepareAppendKey/prepareAppendValue before starting the next record, using try-with-resources
2. Prefer writer.append(key, value), which performs the whole READY->READY cycle internally
3. Treat any IllegalStateException here as a logic bug in the calling loop, not as a retryable condition

Example fix

// before
DataOutputStream k = writer.prepareAppendKey(-1);
k.write(key1);
DataOutputStream k2 = writer.prepareAppendKey(-1); // IllegalStateException: IN_KEY

// after
try (DataOutputStream k = writer.prepareAppendKey(-1)) {
  k.write(key1);
}
DataOutputStream k2 = writer.prepareAppendKey(-1); // OK: back to READY

// Enforce one complete record per iteration; state returns to READY each time
while (records.hasNext()) {
  Record r = records.next();
  try (DataOutputStream k = writer.prepareAppendKey(r.keyLength())) {
    k.write(r.keyBytes());
  }
  try (DataOutputStream v = writer.prepareAppendValue(r.valueLength())) {
    v.write(r.valueBytes());
  }
}

catch (IllegalStateException e) {
  // message ends with the writer state, e.g. 'IN_KEY': an inner stream was left open.
  // close outstanding streams, then close the writer; do not continue appending.
}

When it happens

Trigger: Calling prepareAppendKey() twice without closing the first stream; calling it before closing the value stream of the previous record; calling it after an append stream failed mid-write and was never closed.

Common situations: Loop bodies that reuse a variable for the key stream without closing the previous iteration's stream; exception paths that skip the inner stream close; refactoring a single append into staged prepare calls and forgetting one close.

Related errors

• Cannot close TFile in the middle of key-value insertion.
• Incorrect state to start a new value: {state}
• Incorrect state to start a Meta Block: {state}
• Incorrect key length: expected={expectedLength} actual={len}
• Keys are not added in sorted order

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c52af6f38ce917a4. Report an issue: GitHub.