apache/hadoop · error · IllegalStateException

Cannot close TFile in the middle of key-value insertion.

Error message

Cannot close TFile in the middle of key-value insertion.

What it means

Thrown by TFile.Writer.close() when closing a writer whose state is not READY, meaning a key or value append stream obtained from prepareAppendKey()/prepareAppendValue() was never closed. TFile is a strict state machine; close() only accepts the writer in the READY state (no record in progress). The IllegalStateException fires only when errorCount == 0, i.e. when the previous streams appeared to complete normally but were left open.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/file/tfile/TFile.java:323

    }

    /**
     * Close the Writer. Resources will be released regardless of the exceptions
     * being thrown. Future close calls will have no effect.
     * 
     * The underlying FSDataOutputStream is not closed.
     */
    @Override
    public void close() throws IOException {
      if ((state == State.CLOSED)) {
        return;
      }
      try {
        // First try the normal finish.
        // Terminate upon the first Exception.
        if (errorCount == 0) {
          if (state != State.READY) {
            throw new IllegalStateException(
                "Cannot close TFile in the middle of key-value insertion.");
          }

          finishDataBlock(true);

          // first, write out data:TFile.meta
          BlockAppender outMeta =
              writerBCF
                  .prepareMetaBlock(TFileMeta.BLOCK_NAME, COMPRESSION_NONE);
          try {
            tfileMeta.write(outMeta);
          } finally {
            outMeta.close();
          }

          // second, write out data:TFile.index
          BlockAppender outIndex =
              writerBCF.prepareMetaBlock(TFileIndex.BLOCK_NAME);

View on GitHub (pinned to 2add963021)

Solutions

Close every stream returned by prepareAppendKey()/prepareAppendValue() before calling writer.close(), ideally with try-with-resources or try/finally
Prefer the single-shot writer.append(key, value) API, which manages the state machine internally
If a record must be abandoned, be aware the writer is already inconsistent: close the inner stream (which may throw) and expect close() to surface the first error rather than the IllegalState one

Example fix

// before
DataOutputStream dos = writer.prepareAppendKey(-1);
dos.write(key);
writer.close(); // IllegalStateException: still in IN_KEY state

// after
try (DataOutputStream dos = writer.prepareAppendKey(-1)) {
  dos.write(key);
}
writer.close();

Defensive patterns

Strategy: validation

Validate before calling

// Ensure no append stream is open before closing the writer
// Pattern: every prepare* stream lives in try-with-resources
try (DataOutputStream k = writer.prepareAppendKey(-1)) {
  k.write(key);
}
try (DataOutputStream v = writer.prepareAppendValue(-1)) {
  v.write(value);
}
writer.close(); // only called when writer is back in READY state

Try / catch

catch (IllegalStateException e) {
  // a stream was left open: close inner streams in finally, then close the writer;
  // data after the abandoned record is not recoverable in this file
}

Prevention

Prefer writer.append(key, value) which cannot leave the state machine mid-record
Never call writer.close() from a finally block while a prepared stream may still be open; nest the stream's close first
Keep one record's stream lifecycle inside a single method so no path skips the close

When it happens

Trigger: Calling writer.close() while a DataOutputStream returned by prepareAppendKey() or prepareAppendValue() is still open; abandoning a key/value stream mid-record and then closing the writer; forgetting the close() inside a loop that breaks early.

Common situations: Exception paths where the key stream is opened but the writer is closed in a finally block without closing the intermediate stream; refactoring that removes a stream close; control-flow bugs (break/continue/return) that skip the inner close before the outer close.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/5a278fbd3938fc45. Report an issue: GitHub.