apache/hadoop · critical · IllegalStateException

Already have a finalized segment {} beginning at {} ; journa

Error message

Already have a finalized segment {} beginning at {} ; journal id: {}

What it means

In Journal.startLogSegment, a paranoid sanity check: before a writer starts a segment at txid, the Journal checks whether an edit log file already exists at that txid. Finding a FINALIZED file there is a corruption/fencing violation — finalized segments must never be overwritten — so it throws IllegalStateException to refuse rather than risk destroying committed transactions.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:591

    
    if (curSegment != null) {
      LOG.warn("Client is requesting a new log segment " + txid + 
          " though we are already writing " + curSegment + ". " +
          "Aborting the current segment in order to begin the new one." +
          " ; journal id: " + journalId);
      // The writer may have lost a connection to us and is now
      // re-connecting after the connection came back.
      // We should abort our own old segment.
      abortCurSegment();
    }

    // Paranoid sanity check: we should never overwrite a finalized log file.
    // Additionally, if it's in-progress, it should have at most 1 transaction.
    // This can happen if the writer crashes exactly at the start of a segment.
    EditLogFile existing = fjm.getLogFile(txid);
    if (existing != null) {
      if (!existing.isInProgress()) {
        throw new IllegalStateException("Already have a finalized segment " +
            existing + " beginning at " + txid + " ; journal id: " + journalId);
      }
      
      // If it's in-progress, it should only contain one transaction,
      // because the "startLogSegment" transaction is written alone at the
      // start of each segment. 
      existing.scanLog(Long.MAX_VALUE, false);
      if (existing.getLastTxId() != existing.getFirstTxId()) {
        throw new IllegalStateException("The log file " +
            existing + " seems to contain valid transactions" +
            " ; journal id: " + journalId);
      }
    }
    
    long curLastWriterEpoch = lastWriterEpoch.get();
    if (curLastWriterEpoch != reqInfo.getEpoch()) {
      LOG.info("Updating lastWriterEpoch from " + curLastWriterEpoch +
          " to " + reqInfo.getEpoch() + " for client " +

View on GitHub (pinned to 2add963021)

Solutions

Immediately ensure exactly one NameNode can write: stop any second active for this nameservice and verify ZKFC fencing is active — concurrent writers are the most common cause.
Preserve the JN dir as evidence (copy it aside) before changing anything; identify which finalized segment is at that txid and compare it across all JNs.
Rebuild consistency: align the divergent JN from a healthy quorum member's journal dir, or use the NN recovery flow ('hdfs namenode -recover' as a last resort) to pick a canonical edit history.
If this reproduces with a single writer, capture the NN/JN logs around the event — startLogSegment hitting an existing finalized file with proper fencing indicates a genuine bug worth a JIRA.

Defensive patterns

Strategy: try-catch

Validate before calling

// Server-side guard before accepting a new segment: refuse early if a
// finalized file already exists at that txid (same check startLogSegment does)
EditLogFile existing = fjm.getLogFile(txid);
if (existing != null && !existing.isInProgress()) {
  throw new IllegalStateException(
      "Refusing startLogSegment at " + txid + ": finalized segment exists: "
      + existing);
}

Type guard

static boolean isFinalizedSegmentConflict(Throwable t) {
  return t instanceof IllegalStateException
      && t.getMessage() != null
      && t.getMessage().startsWith("Already have a finalized segment");
}

Try / catch

try {
  jn.startLogSegment(reqInfo, txid, writerEpoch);
} catch (IllegalStateException ise) {
  if (isFinalizedSegmentConflict(ise)) {
    // corruption / fencing event: preserve evidence, stop the writer,
    // and reconcile JN dirs — never overwrite the finalized segment
    quiesceAndPageAdmin(ise);
  } else {
    throw ise;
  }
}

Prevention

Never force two NameNodes active for one nameservice — always fail over via ZKFC.
Never copy edit segment files between JN directories by hand; use JournalNodeSyncer or documented recovery.
When restoring a JN, restore the whole directory from one healthy peer at one point in time.

When it happens

Trigger: A second writer starts a log segment at the same txid where this JN already holds a finalized segment: fencing hole (two NNs writing the same journal id), a JN restored from an inconsistent copy, manual file copying into the journal dir, or a rollback/recovery sequence that resurrected an old finalized segment at the same start txid.

Common situations: Manually forced active-active during HA troubleshooting; JN directory restored from a mixture of backups; files copied between JN dirs by hand; replaying old segment files into current; bugs where an old writer's segment raced a new writer. Because finalized (committed) data is at risk, treat as a corruption event.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/96edc6847013fc70. Report an issue: GitHub.