apache/hadoop · error · IllegalStateException

The log file {} seems to contain valid transactions ; journa

Error message

The log file {} seems to contain valid transactions ; journal id: {}

What it means

Thrown by the JournalNode's Journal.startLogSegment when a writer asks to start a segment at a txid where an in-progress edit log already exists, and scanning shows that file contains more than one transaction (lastTxId != firstTxId). A legitimate in-progress segment must contain only the single START_LOG_SEGMENT transaction, so extra transactions mean the file holds real writes that were never recovered. It protects the journal from silently overwriting committed edits and implies the writer and journal have diverged (missing recovery/fencing).

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:600

      abortCurSegment();
    }

    // Paranoid sanity check: we should never overwrite a finalized log file.
    // Additionally, if it's in-progress, it should have at most 1 transaction.
    // This can happen if the writer crashes exactly at the start of a segment.
    EditLogFile existing = fjm.getLogFile(txid);
    if (existing != null) {
      if (!existing.isInProgress()) {
        throw new IllegalStateException("Already have a finalized segment " +
            existing + " beginning at " + txid + " ; journal id: " + journalId);
      }
      
      // If it's in-progress, it should only contain one transaction,
      // because the "startLogSegment" transaction is written alone at the
      // start of each segment. 
      existing.scanLog(Long.MAX_VALUE, false);
      if (existing.getLastTxId() != existing.getFirstTxId()) {
        throw new IllegalStateException("The log file " +
            existing + " seems to contain valid transactions" +
            " ; journal id: " + journalId);
      }
    }
    
    long curLastWriterEpoch = lastWriterEpoch.get();
    if (curLastWriterEpoch != reqInfo.getEpoch()) {
      LOG.info("Updating lastWriterEpoch from " + curLastWriterEpoch +
          " to " + reqInfo.getEpoch() + " for client " +
          Server.getRemoteIp() + " ; journal id: " + journalId);
      lastWriterEpoch.set(reqInfo.getEpoch());
    }

    // The fact that we are starting a segment at this txid indicates
    // that any previous recovery for this same segment was aborted.
    // Otherwise, no writer would have started writing. So, we can
    // remove the record of the older segment here.
    purgePaxosDecision(txid);

View on GitHub (pinned to 2add963021)

Solutions

Trigger journal recovery so the JournalNode fences old writers and finalizes/purges the stray segment: restart the standby/failover flow, or call QJournalProtocol newEpoch + recoverSegments (e.g., via 'hdfs namenode -recoverJournal' style flow / zkc failover controller) before retrying startLogSegment.
If the journal dir is orphaned or from a test cluster, reformat the journal: 'hdfs journalnode -format' (or wipe the dfs.journalnode.edits.dir for that journal id) after confirming no live NameNode needs those edits.
Verify HA fencing (dfs.ha.fencing.methods, ZKFC) so only one Active can ever write; two writers at the same txid is the classic producer of this state.
Inspect the segment with 'hdfs oev' (OfflineEditsViewer) on the in-progress file to see whose transactions they are before discarding anything.

Example fix

# before: stale in-progress segment from a crashed writer blocks startLogSegment
# Journal throws IllegalStateException: 'The log file ... seems to contain valid transactions ; journal id: myjournal'

# after: fence old writers and recover segments, then the NN can start a fresh segment
hdfs zkfc -formatZK   # if fencing state is broken
# restart JournalNodes and let failover controller run recovery
hdfs haadmin -transitionToActive --forcemanual nn1  # only after recovery completes

Defensive patterns

Strategy: try-catch

Validate before calling

// Before starting a segment, confirm no conflicting in-progress segment exists
RemoteEditLogManifest m = journal.getEditLogManifest(startTxId, true);
boolean conflict = m.getLogs().stream()
    .anyMatch(l -> l.getStartTxId() == startTxId && l.isInProgress());
if (conflict) { /* run recovery (newEpoch + recoverSegments) first */ }

Try / catch

try {
  journal.startLogSegment(reqInfo, startTxId, segmentTxId);
} catch (IllegalStateException e) {
  // journal holds an unrecovered in-progress segment with real txns
  // fence + recover the journal (new epoch), then retry exactly once
  recoverJournalWithNewEpoch();
  journal.startLogSegment(reqInfo, startTxId, segmentTxId);
}

Prevention

Run journal recovery (newEpoch/recoverSegments) after every writer crash before a new writer starts.
Configure proper HA fencing (ZKFC, dfs.ha.fencing.methods) so two NameNodes never write the same journal concurrently.
Never re-use a journal directory across clusters or test runs without formatting.

When it happens

Trigger: QJournalProtocol.startLogSegment(txid) is called while an in-progress EditLogFile beginning at that txid exists on disk and existing.scanLog() reports firstTxId != lastTxId. Typically happens when a previous writer crashed mid-segment and no newEpoch/recoverSegments ran, or a second (unfenced) NameNode starts writing at the same txid.

Common situations: NameNode crash followed by failover without proper journal recovery; re-using a journal directory from another/older cluster; manual copying or partial deletion of journal dir contents; ZK fencing not configured so an old Active keeps writing; replaying txids after an edit-log roll back.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fee5ebd24105b581. Report an issue: GitHub.