{"record":{"id":"96edc6847013fc70","repo":"apache/hadoop","slug":"already-have-a-finalized-segment-beginning-at","errorCode":null,"errorMessage":"Already have a finalized segment {} beginning at {} ; journal id: {}","messagePattern":"Already have a finalized segment (.+?) beginning at (.+?) ; journal id: (.+?)","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"critical","filePath":"hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java","lineNumber":591,"sourceCode":"    \n    if (curSegment != null) {\n      LOG.warn(\"Client is requesting a new log segment \" + txid + \n          \" though we are already writing \" + curSegment + \". \" +\n          \"Aborting the current segment in order to begin the new one.\" +\n          \" ; journal id: \" + journalId);\n      // The writer may have lost a connection to us and is now\n      // re-connecting after the connection came back.\n      // We should abort our own old segment.\n      abortCurSegment();\n    }\n\n    // Paranoid sanity check: we should never overwrite a finalized log file.\n    // Additionally, if it's in-progress, it should have at most 1 transaction.\n    // This can happen if the writer crashes exactly at the start of a segment.\n    EditLogFile existing = fjm.getLogFile(txid);\n    if (existing != null) {\n      if (!existing.isInProgress()) {\n        throw new IllegalStateException(\"Already have a finalized segment \" +\n            existing + \" beginning at \" + txid + \" ; journal id: \" + journalId);\n      }\n      \n      // If it's in-progress, it should only contain one transaction,\n      // because the \"startLogSegment\" transaction is written alone at the\n      // start of each segment. \n      existing.scanLog(Long.MAX_VALUE, false);\n      if (existing.getLastTxId() != existing.getFirstTxId()) {\n        throw new IllegalStateException(\"The log file \" +\n            existing + \" seems to contain valid transactions\" +\n            \" ; journal id: \" + journalId);\n      }\n    }\n    \n    long curLastWriterEpoch = lastWriterEpoch.get();\n    if (curLastWriterEpoch != reqInfo.getEpoch()) {\n      LOG.info(\"Updating lastWriterEpoch from \" + curLastWriterEpoch +\n          \" to \" + reqInfo.getEpoch() + \" for client \" +","sourceCodeStart":573,"sourceCodeEnd":609,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java#L573-L609","documentation":"In Journal.startLogSegment, a paranoid sanity check: before a writer starts a segment at txid, the Journal checks whether an edit log file already exists at that txid. Finding a FINALIZED file there is a corruption/fencing violation — finalized segments must never be overwritten — so it throws IllegalStateException to refuse rather than risk destroying committed transactions.","triggerScenarios":"A second writer starts a log segment at the same txid where this JN already holds a finalized segment: fencing hole (two NNs writing the same journal id), a JN restored from an inconsistent copy, manual file copying into the journal dir, or a rollback/recovery sequence that resurrected an old finalized segment at the same start txid.","commonSituations":"Manually forced active-active during HA troubleshooting; JN directory restored from a mixture of backups; files copied between JN dirs by hand; replaying old segment files into current; bugs where an old writer's segment raced a new writer. Because finalized (committed) data is at risk, treat as a corruption event.","solutions":["Immediately ensure exactly one NameNode can write: stop any second active for this nameservice and verify ZKFC fencing is active — concurrent writers are the most common cause.","Preserve the JN dir as evidence (copy it aside) before changing anything; identify which finalized segment is at that txid and compare it across all JNs.","Rebuild consistency: align the divergent JN from a healthy quorum member's journal dir, or use the NN recovery flow ('hdfs namenode -recover' as a last resort) to pick a canonical edit history.","If this reproduces with a single writer, capture the NN/JN logs around the event — startLogSegment hitting an existing finalized file with proper fencing indicates a genuine bug worth a JIRA."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// Server-side guard before accepting a new segment: refuse early if a\n// finalized file already exists at that txid (same check startLogSegment does)\nEditLogFile existing = fjm.getLogFile(txid);\nif (existing != null && !existing.isInProgress()) {\n  throw new IllegalStateException(\n      \"Refusing startLogSegment at \" + txid + \": finalized segment exists: \"\n      + existing);\n}","typeGuard":"static boolean isFinalizedSegmentConflict(Throwable t) {\n  return t instanceof IllegalStateException\n      && t.getMessage() != null\n      && t.getMessage().startsWith(\"Already have a finalized segment\");\n}","tryCatchPattern":"try {\n  jn.startLogSegment(reqInfo, txid, writerEpoch);\n} catch (IllegalStateException ise) {\n  if (isFinalizedSegmentConflict(ise)) {\n    // corruption / fencing event: preserve evidence, stop the writer,\n    // and reconcile JN dirs — never overwrite the finalized segment\n    quiesceAndPageAdmin(ise);\n  } else {\n    throw ise;\n  }\n}","preventionTips":["Never force two NameNodes active for one nameservice — always fail over via ZKFC.","Never copy edit segment files between JN directories by hand; use JournalNodeSyncer or documented recovery.","When restoring a JN, restore the whole directory from one healthy peer at one point in time."],"tags":["qjm","edit-log","segment-conflict","corruption","split-brain","fencing"],"backgroundTag":"duplicate-finalized-segment","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}