apache/hadoop · error · IOException

Cannot start a new log segment at txid {} since only up to t

Error message

Cannot start a new log segment at txid {} since only up to txid {} have been written in the log segment starting at {}.

What it means

Error "Cannot start a new log segment at txid {} since only up to txid {} have been written in the log segment starting at {}." thrown in apache/hadoop.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java:1401

  public synchronized void startLogSegment(long txid, 
      boolean abortCurrentLogSegment, int layoutVersion) throws IOException {
    LOG.info("Started a new log segment at txid " + txid);
    if (isSegmentOpen()) {
      if (getLastWrittenTxId() == txid - 1) {
        //In sync with the NN, so end and finalize the current segment`
        endCurrentLogSegment(false);
      } else {
        //Missed some transactions: probably lost contact with NN temporarily.
        final String mess = "Cannot start a new log segment at txid " + txid
            + " since only up to txid " + getLastWrittenTxId()
            + " have been written in the log segment starting at "
            + getCurSegmentTxId() + ".";
        if (abortCurrentLogSegment) {
          //Mark the current segment as aborted.
          LOG.warn(mess);
          abortCurrentLogSegment();
        } else {
          throw new IOException(mess);
        }
      }
    }
    setNextTxId(txid);
    startLogSegment(txid, layoutVersion);
  }
  
  /**
   * Start writing to the log segment with the given txid.
   * Transitions from BETWEEN_LOG_SEGMENTS state to IN_LOG_SEGMENT state. 
   */
  private void startLogSegment(final long segmentTxId, int layoutVersion)
      throws IOException {
    assert Thread.holdsLock(this);

    LOG.info("Starting log segment at " + segmentTxId);
    Preconditions.checkArgument(segmentTxId > 0,
        "Bad txid: %s", segmentTxId);

View on GitHub (pinned to 2add963021)

Solutions

1. Let the current log segment finish writing up to the requested txid before rolling; check for stalled journal writes.
2. Investigate missing/corrupt edits in the current segment and recover from a healthy journal or checkpoint.

When it happens

Trigger: Rolling the edit log (startLogSegment) to a txid beyond what the current in-progress segment has actually written, e.g. during shared-storage HA transition.

Common situations: See trigger scenarios.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/27f3350fc1ef2f24. Report an issue: GitHub.