apache/hadoop · error · java.io.IOException

Can't scan a pre-transactional edit log.

Error message

Can't scan a pre-transactional edit log.

What it means

LegacyReader.scanOp() refuses to scan an edit log whose layout version predates LayoutVersion.Feature.STORED_TXIDS (introduced in Hadoop 0.21). Older formats do not persist a transaction id with each operation, so a txid-based scan cannot return meaningful ids and throws instead. decodeOp() still works on such logs (it substitutes INVALID_TXID); only the scan API is restricted.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java:5384

      FSEditLogOp op = cache.get(opCode);
      if (op == null) {
        throw new IOException("Read invalid opcode " + opCode);
      }
      if (NameNodeLayoutVersion.supports(
            LayoutVersion.Feature.STORED_TXIDS, logVersion)) {
        op.setTransactionId(in.readLong());
      } else {
        op.setTransactionId(HdfsServerConstants.INVALID_TXID);
      }
      op.readFields(in, logVersion);
      return op;
    }

    @Override
    public long scanOp() throws IOException {
      if (!NameNodeLayoutVersion.supports(
          LayoutVersion.Feature.STORED_TXIDS, logVersion)) {
        throw new IOException("Can't scan a pre-transactional edit log.");
      }
      FSEditLogOp op = decodeOp();
      return op == null ?
          HdfsServerConstants.INVALID_TXID : op.getTransactionId();
    }
  }

  public void outputToXml(ContentHandler contentHandler) throws SAXException {
    contentHandler.startElement("", "", "RECORD", new AttributesImpl());
    XMLUtils.addSaxString(contentHandler, "OPCODE", opCode.toString());
    contentHandler.startElement("", "", "DATA", new AttributesImpl());
    XMLUtils.addSaxString(contentHandler, "TXID", "" + txid);
    toXml(contentHandler);
    contentHandler.endElement("", "", "DATA");
    contentHandler.endElement("", "", "RECORD");
  }

  protected abstract void toXml(ContentHandler contentHandler)

View on GitHub (pinned to 2add963021)

Solutions

1. Confirm the log's era: the first bytes of a binary edits file are the (negative) layout version — inspect with 'xxd old_edits | head -2'.
2. Use a decode-based reader instead of scan-based processing, e.g. 'hdfs oev -i old_edits -o /tmp/old.xml'.
3. Replay the legacy log with a matching-era Hadoop distribution to convert it into a modern namespace.
4. Keep pre-0.21 format files out of directories that modern scan tooling iterates.

Example fix

# before: scan-based tooling fails
hdfs namenode -recover   # IOException: Can't scan a pre-transactional edit log.

# after: check version, then decode instead of scan
xxd old_edits | head -2
hdfs oev -i old_edits -o /tmp/old.xml

short logVersion;
try (DataInputStream in = new DataInputStream(
        new BufferedInputStream(new FileInputStream(editsFile)))) {
  logVersion = in.readShort();
}
boolean scannable = NameNodeLayoutVersion.supports(
    LayoutVersion.Feature.STORED_TXIDS, logVersion);
if (!scannable) {
  // skip txid-scan tooling; use a decode-based reader instead
}

try {
  long txid = stream.scanNextOp();
} catch (IOException e) {
  if (e.getMessage().contains("pre-transactional")) {
    // fall back to decodeOp()-based processing on this log
  }
}

When it happens

Trigger: Calling EditLogInputStream.scanOp()/scanNextOp() consumers on a 0.20.x-format edits file; txid-scanning recovery tooling or edit-log scanners pointed at a pre-0.21 log; mixing legacy edits files into a directory a modern tool scans.

Common situations: Reviving archives from very old 0.20-era clusters; pointing recovery or audit tooling at historical edits; metadata restored from pre-0.21 backups into a modern deployment.

Related errors

• got unexpected exception {message}
• Illegal option {}
• Not enough arguments: expected {} but got {}
• Too many arguments: expected {} but got {}
• No attribute for {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c15ea003a5ffedd2. Report an issue: GitHub.