apache/hadoop · error · IOException

Tried to read {} byte(s) past the limit at offset {}

Error message

Tried to read {} byte(s) past the limit at offset {}

What it means

PositionTrackingInputStream.checkLimit bounds every read while FSEditLogOp.Reader decodes an op frame: the reader calls limiter.setLimit(maxOpSize) before each op (FSEditLogOp.java:5229), so a single op can never read past maxOpSize bytes (dfs.namenode.max.op.size, default 50MB). The throw means the stream's length or checksum fields directed a read beyond that bound: the anti-OOM guard for garbage data, per the comment at FSEditLogOp.java:5242.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java:1355

   * Stream wrapper that keeps track of the current stream position.
   * 
   * This stream also allows us to set a limit on how many bytes we can read
   * without getting an exception.
   */
  public static class PositionTrackingInputStream extends FilterInputStream
      implements StreamLimiter {
    private long curPos = 0;
    private long markPos = -1;
    private long limitPos = Long.MAX_VALUE;

    public PositionTrackingInputStream(InputStream is) {
      super(is);
    }

    private void checkLimit(long amt) throws IOException {
      long extra = (curPos + amt) - limitPos;
      if (extra > 0) {
        throw new IOException("Tried to read " + amt + " byte(s) past " +
            "the limit at offset " + limitPos);
      }
    }
    
    @Override
    public int read() throws IOException {
      checkLimit(1);
      int ret = super.read();
      if (ret != -1) curPos++;
      return ret;
    }

    @Override
    public int read(byte[] data) throws IOException {
      checkLimit(data.length);
      int ret = super.read(data);
      if (ret > 0) curPos += ret;
      return ret;

View on GitHub (pinned to 2add963021)

Solutions

1. Treat the segment as corrupt: validate with 'hdfs oev' and recover with 'hdfs namenode -recover'
2. Only if logs legitimately contain ops larger than 50MB, raise dfs.namenode.max.op.size; note that a garbage length field will still fail checksum verification
3. Restore a consistent fsimage plus edits from backup if recovery cannot salvage the segment

Example fix

# before: replay trips the per-op size limiter
hdfs --daemon start namenode   # Tried to read 33554433 byte(s) past the limit ...

# after: confirm corruption, then recover
hdfs oev -i <segment> -o /tmp/check.xml -p xml   # parse fails at the same op
hdfs namenode -recover
# alternative for genuinely huge ops only:
#   <property><name>dfs.namenode.max.op.size</name><value>104857600</value></property>

hdfs oev -i <segment> -o /tmp/check.xml -p xml
# the parser applies the same maxOpSize bound and checksum checks before replay does

// mirror scanEditLog: bound the damage, log, resync, continue only with progress
try {
  op = in.readOp();
} catch (IOException ioe) {          // includes 'Tried to read ... past the limit'
  LOG.warn("Bad op at offset " + in.getPosition(), ioe);
  in.resync();                        // skip to the next boundary; stop if no progress
}

When it happens

Trigger: Replaying or scanning (scanEditLog, EditLogFileInputStream.scanEditLog) a corrupt or torn edit segment where a bogus length field makes one op appear larger than maxOpSize; also possible on genuinely huge ops if the limit was lowered below their real size.

Common situations: Crash-torn final segment; disk corruption; dfs.namenode.max.op.size tuned far below actual op sizes (for example very long paths or large xattr ops).

Related errors

• Tried to skip {} bytes past the limit at offset {}
• Error replaying edit log at offset {}. Expected transaction
• The layout version {} supports inodeId but gave bogus inodeI
• Invalid operation read {}
• Mismatched block IDs or generation stamps for the old last b

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/693697026bf8184c. Report an issue: GitHub.