apache/hadoop · error · IOException

Mark/reset not supported

Error message

Mark/reset not supported

What it means

DFSInputStream.markSupported() returns false, mark() is a no-op, and reset() unconditionally throws IOException('Mark/reset not supported'). HDFS input streams do no front buffering (data streams straight from a Datanode), so the mark/reset contract cannot be honored. Any library that formats-sniffs by calling mark()/reset() will blow up on a raw FSDataInputStream.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java:1756

    }

    final long remaining = getFileLength() - pos;
    return remaining <= Integer.MAX_VALUE? (int)remaining: Integer.MAX_VALUE;
  }

  /**
   * We definitely don't support marks
   */
  @Override
  public boolean markSupported() {
    return false;
  }
  @Override
  public void mark(int readLimit) {
  }
  @Override
  public void reset() throws IOException {
    throw new IOException("Mark/reset not supported");
  }

  @Override
  public int read(long position, final ByteBuffer buf) throws IOException {
    if (!buf.hasRemaining()) {
      return 0;
    }
    return pread(position, buf);
  }

  @Override
  public void readFully(long position, final ByteBuffer buf)
      throws IOException {
    int nread = 0;
    while (buf.hasRemaining()) {
      int nbytes = read(position + nread, buf);
      if (nbytes < 0) {
        throw new EOFException(FSExceptionMessages.EOF_IN_READ_FULLY);

View on GitHub (pinned to 2add963021)

Solutions

1. Wrap the HDFS stream in a BufferedInputStream large enough for the detector's readLimit: new BufferedInputStream(fsin, 64*1024) - mark/reset then works.
2. For small files, read fully into memory and hand the library a ByteArrayInputStream (mark/reset supported, no wrapping complexity).
3. Use library helpers that buffer for you, e.g. TikaInputStream.get(stream, metadata) instead of the raw stream.
4. If it is your own code, replace mark/reset bookkeeping with explicit getPos()/seek(), which HDFS supports natively.

Example fix

// before
FSDataInputStream in = fs.open(path);
String type = detector.detect(in, metadata); // calls mark()/reset() -> throws

// after
FSDataInputStream in = fs.open(path);
BufferedInputStream buffered = new BufferedInputStream(in, 64 * 1024);
String type = detector.detect(buffered, metadata); // mark/reset on the buffer
in.seek(0); // when the raw stream must be re-read afterwards

InputStream probe = fs.open(path);
if (!probe.markSupported()) {
  probe = new BufferedInputStream(probe, 64 * 1024); // now mark/reset works
}

boolean supportsMarkReset(InputStream in) {
  return in != null && in.markSupported();
}

// usage
if (supportsMarkReset(in)) {
  in.mark(readLimit);
  sniff(in);
  in.reset();
} else {
  in = new BufferedInputStream(in, readLimit); // then mark/reset safely
}

When it happens

Trigger: Passing a raw HDFS FSDataInputStream to code that calls reset(): content-type detection (Apache Tika, mime4j), ImageIO.read, audio/video probe libraries, XML BOM sniffing. Sequence: library calls in.mark(readLimit), reads magic bytes, calls in.reset() -> IOException.

Common situations: Running Tika/document parsers directly over fs.open(path) output; ingest pipelines probing file type before dispatch; porting code that worked on FileInputStream/BufferedInputStream to HDFS paths.

Related errors

• Cannot seek to negative offset
• can't read a negative number of bytes.
• ${message}: ${currentFilePath} [${t}]
• Mark not supported
• Mark not set

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/0fd0a8f45b6b5604. Report an issue: GitHub.