juicedata/juicefs · error · IOException

stream was closed

Error message

stream was closed

What it means

FSDataInputStream.getPos() on JuiceFileSystemImpl.FileInputStream throws IOException("stream was closed") once the stream has been closed. Internally the buffered read implementation nulls its ByteBuffer in close(); any subsequent getPos() sees buf == null and throws rather than returning a stale position. This enforces the Hadoop FSInputStream contract that a closed stream is unusable.

Source

Thrown at sdk/java/src/main/java/io/juicefs/JuiceFileSystemImpl.java:1075

    private final Path path;

    private ByteBuffer buf;
    private long position;
    private long fileLen;

    public FileInputStream(Path f, int fd, int size, long fileLen) throws IOException {
      path = f;
      this.fd = fd;
      buf = directBufferPool.getBuffer(size);
      buf.limit(0);
      position = 0;
      this.fileLen = fileLen;
    }

    @Override
    public synchronized long getPos() throws IOException {
      if (buf == null)
        throw new IOException("stream was closed");
      return position - buf.remaining();
    }

    @Override
    public boolean seekToNewSource(long targetPos) throws IOException {
      return false;
    }

    @Override
    public synchronized int available() throws IOException {
      if (buf == null)
        throw new IOException("stream was closed");
      long remaining = fileLen - position + buf.remaining();
      if (remaining > Integer.MAX_VALUE) {
        return Integer.MAX_VALUE;
      }
      return (int)remaining;
    }

View on GitHub (pinned to c9a67b23e8)

Solutions

1. Capture getPos() before closing the stream.
2. Remove/stop any code that touches the stream after close() (progress reporters, logging callbacks).
3. Track closed state in the caller (boolean flag set at close) and skip stream operations afterwards.

Example fix

// before
long pos = in.getPos();
in.close();
LOG.info("stopped at " + in.getPos());
// after
long pos = in.getPos();
in.close();
LOG.info("stopped at " + pos);

boolean streamUsable = (in != null); // track close() yourself; set false in finally

try {
  long pos = in.getPos();
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("stream was closed")) {
    // stream already closed; use last recorded position
  } else {
    throw e;
  }
}

When it happens

Trigger: Calling in.getPos() after in.close() — e.g. a finally block that closes the stream and then logs/reports progress using getPos(), or a background progress thread still polling a stream another thread closed.

Common situations: MapReduce/Spark task cleanup ordering issues; double-close of FSDataInputStream in application code; reading statistics after try-with-resources has closed the stream.

Related errors

• Invalid start or len parameter
• arguments: " + off + " " + len
• position is negative
• Path already exists: " + f
• File already exists: " + f

AI-assisted analysis of juicedata/juicefs@c9a67b23e8 (2026-09-06). Data as JSON: /api/errors/511a6be1e269a4b0. Report an issue: GitHub.