apache/hadoop · error · IOException

Unable to write to output stream.

Error message

Unable to write to output stream.

What it means

Thrown by IOUtils.copyBytes(InputStream, OutputStream, int) when the destination is a PrintStream whose checkError() flag is set after a write. PrintStream swallows IOExceptions internally (it never throws them), so checkError() is the only way to detect that the underlying write failed. The message tells you bytes were copied into a stream that silently stopped accepting them.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:99

  }
  
  /**
   * Copies from one stream to another.
   * 
   * @param in InputStrem to read from
   * @param out OutputStream to write to
   * @param buffSize the size of the buffer.
   * @throws IOException raised on errors performing I/O.
   */
  public static void copyBytes(InputStream in, OutputStream out, int buffSize) 
    throws IOException {
    PrintStream ps = out instanceof PrintStream ? (PrintStream)out : null;
    byte buf[] = new byte[buffSize];
    int bytesRead = in.read(buf);
    while (bytesRead >= 0) {
      out.write(buf, 0, bytesRead);
      if ((ps != null) && ps.checkError()) {
        throw new IOException("Unable to write to output stream.");
      }
      bytesRead = in.read(buf);
    }
  }

  /**
   * Copies from one stream to another. <strong>closes the input and output streams 
   * at the end</strong>.
   *
   * @param in InputStrem to read from
   * @param out OutputStream to write to
   * @param conf the Configuration object.
   * @throws IOException raised on errors performing I/O.
   */
  public static void copyBytes(InputStream in, OutputStream out, Configuration conf)
    throws IOException {
    copyBytes(in, out, conf.getInt(
        IO_FILE_BUFFER_SIZE_KEY, IO_FILE_BUFFER_SIZE_DEFAULT), true);

View on GitHub (pinned to 2add963021)

Solutions

1. Identify why the PrintStream failed: if piping to a pager like 'head', that is expected broken-pipe behavior — pipe to 'cat' or consume the full output instead.
2. If redirecting to a file, check disk space (df -h) and write permissions of the target path.
3. Wrap the copyBytes call in try-catch (IOException) and stop the copy cleanly instead of letting the loop spin on a dead stream.
4. For programmatic bulk copies, pass a raw FSDataOutputStream/FileOutputStream rather than a PrintStream so failures surface as normal IOExceptions with a cause.

Example fix

// before: failures are silent until checkError trips, no cause available
IOUtils.copyBytes(fs.open(src), System.out, 4096);

// after: use a real OutputStream so IOException carries the cause; handle broken pipe explicitly
try (FSDataInputStream in = fs.open(src);
     OutputStream out = Files.newOutputStream(Paths.get("/tmp/out"))) {
  IOUtils.copyBytes(in, out, 4096, true);
} catch (IOException e) {
  throw new IOException("Copy to output failed: " + e.getMessage(), e);
}

if (out instanceof PrintStream && ((PrintStream) out).checkError()) {
  throw new IOException("PrintStream already in error state — aborting copy");
}
IOUtils.copyBytes(in, out, buffSize);

try {
  IOUtils.copyBytes(in, out, buffSize);
} catch (IOException e) {
  if (e.getMessage().equals("Unable to write to output stream.")) {
    // PrintStream swallowed the real cause (broken pipe / disk full) —
    // stop producing, nothing you write will succeed
    stopProducer();
  } else { throw e; }
}

When it happens

Trigger: Calling the 3-arg IOUtils.copyBytes(in, out, buffSize) where 'out' is a PrintStream (e.g. System.out or System.err) and the consumer of that stream has failed: reader closed the pipe, disk full when redirected to a file, or the print stream was closed. The check only runs because 'out instanceof PrintStream'; regular OutputStreams throw their own IOException instead.

Common situations: Running a Hadoop CLI tool and piping stdout to 'head' or 'less' then quitting (broken pipe / SIGPIPE); redirecting tool output to a full disk or read-only location; dumping file contents to System.out via copyBytes and the terminal/pipe dies mid-copy.

Related errors

• Error while reading compressed data
• Premature EOF from inputStream
• Premature EOF from inputStream after skipping {len-amt} byte
• File/Directory {path} does not exist
• Key length out of range: {klen}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/417bfd4b86fc332c. Report an issue: GitHub.