apache/hadoop · error · IOException

Unexpect end of stream, expected to write length:%s, actual

Error message

Unexpect end of stream, expected to write length:%s, actual written:%s

What it means

FileStore.appendInputStreamToFile() copies exactly contentLength bytes from the caller's InputStreamProvider into the destination file and throws this IOException (wrapped in RuntimeException) when IOUtils.copyLarge returns fewer bytes than declared: the supplied stream ended before the promised length. This is a caller contract violation — the declared contentLength does not match the actual bytes the stream can produce.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java:371

    File partFile = new File(uploadDir, String.valueOf(partNum));
    copyInputStreamToFile(streamProvider.newStream(), partFile, contentLength);

    try {
      byte[] data = Files.readAllBytes(partFile.toPath());
      return new Part(partNum, data.length, DigestUtils.md5Hex(data));
    } catch (IOException e) {
      LOG.error("failed to locate the part file: {}", partFile.getAbsolutePath());
      throw new RuntimeException(e);
    }
  }

  private static void appendInputStreamToFile(InputStream in, File partFile, long contentLength) {
    try (FileOutputStream out = new FileOutputStream(partFile, true)) {
      long copiedBytes = IOUtils.copyLarge(in, out, 0, contentLength);

      if (copiedBytes < contentLength) {
        throw new IOException(String.format("Unexpect end of stream, expected to write length:%s,"
                + " actual written:%s", contentLength, copiedBytes));
      }
    } catch (IOException e) {
      throw new RuntimeException(e);
    } finally {
      CommonUtils.runQuietly(in::close);
    }
  }

  private static void copyInputStreamToFile(InputStream in, File partFile, long contentLength) {
    File tmpFile = createTmpFile(partFile);
    try (FileOutputStream out = new FileOutputStream(tmpFile)) {
      long copiedBytes = IOUtils.copyLarge(in, out, 0, contentLength);

      if (copiedBytes < contentLength) {
        throw new IOException(
            String.format("Unexpect end of stream, expected length:%s, actual:%s", contentLength,
                tmpFile.length()));

View on GitHub (pinned to 2add963021)

Solutions

Buffer the payload once and derive the length from the buffer: byte[] data = IOUtils.toByteArray(source); append(key, () -> new ByteArrayInputStream(data), data.length)
Audit the code path that computes contentLength — use the count of bytes actually written to the buffer, not the requested read size
Ensure the stream is fresh and unread when passed to the InputStreamProvider; never reuse or pre-read the same stream instance
If the producer can truncate, read it fully into memory or a temp file first and validate the size before calling append

Example fix

// before: declared length larger than what the stream yields
storage.append(key, () -> partialStream, declaredLen); // Unexpect end of stream, expected ... actual written ...
// after: materialize bytes once, declare the true length
byte[] data = org.apache.commons.io.IOUtils.toByteArray(supplier);
storage.append(key, () -> new java.io.ByteArrayInputStream(data), data.length);

Defensive patterns

Strategy: validation

Validate before calling

// read the payload once and let its real length drive the call
byte[] data = org.apache.commons.io.IOUtils.toByteArray(source);
if (data.length != declaredLength) {
  throw new IOException("stream length " + data.length
      + " differs from declared " + declaredLength);
}
storage.append(key, () -> new java.io.ByteArrayInputStream(data), data.length);

Prevention

Compute contentLength from bytes actually buffered (position/count), never from buffer capacity or request size
Never hand over a stream that was already read (e.g. pre-consumed for hashing) — supply a fresh InputStreamProvider
Buffer or spool-to-disk first when the producer can truncate mid-stream, and validate size before the call
Add unit tests with short streams to catch length-math regressions early

When it happens

Trigger: storage.append(key, streamProvider, contentLength) where the provider's stream yields fewer than contentLength bytes: wrong length computed upstream, a stream already partially consumed or closed before being handed over, or a producer that truncates on error.

Common situations: Buffering layers that pass buffer capacity instead of buffer position as the length; streams wrapped after being read once (e.g. for checksumming) so they arrive pre-drained; network/producer sources that hit an error mid-stream and close early.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/cecdefdfb4cc44bd. Report an issue: GitHub.