apache/hadoop · error · IOException

Unexpect end of stream, expected length:%s, actual:%s

Error message

Unexpect end of stream, expected length:%s, actual:%s

What it means

FileStore.copyInputStreamToFile() (the staging path used by put() and uploadPart()) copies exactly contentLength bytes into a .tmp file and throws this IOException (wrapped in RuntimeException) when the source stream provides fewer bytes than the declared contentLength. The tmp file is deleted and the whole put/uploadPart fails: the declared length and the actual stream length disagree.

Source

Thrown at hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java:387

      if (copiedBytes < contentLength) {
        throw new IOException(String.format("Unexpect end of stream, expected to write length:%s,"
                + " actual written:%s", contentLength, copiedBytes));
      }
    } catch (IOException e) {
      throw new RuntimeException(e);
    } finally {
      CommonUtils.runQuietly(in::close);
    }
  }

  private static void copyInputStreamToFile(InputStream in, File partFile, long contentLength) {
    File tmpFile = createTmpFile(partFile);
    try (FileOutputStream out = new FileOutputStream(tmpFile)) {
      long copiedBytes = IOUtils.copyLarge(in, out, 0, contentLength);

      if (copiedBytes < contentLength) {
        throw new IOException(
            String.format("Unexpect end of stream, expected length:%s, actual:%s", contentLength,
                tmpFile.length()));
      }
    } catch (IOException e) {
      CommonUtils.runQuietly(() -> FileUtils.delete(tmpFile));
      throw new RuntimeException(e);
    } finally {
      CommonUtils.runQuietly(in::close);
    }

    if (!tmpFile.renameTo(partFile)) {
      throw new RuntimeException("failed to put file since rename fail.");
    }
  }

  @Override
  public byte[] completeUpload(String key, String uploadId, List<Part> uploadParts) {
    Preconditions.checkArgument(uploadParts != null && uploadParts.size() > 0,

View on GitHub (pinned to 2add963021)

Solutions

1. Materialize the payload first and use its real size: byte[] data = IOUtils.toByteArray(source); put(key, () -> new ByteArrayInputStream(data), data.length)
2. Fix the length computation at the call site to be the number of bytes actually produced, not the requested/allocated size
3. Hand a brand-new, unread stream to the provider; do not pre-read or wrap the same stream instance twice
4. On failure the tmp file is already cleaned up — correct the length and simply retry the put

Example fix

// before: length does not match the stream
storage.put(key, () -> shortStream, declaredLen); // Unexpect end of stream, expected length ... actual ...
// after: buffer once, declare the true length
byte[] data = org.apache.commons.io.IOUtils.toByteArray(source);
storage.put(key, () -> new java.io.ByteArrayInputStream(data), data.length);

byte[] data = org.apache.commons.io.IOUtils.toByteArray(source);
if (data.length != declaredLength) {
  throw new IOException("refusing put: stream has " + data.length
      + " bytes but " + declaredLength + " were declared");
}
storage.put(key, () -> new java.io.ByteArrayInputStream(data), data.length);

When it happens

Trigger: storage.put(key, streamProvider, contentLength) or uploadPart(...) where streamProvider.newStream() yields fewer than contentLength bytes before EOF — wrong length passed, stream pre-consumed, or producer truncation.

Common situations: Callers passing buffer.length instead of bytesActuallyWritten; streams drained by an earlier checksum computation; unit tests feeding short ByteArrayInputStreams with optimistic lengths; OutputStream wrappers that compute remaining() incorrectly.

Related errors

• File not found %s
• %s is not appendable because append non-existed object with
• Unexpect end of stream, expected to write length:%s, actual
• Failed to create root dir. %s
• failed to create tmp file

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ae418c28c44749cd. Report an issue: GitHub.