{"record":{"id":"ae418c28c44749cd","repo":"apache/hadoop","slug":"unexpect-end-of-stream-expected-length-s-actual","errorCode":null,"errorMessage":"Unexpect end of stream, expected length:%s, actual:%s","messagePattern":"Unexpect end of stream, expected length:(.+?), actual:(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java","lineNumber":387,"sourceCode":"\n      if (copiedBytes < contentLength) {\n        throw new IOException(String.format(\"Unexpect end of stream, expected to write length:%s,\"\n                + \" actual written:%s\", contentLength, copiedBytes));\n      }\n    } catch (IOException e) {\n      throw new RuntimeException(e);\n    } finally {\n      CommonUtils.runQuietly(in::close);\n    }\n  }\n\n  private static void copyInputStreamToFile(InputStream in, File partFile, long contentLength) {\n    File tmpFile = createTmpFile(partFile);\n    try (FileOutputStream out = new FileOutputStream(tmpFile)) {\n      long copiedBytes = IOUtils.copyLarge(in, out, 0, contentLength);\n\n      if (copiedBytes < contentLength) {\n        throw new IOException(\n            String.format(\"Unexpect end of stream, expected length:%s, actual:%s\", contentLength,\n                tmpFile.length()));\n      }\n    } catch (IOException e) {\n      CommonUtils.runQuietly(() -> FileUtils.delete(tmpFile));\n      throw new RuntimeException(e);\n    } finally {\n      CommonUtils.runQuietly(in::close);\n    }\n\n    if (!tmpFile.renameTo(partFile)) {\n      throw new RuntimeException(\"failed to put file since rename fail.\");\n    }\n  }\n\n  @Override\n  public byte[] completeUpload(String key, String uploadId, List<Part> uploadParts) {\n    Preconditions.checkArgument(uploadParts != null && uploadParts.size() > 0,","sourceCodeStart":369,"sourceCodeEnd":405,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java#L369-L405","documentation":"FileStore.copyInputStreamToFile() (the staging path used by put() and uploadPart()) copies exactly contentLength bytes into a .tmp file and throws this IOException (wrapped in RuntimeException) when the source stream provides fewer bytes than the declared contentLength. The tmp file is deleted and the whole put/uploadPart fails: the declared length and the actual stream length disagree.","triggerScenarios":"storage.put(key, streamProvider, contentLength) or uploadPart(...) where streamProvider.newStream() yields fewer than contentLength bytes before EOF — wrong length passed, stream pre-consumed, or producer truncation.","commonSituations":"Callers passing buffer.length instead of bytesActuallyWritten; streams drained by an earlier checksum computation; unit tests feeding short ByteArrayInputStreams with optimistic lengths; OutputStream wrappers that compute remaining() incorrectly.","solutions":["Materialize the payload first and use its real size: byte[] data = IOUtils.toByteArray(source); put(key, () -> new ByteArrayInputStream(data), data.length)","Fix the length computation at the call site to be the number of bytes actually produced, not the requested/allocated size","Hand a brand-new, unread stream to the provider; do not pre-read or wrap the same stream instance twice","On failure the tmp file is already cleaned up — correct the length and simply retry the put"],"exampleFix":"// before: length does not match the stream\nstorage.put(key, () -> shortStream, declaredLen); // Unexpect end of stream, expected length ... actual ...\n// after: buffer once, declare the true length\nbyte[] data = org.apache.commons.io.IOUtils.toByteArray(source);\nstorage.put(key, () -> new java.io.ByteArrayInputStream(data), data.length);","handlingStrategy":"validation","validationCode":"byte[] data = org.apache.commons.io.IOUtils.toByteArray(source);\nif (data.length != declaredLength) {\n  throw new IOException(\"refusing put: stream has \" + data.length\n      + \" bytes but \" + declaredLength + \" were declared\");\n}\nstorage.put(key, () -> new java.io.ByteArrayInputStream(data), data.length);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Derive length from the same buffer you pass: byte count written, not allocated size","Reuse the byte[] pattern (materialize once, ByteArrayInputStream provider) for all small-object writes","Keep checksum computation off the primary stream; compute from the materialized bytes instead","Treat any 'Unexpect end of stream' as a caller bug in length math, not a store failure"],"tags":["hadoop","tos","filestore","stream-length","object-storage"],"backgroundTag":"truncated-input-stream","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}