{"record":{"id":"cecdefdfb4cc44bd","repo":"apache/hadoop","slug":"unexpect-end-of-stream-expected-to-write-length","errorCode":null,"errorMessage":"Unexpect end of stream, expected to write length:%s, actual written:%s","messagePattern":"Unexpect end of stream, expected to write length:(.+?), actual written:(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java","lineNumber":371,"sourceCode":"\n    File partFile = new File(uploadDir, String.valueOf(partNum));\n    copyInputStreamToFile(streamProvider.newStream(), partFile, contentLength);\n\n    try {\n      byte[] data = Files.readAllBytes(partFile.toPath());\n      return new Part(partNum, data.length, DigestUtils.md5Hex(data));\n    } catch (IOException e) {\n      LOG.error(\"failed to locate the part file: {}\", partFile.getAbsolutePath());\n      throw new RuntimeException(e);\n    }\n  }\n\n  private static void appendInputStreamToFile(InputStream in, File partFile, long contentLength) {\n    try (FileOutputStream out = new FileOutputStream(partFile, true)) {\n      long copiedBytes = IOUtils.copyLarge(in, out, 0, contentLength);\n\n      if (copiedBytes < contentLength) {\n        throw new IOException(String.format(\"Unexpect end of stream, expected to write length:%s,\"\n                + \" actual written:%s\", contentLength, copiedBytes));\n      }\n    } catch (IOException e) {\n      throw new RuntimeException(e);\n    } finally {\n      CommonUtils.runQuietly(in::close);\n    }\n  }\n\n  private static void copyInputStreamToFile(InputStream in, File partFile, long contentLength) {\n    File tmpFile = createTmpFile(partFile);\n    try (FileOutputStream out = new FileOutputStream(tmpFile)) {\n      long copiedBytes = IOUtils.copyLarge(in, out, 0, contentLength);\n\n      if (copiedBytes < contentLength) {\n        throw new IOException(\n            String.format(\"Unexpect end of stream, expected length:%s, actual:%s\", contentLength,\n                tmpFile.length()));","sourceCodeStart":353,"sourceCodeEnd":389,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-tos/src/main/java/org/apache/hadoop/fs/tosfs/object/FileStore.java#L353-L389","documentation":"FileStore.appendInputStreamToFile() copies exactly contentLength bytes from the caller's InputStreamProvider into the destination file and throws this IOException (wrapped in RuntimeException) when IOUtils.copyLarge returns fewer bytes than declared: the supplied stream ended before the promised length. This is a caller contract violation — the declared contentLength does not match the actual bytes the stream can produce.","triggerScenarios":"storage.append(key, streamProvider, contentLength) where the provider's stream yields fewer than contentLength bytes: wrong length computed upstream, a stream already partially consumed or closed before being handed over, or a producer that truncates on error.","commonSituations":"Buffering layers that pass buffer capacity instead of buffer position as the length; streams wrapped after being read once (e.g. for checksumming) so they arrive pre-drained; network/producer sources that hit an error mid-stream and close early.","solutions":["Buffer the payload once and derive the length from the buffer: byte[] data = IOUtils.toByteArray(source); append(key, () -> new ByteArrayInputStream(data), data.length)","Audit the code path that computes contentLength — use the count of bytes actually written to the buffer, not the requested read size","Ensure the stream is fresh and unread when passed to the InputStreamProvider; never reuse or pre-read the same stream instance","If the producer can truncate, read it fully into memory or a temp file first and validate the size before calling append"],"exampleFix":"// before: declared length larger than what the stream yields\nstorage.append(key, () -> partialStream, declaredLen); // Unexpect end of stream, expected ... actual written ...\n// after: materialize bytes once, declare the true length\nbyte[] data = org.apache.commons.io.IOUtils.toByteArray(supplier);\nstorage.append(key, () -> new java.io.ByteArrayInputStream(data), data.length);","handlingStrategy":"validation","validationCode":"// read the payload once and let its real length drive the call\nbyte[] data = org.apache.commons.io.IOUtils.toByteArray(source);\nif (data.length != declaredLength) {\n  throw new IOException(\"stream length \" + data.length\n      + \" differs from declared \" + declaredLength);\n}\nstorage.append(key, () -> new java.io.ByteArrayInputStream(data), data.length);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Compute contentLength from bytes actually buffered (position/count), never from buffer capacity or request size","Never hand over a stream that was already read (e.g. pre-consumed for hashing) — supply a fresh InputStreamProvider","Buffer or spool-to-disk first when the producer can truncate mid-stream, and validate size before the call","Add unit tests with short streams to catch length-math regressions early"],"tags":["hadoop","tos","filestore","stream-length","append"],"backgroundTag":"truncated-input-stream","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}