{"record":{"id":"4a2fba8d781db3ec","repo":"apache/cassandra","slug":"could-not-read-required-number-of-bytes-from-file","errorCode":null,"errorMessage":"could not read required number of bytes from file to be streamed: read %d bytes, wanted %d bytes","messagePattern":"could not read required number of bytes from file to be streamed: read (.+?) bytes, wanted (.+?) bytes","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/net/AsyncStreamingOutputPlus.java","lineNumber":187,"sourceCode":"\n    @VisibleForTesting\n    long writeFileToChannel(FileChannel fc, RateLimiter limiter, int batchSize) throws IOException\n    {\n        final long length = fc.size();\n        long bytesTransferred = 0;\n\n        try\n        {\n            while (bytesTransferred < length)\n            {\n                int toWrite = (int) min(batchSize, length - bytesTransferred);\n                final long position = bytesTransferred;\n\n                writeToChannel(bufferSupplier -> {\n                    ByteBuffer outBuffer = bufferSupplier.get(toWrite);\n                    long read = fc.read(outBuffer, position);\n                    if (read != toWrite)\n                        throw new IOException(String.format(\"could not read required number of bytes from \" +\n                                                            \"file to be streamed: read %d bytes, wanted %d bytes\",\n                                                            read, toWrite));\n                    outBuffer.flip();\n                }, limiter);\n\n                if (logger.isTraceEnabled())\n                    logger.trace(\"Writing {} bytes at position {} of {}\", toWrite, bytesTransferred, length);\n                bytesTransferred += toWrite;\n            }\n        }\n        finally\n        {\n            // we don't need to wait until byte buffer is flushed by netty\n            fc.close();\n        }\n\n        return bytesTransferred;\n    }","sourceCodeStart":169,"sourceCodeEnd":205,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/net/AsyncStreamingOutputPlus.java#L169-L205","documentation":"AsyncStreamingOutputPlus.writeFileToChannel streams a file to the network channel by reading slices via FileChannel.read into a buffer. If a read returns fewer bytes than requested (toWrite), the file could not supply the expected bytes and an IOException is thrown, since a short read would corrupt the streamed data.","triggerScenarios":"During writeFileToChannel, FileChannel.read(buffer, position) returns a value less than the requested byte count — e.g. the file was truncated or modified concurrently so fewer bytes remain at `position` than the planned `toWrite` size.","commonSituations":"A file being streamed is truncated or compacted/deleted concurrently by another process (e.g. compaction removing an SSTable mid-stream); filesystem errors; incorrect byte counts computed from a stale file length.","solutions":["Ensure the file being streamed is not deleted/truncated concurrently (hold a reference/retry logic that re-checks file length)","Re-check the file length before each chunk and clamp toWrite to the remaining bytes","Retry the streaming operation; the source file state was inconsistent","Verify filesystem health if short reads occur without concurrent modification"],"exampleFix":"// before\nlong toWrite = Math.min(chunkSize, fileLength - position);\nlong read = fc.read(outBuffer, position);\nif (read != toWrite) throw new IOException(...);\n\n// after\nlong fileLength = fc.size();\nlong toWrite = Math.min(chunkSize, fileLength - position);\nif (toWrite <= 0) break; // file shrank; stop instead of short-read\nlong read = fc.read(outBuffer, position);\nif (read != toWrite) throw new IOException(...);\n","handlingStrategy":"try-catch","validationCode":"long remaining = fileChannel.size() - position;\nif (remaining < toWrite) throw new IOException(\"File shorter than expected: \" + remaining + \" < \" + toWrite);","typeGuard":null,"tryCatchPattern":"try { output.writeFileToChannel(fileChannel, length, limits); }\ncatch (IOException e) {\n  if (e.getMessage().contains(\"could not read required number of bytes\")) {\n    logger.warn(\"Source file changed during stream, retrying\", e);\n    retryStream();\n  } else throw e;\n}","preventionTips":["Do not delete/compact files concurrently with streaming them","Verify file size immediately before streaming and clamp chunk sizes","Design streaming consumers to tolerate and retry short-read failures"],"tags":["streaming","file-io","short-read"],"backgroundTag":"file-read-failed","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}