{"record":{"id":"92f7ccfa006700c3","repo":"prestodb/presto","slug":"read-past-end-of-buffer-rle-byte","errorCode":null,"errorMessage":"Read past end of buffer RLE byte","messagePattern":"Read past end of buffer RLE byte","errorType":"exception","errorClass":"OrcCorruptionException","httpStatus":null,"severity":"error","filePath":"presto-orc/src/main/java/com/facebook/presto/orc/stream/ByteInputStream.java","lineNumber":49,"sourceCode":"    private int length;\n    private int offset;\n    private long lastReadInputCheckpoint;\n\n    public ByteInputStream(OrcInputStream input)\n    {\n        this.input = input;\n        lastReadInputCheckpoint = input.getCheckpoint();\n    }\n\n    // This is based on the Apache Hive ORC code\n    private void readNextBlock()\n            throws IOException\n    {\n        lastReadInputCheckpoint = input.getCheckpoint();\n\n        int control = input.read();\n        if (control == -1) {\n            throw new OrcCorruptionException(input.getOrcDataSourceId(), \"Read past end of buffer RLE byte\");\n        }\n\n        offset = 0;\n\n        // if byte high bit is not set, this is a repetition; otherwise it is a literal sequence\n        if ((control & 0x80) == 0) {\n            length = control + MIN_REPEAT_SIZE;\n\n            // read the repeated value\n            int value = input.read();\n            if (value == -1) {\n                throw new OrcCorruptionException(input.getOrcDataSourceId(), \"Reading RLE byte got EOF\");\n            }\n\n            // fill buffer with the value\n            Arrays.fill(buffer, 0, length, (byte) value);\n        }\n        else {","sourceCodeStart":31,"sourceCodeEnd":67,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-orc/src/main/java/com/facebook/presto/orc/stream/ByteInputStream.java#L31-L67","documentation":"Thrown by ByteInputStream.readNextBlock when the underlying input returns EOF (-1) where a byte-level Run-Length-Encoding control byte was expected. Byte streams in ORC use RLE; losing the control byte means the stream data ended before the number of values promised by the metadata was delivered. This indicates a truncated or corrupt ORC file.","triggerScenarios":"Any decoder calling ByteInputStream.next(...) or skip(...) when the compressed/encoded stream has fewer bytes than the row group requires; readNextBlock reads the control byte and input.read() returns -1.","commonSituations":"Files truncated by failed uploads/downloads (partial S3 multipart copy, interrupted HDFS write); disk corruption; reading a file that is still being written; mixing up file versions (older partial file cached); wrong checksums disabled on the storage layer.","solutions":["Validate the file with orc-tools meta/orc-dump and check the ORC postscript/footer checksums to confirm truncation.","Re-transfer the file from the authoritative source (re-copy in HDFS/S3) and verify sizes and checksums.","Re-enable/verify storage-level checksumming (HDFS dfs checksums, S3 ETag comparison) so silent corruption is detected at read time.","Rewrite the affected ORC file from source data.","If reading while a job is still writing, wait for the writer/commit protocol to finish before querying."],"exampleFix":"// before: streaming a partially-uploaded S3 object\ns3.getObject(new GetObjectRequest(bucket, key)); // key upload incomplete\n// after: verify completeness first\nHeadObjectResponse head = s3.headObject(...);\nif (!head.etag().equals(expectedEtag)) { throw new IllegalStateException(\"incomplete copy\"); }\ns3.getObject(new GetObjectRequest(bucket, key));","handlingStrategy":"try-catch","validationCode":"// verify declared vs actual size before reading\nFile f = new File(path);\nif (f.length() < declaredFileLengthFromManifest) {\n    throw new IllegalStateException(\"file truncated: \" + path);\n}","typeGuard":null,"tryCatchPattern":"try {\n    return orcRecordReader.next();\n} catch (OrcCorruptionException e) {\n    if (e.getMessage().contains(\"Read past end of buffer RLE byte\")) {\n        return recoveryStrategy.recopyAndReopen(path);\n    }\n    throw e;\n}","preventionTips":["Compare S3 ETag / HDFS checksum after every copy before publishing the file.","Use commit protocols (Hive ACID, S3 'commit on success') so partial files never become visible.","Monitor writer job completion before scheduling readers.","Re-verify legacy cached files with orc-tools meta."],"tags":["orc","corruption","rle","eof","truncated-file"],"backgroundTag":"orc-file-corruption","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}