prestodb/presto · error · OrcCorruptionException

Read past end of buffer RLE byte

Error message

Read past end of buffer RLE byte

What it means

Thrown by ByteInputStream.readNextBlock when the underlying input returns EOF (-1) where a byte-level Run-Length-Encoding control byte was expected. Byte streams in ORC use RLE; losing the control byte means the stream data ended before the number of values promised by the metadata was delivered. This indicates a truncated or corrupt ORC file.

Source

Thrown at presto-orc/src/main/java/com/facebook/presto/orc/stream/ByteInputStream.java:49

    private int length;
    private int offset;
    private long lastReadInputCheckpoint;

    public ByteInputStream(OrcInputStream input)
    {
        this.input = input;
        lastReadInputCheckpoint = input.getCheckpoint();
    }

    // This is based on the Apache Hive ORC code
    private void readNextBlock()
            throws IOException
    {
        lastReadInputCheckpoint = input.getCheckpoint();

        int control = input.read();
        if (control == -1) {
            throw new OrcCorruptionException(input.getOrcDataSourceId(), "Read past end of buffer RLE byte");
        }

        offset = 0;

        // if byte high bit is not set, this is a repetition; otherwise it is a literal sequence
        if ((control & 0x80) == 0) {
            length = control + MIN_REPEAT_SIZE;

            // read the repeated value
            int value = input.read();
            if (value == -1) {
                throw new OrcCorruptionException(input.getOrcDataSourceId(), "Reading RLE byte got EOF");
            }

            // fill buffer with the value
            Arrays.fill(buffer, 0, length, (byte) value);
        }
        else {

View on GitHub (pinned to 55bb57d202)

Solutions

1. Validate the file with orc-tools meta/orc-dump and check the ORC postscript/footer checksums to confirm truncation.
2. Re-transfer the file from the authoritative source (re-copy in HDFS/S3) and verify sizes and checksums.
3. Re-enable/verify storage-level checksumming (HDFS dfs checksums, S3 ETag comparison) so silent corruption is detected at read time.
4. Rewrite the affected ORC file from source data.
5. If reading while a job is still writing, wait for the writer/commit protocol to finish before querying.

Example fix

// before: streaming a partially-uploaded S3 object
s3.getObject(new GetObjectRequest(bucket, key)); // key upload incomplete
// after: verify completeness first
HeadObjectResponse head = s3.headObject(...);
if (!head.etag().equals(expectedEtag)) { throw new IllegalStateException("incomplete copy"); }
s3.getObject(new GetObjectRequest(bucket, key));

// verify declared vs actual size before reading
File f = new File(path);
if (f.length() < declaredFileLengthFromManifest) {
    throw new IllegalStateException("file truncated: " + path);
}

try {
    return orcRecordReader.next();
} catch (OrcCorruptionException e) {
    if (e.getMessage().contains("Read past end of buffer RLE byte")) {
        return recoveryStrategy.recopyAndReopen(path);
    }
    throw e;
}

When it happens

Trigger: Any decoder calling ByteInputStream.next(...) or skip(...) when the compressed/encoded stream has fewer bytes than the row group requires; readNextBlock reads the control byte and input.read() returns -1.

Common situations: Files truncated by failed uploads/downloads (partial S3 multipart copy, interrupted HDFS write); disk corruption; reading a file that is still being written; mixing up file versions (older partial file cached); wrong checksums disabled on the storage layer.

Related errors

• Reading RLE byte got EOF
• Unexpected end of stream
• End of stream in RLE Integer
• Reading BigInteger past EOF
• EOF while reading unsigned vint

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/92f7ccfa006700c3. Report an issue: GitHub.