prestodb/presto · error · IllegalArgumentException

Invalid offset: %s > %s

Error message

Invalid offset: %s > %s

What it means

When reading a string from the Variant metadata dictionary, the accessor reads the offset and nextOffset for the requested id and requires offset <= nextOffset (offsets must be non-decreasing). This IllegalArgumentException signals the dictionary offsets are inconsistent — the variant metadata bytes are malformed, so the string span [offset, nextOffset) is invalid and the reader refuses to slice it.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/spark/VariantUtil.java:505

    // Throw `MALFORMED_VARIANT` if the variant is malformed. An out-of-bound `id` is also considered
    // a malformed variant because it is read from the corresponding variant value.
    public static String getMetadataKey(byte[] metadata, int id)
    {
        checkIndex(0, metadata.length);
        // Extracts the highest 2 bits in the metadata header to determine the integer size of the
        // offset list.
        int offsetSize = ((metadata[0] >> 6) & 0x3) + 1;
        int dictSize = readUnsigned(metadata, 1, offsetSize);
        if (id >= dictSize) {
            throw new IllegalArgumentException(String.format("Index out of bound: %s (size: %s)", id, dictSize));
        }
        // There are a header byte, a `dictSize` with `offsetSize` bytes, and `(dictSize + 1)` offsets
        // before the string data.
        int stringStart = 1 + (dictSize + 2) * offsetSize;
        int offset = readUnsigned(metadata, 1 + (id + 1) * offsetSize, offsetSize);
        int nextOffset = readUnsigned(metadata, 1 + (id + 2) * offsetSize, offsetSize);
        if (offset > nextOffset) {
            throw new IllegalArgumentException(String.format("Invalid offset: %s > %s", offset, nextOffset));
        }
        checkIndex(stringStart + nextOffset - 1, metadata.length);
        return new String(metadata, stringStart + offset, nextOffset - offset, UTF_8);
    }
}

View on GitHub (pinned to 55bb57d202)

Solutions

1. Validate that the metadata offset list is non-decreasing before dereferencing entries
2. Confirm the metadata header byte (offset size bits) is intact and matches the array length
3. Re-read/regenerate the source data or re-write the variant column with a conforming writer
4. Catch IllegalArgumentException and treat the row's variant metadata as corrupt

Example fix

// before
String s = VariantUtil.getMetadataString(metadata, id);
// after
try {
    String s = VariantUtil.getMetadataString(metadata, id);
} catch (IllegalArgumentException e) {
    // malformed variant metadata: log row and skip
}

// Java
static boolean hasMonotonicOffsets(byte[] metadata, int id, int offsetSize) {
    int a = VariantUtil.readUnsigned(metadata, 1 + (id + 1) * offsetSize, offsetSize);
    int b = VariantUtil.readUnsigned(metadata, 1 + (id + 2) * offsetSize, offsetSize);
    return a <= b;
}

// Java
try {
    String s = VariantUtil.getMetadataString(metadata, id);
} catch (IllegalArgumentException e) {
    log.warn("Corrupt variant metadata offsets", e);
    // return null / skip row
}

When it happens

Trigger: Reading dictionary entry `id` from variant metadata where readUnsigned(metadata, 1+(id+1)*offsetSize, offsetSize) > readUnsigned(metadata, 1+(id+2)*offsetSize, offsetSize) — i.e. corrupt or out-of-spec offset table in the metadata byte[].

Common situations: Corrupted Parquet variant columns; metadata written by a non-conforming producer; parsing metadata with the wrong offsetSize because the header byte was damaged; truncation that garbles the offset list.

Related errors

• Unexpected type:
• Index out of bound: %s (size: %s)
• Decimal out of bound:
• HIVE_PARTITION_SCHEMA_MISMATCH
• HIVE_MISSING_DATA

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/9d5c784d925cb67c. Report an issue: GitHub.