prestodb/presto · error · IllegalArgumentException

Index out of bound: %s (size: %s)

Error message

Index out of bound: %s (size: %s)

What it means

This is thrown by the Variant metadata/dictionary accessor in VariantUtil: the metadata header's first two bits give the offset-size, the next bytes give the dictionary size (dictSize), and `id` must be a valid index into that dictionary. This IllegalArgumentException is raised when the requested variant dictionary id is >= dictSize, i.e. the caller asked for a string key that does not exist in the variant's metadata dictionary — normally a sign of malformed variant bytes or a misread offset.

Source

Thrown at presto-parquet/src/main/java/com/facebook/presto/parquet/spark/VariantUtil.java:497

        // Extracts b1b0 to determine the integer size of the offset list.
        int offsetSize = (typeInfo & 0x3) + 1;
        int offsetStart = position + 1 + sizeBytes;
        int dataStart = offsetStart + (size + 1) * offsetSize;
        return handler.apply(size, offsetSize, offsetStart, dataStart);
    }

    // Get a key at `id` in the variant metadata.
    // Throw `MALFORMED_VARIANT` if the variant is malformed. An out-of-bound `id` is also considered
    // a malformed variant because it is read from the corresponding variant value.
    public static String getMetadataKey(byte[] metadata, int id)
    {
        checkIndex(0, metadata.length);
        // Extracts the highest 2 bits in the metadata header to determine the integer size of the
        // offset list.
        int offsetSize = ((metadata[0] >> 6) & 0x3) + 1;
        int dictSize = readUnsigned(metadata, 1, offsetSize);
        if (id >= dictSize) {
            throw new IllegalArgumentException(String.format("Index out of bound: %s (size: %s)", id, dictSize));
        }
        // There are a header byte, a `dictSize` with `offsetSize` bytes, and `(dictSize + 1)` offsets
        // before the string data.
        int stringStart = 1 + (dictSize + 2) * offsetSize;
        int offset = readUnsigned(metadata, 1 + (id + 1) * offsetSize, offsetSize);
        int nextOffset = readUnsigned(metadata, 1 + (id + 2) * offsetSize, offsetSize);
        if (offset > nextOffset) {
            throw new IllegalArgumentException(String.format("Invalid offset: %s > %s", offset, nextOffset));
        }
        checkIndex(stringStart + nextOffset - 1, metadata.length);
        return new String(metadata, stringStart + offset, nextOffset - offset, UTF_8);
    }
}

View on GitHub (pinned to 55bb57d202)

Solutions

  1. Ensure the value[] and metadata[] byte arrays passed together come from the same variant cell
  2. Validate the metadata header (offsetSize, dictSize) before indexing and reject inconsistent pairs
  3. Re-read/regenerate the source Parquet data to rule out corruption
  4. Catch IllegalArgumentException and treat the variant object as malformed

Example fix

// before
String key = VariantUtil.getMetadataString(metadata, fieldId);
// after
try {
    String key = VariantUtil.getMetadataString(metadata, fieldId);
} catch (IllegalArgumentException e) {
    throw new IllegalArgumentException("Field id out of variant dictionary bounds: " + fieldId, e);
}
Defensive patterns

Strategy: validation

Validate before calling

// Java
static boolean isIdInDictionary(byte[] metadata, int id) {
    if (metadata == null || metadata.length == 0) return false;
    int offsetSize = ((metadata[0] >> 6) & 0x3) + 1;
    int dictSize = VariantUtil.readUnsigned(metadata, 1, offsetSize);
    return id >= 0 && id < dictSize;
}

Try / catch

// Java
try {
    String key = VariantUtil.getMetadataString(metadata, id);
} catch (IllegalArgumentException e) {
    // value/metadata mismatch or corrupt metadata: skip the row
}

Prevention

When it happens

Trigger: Resolving an object field or array element in a Variant whose field-id `id` (read from the value's object header) is greater than or equal to the dictSize parsed from the metadata header — happens when value and metadata arrays do not belong together, or the metadata bytes are corrupt.

Common situations: Pairing a variant value[] from one row with a metadata[] from another; corrupted Parquet variant columns; writers that violate the Variant metadata layout so dictSize is parsed wrong.

Related errors

AI-assisted analysis of prestodb/presto@55bb57d202 (2026-09-04). Data as JSON: /api/errors/ca4434d18fb07299. Report an issue: GitHub.