apache/seatunnel · error · IllegalArgumentException

Sparse vector index too large: %d

Error message

Sparse vector index too large: %d

What it means

VectorUtils.convertSparseVectorToFloatArray throws this IllegalArgumentException when a sparse vector contains an index greater than 1,000,000. The limit exists to prevent an out-of-memory error when a single huge index would force allocation of a dense Float[maxIndex+1] array. It is a defensive sanity guard, not a semantic rule about the vector.

Source

Thrown at seatunnel-common/src/main/java/org/apache/seatunnel/common/utils/VectorUtils.java:153

            return new Float[0];
        }
        int maxIndex = -1;
        for (Map.Entry<?, ?> entry : sparseVector.entrySet()) {
            Object key = entry.getKey();
            if (!(key instanceof Integer)) {
                throw new IllegalArgumentException(
                        String.format(
                                "Sparse vector key must be Integer, but got: %s,",
                                key.getClass().getName()));
            }
            int index = (Integer) key;
            if (index < 0) {
                throw new IllegalArgumentException(
                        String.format("Sparse vector index cannot be negative: %d", index));
            }
            // prevent OOM
            if (index > 1000000) {
                throw new IllegalArgumentException(
                        String.format("Sparse vector index too large: %d", index));
            }
            maxIndex = Math.max(maxIndex, index);
        }
        Float[] denseVector = new Float[maxIndex + 1];
        Arrays.fill(denseVector, 0.0f);
        for (Map.Entry<?, ?> entry : sparseVector.entrySet()) {
            Object key = entry.getKey();
            Object value = entry.getValue();
            if (!(value instanceof Number)) {
                throw new IllegalArgumentException(
                        String.format(
                                "Sparse vector value must be a Number, but got: %s",
                                value.getClass().getName()));
            }
            int index = (Integer) key;
            denseVector[index] = ((Number) value).floatValue();
        }

View on GitHub (pinned to cf67b549a7)

Solutions

1. Check the sparse vector's indices before conversion and cap/validate them against 1000000.
2. If legitimate large indices are needed, split the vector or use a true sparse representation instead of densifying.
3. Remap indices to a dense 0..n-1 range before calling the utility.
4. If larger vectors must be supported, adjust the guard locally with an explicit memory budget decision.

Example fix

// before
Map<Integer, Float> v = Map.of(2500000, 0.5f);
Float[] dense = VectorUtils.convertSparseVectorToFloatArray(v); // throws
// after
Map<Integer, Float> v2 = v.entrySet().stream()
    .filter(e -> e.getKey() >= 0 && e.getKey() <= 1000000)
    .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
Float[] dense2 = VectorUtils.convertSparseVectorToFloatArray(v2);

boolean isWithinIndexLimit(Map<Integer,?> v) {
    return v.keySet().stream().allMatch(i -> i >= 0 && i <= 1000000);
}
if (!isWithinIndexLimit(sparse)) throw new IllegalArgumentException("index out of range");

boolean validIndex(Object key) { return key instanceof Integer && ((Integer) key) >= 0 && ((Integer) key) <= 1000000; }

try { Float[] d = VectorUtils.convertSparseVectorToFloatArray(v); } catch (IllegalArgumentException e) { log.error("Sparse vector rejected: {}", e.getMessage()); }

When it happens

Trigger: Calling convertSparseVectorToFloatArray with a map whose key (index) exceeds 1000000, e.g. a token-ID or feature-index mapping built from a large vocabulary or corrupted data.

Common situations: Feeding sparse embeddings from large-vocabulary models (vocab > 1M), machine-generated sparse vectors where an ID field was mistakenly used as the vector index, or data corruption producing absurd indices.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Sparse vector value must be a Number, but got: %s
• Sparse vector key must be Integer, but got: %s,
• Unsupported convert ${value.getClass()} to LocalTime, typeDe
• Unsupported convert ${value.getClass()} to LocalTime
• Time values must use number of milliseconds greater than 0 a

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/28302f1da367011c. Report an issue: GitHub.