{"record":{"id":"28302f1da367011c","repo":"apache/seatunnel","slug":"sparse-vector-index-too-large-d","errorCode":null,"errorMessage":"Sparse vector index too large: %d","messagePattern":"Sparse vector index too large: (.+?)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"seatunnel-common/src/main/java/org/apache/seatunnel/common/utils/VectorUtils.java","lineNumber":153,"sourceCode":"            return new Float[0];\n        }\n        int maxIndex = -1;\n        for (Map.Entry<?, ?> entry : sparseVector.entrySet()) {\n            Object key = entry.getKey();\n            if (!(key instanceof Integer)) {\n                throw new IllegalArgumentException(\n                        String.format(\n                                \"Sparse vector key must be Integer, but got: %s,\",\n                                key.getClass().getName()));\n            }\n            int index = (Integer) key;\n            if (index < 0) {\n                throw new IllegalArgumentException(\n                        String.format(\"Sparse vector index cannot be negative: %d\", index));\n            }\n            // prevent OOM\n            if (index > 1000000) {\n                throw new IllegalArgumentException(\n                        String.format(\"Sparse vector index too large: %d\", index));\n            }\n            maxIndex = Math.max(maxIndex, index);\n        }\n        Float[] denseVector = new Float[maxIndex + 1];\n        Arrays.fill(denseVector, 0.0f);\n        for (Map.Entry<?, ?> entry : sparseVector.entrySet()) {\n            Object key = entry.getKey();\n            Object value = entry.getValue();\n            if (!(value instanceof Number)) {\n                throw new IllegalArgumentException(\n                        String.format(\n                                \"Sparse vector value must be a Number, but got: %s\",\n                                value.getClass().getName()));\n            }\n            int index = (Integer) key;\n            denseVector[index] = ((Number) value).floatValue();\n        }","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-common/src/main/java/org/apache/seatunnel/common/utils/VectorUtils.java#L135-L171","documentation":"VectorUtils.convertSparseVectorToFloatArray throws this IllegalArgumentException when a sparse vector contains an index greater than 1,000,000. The limit exists to prevent an out-of-memory error when a single huge index would force allocation of a dense Float[maxIndex+1] array. It is a defensive sanity guard, not a semantic rule about the vector.","triggerScenarios":"Calling convertSparseVectorToFloatArray with a map whose key (index) exceeds 1000000, e.g. a token-ID or feature-index mapping built from a large vocabulary or corrupted data.","commonSituations":"Feeding sparse embeddings from large-vocabulary models (vocab > 1M), machine-generated sparse vectors where an ID field was mistakenly used as the vector index, or data corruption producing absurd indices.","solutions":["Check the sparse vector's indices before conversion and cap/validate them against 1000000.","If legitimate large indices are needed, split the vector or use a true sparse representation instead of densifying.","Remap indices to a dense 0..n-1 range before calling the utility.","If larger vectors must be supported, adjust the guard locally with an explicit memory budget decision."],"exampleFix":"// before\nMap<Integer, Float> v = Map.of(2500000, 0.5f);\nFloat[] dense = VectorUtils.convertSparseVectorToFloatArray(v); // throws\n// after\nMap<Integer, Float> v2 = v.entrySet().stream()\n    .filter(e -> e.getKey() >= 0 && e.getKey() <= 1000000)\n    .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));\nFloat[] dense2 = VectorUtils.convertSparseVectorToFloatArray(v2);","handlingStrategy":"validation","validationCode":"boolean isWithinIndexLimit(Map<Integer,?> v) {\n    return v.keySet().stream().allMatch(i -> i >= 0 && i <= 1000000);\n}\nif (!isWithinIndexLimit(sparse)) throw new IllegalArgumentException(\"index out of range\");","typeGuard":"boolean validIndex(Object key) { return key instanceof Integer && ((Integer) key) >= 0 && ((Integer) key) <= 1000000; }","tryCatchPattern":"try { Float[] d = VectorUtils.convertSparseVectorToFloatArray(v); } catch (IllegalArgumentException e) { log.error(\"Sparse vector rejected: {}\", e.getMessage()); }","preventionTips":["Validate max index against 1000000 before densifying","Keep vectors sparse instead of densifying huge-vocabulary vectors","Remap indices to a dense range before conversion"],"tags":["vector","sparse-vector","argument-out-of-range","java"],"backgroundTag":"value-out-of-range","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}