{"record":{"id":"ea83c0acd99d4a17","repo":"NationalSecurityAgency/ghidra","slug":"document-does-not-contain-complete-features","errorCode":null,"errorMessage":"Document does not contain complete \"features\"","messagePattern":"Document does not contain complete \"features\"","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/VectorCompareScriptFactory.java","lineNumber":110,"sourceCode":"\t\t\t\t}\n\t\t\t\telse {\n\t\t\t\t\tthrow new IOException(\"Document is missing \\\"features\\\"\");\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn offset;\n\t\t}\n\n\t\tprivate static int scanForLength(BytesRef byteRef, int startOffset) throws IOException {\n\t\t\tint finalLength = 0;\n\t\t\tint maxLength = byteRef.length - (startOffset - byteRef.offset);\n\t\t\twhile (finalLength < maxLength) {\n\t\t\t\tif (byteRef.bytes[finalLength + startOffset] == '\\\"') {\n\t\t\t\t\tbreak;\n\t\t\t\t}\n\t\t\t\tfinalLength += 1;\n\t\t\t}\n\t\t\tif (finalLength == byteRef.length) {\n\t\t\t\tthrow new IOException(\"Document does not contain complete \\\"features\\\"\");\n\t\t\t}\n\t\t\treturn finalLength;\n\t\t}\n\n\t\t@Override\n\t\tpublic ScoreScript newInstance(DocReader docReader) throws IOException {\n\t\t\treturn new ScoreScript(params, lookup, docReader) {\n\t\t\t\t@Override\n\t\t\t\tpublic double execute(ExplanationHolder explanation) {\n\t\t\t\t\ttry {\n\t\t\t\t\t\tDocValuesDocReader dvReader = (DocValuesDocReader) docReader;\n\t\t\t\t\t\tDocument document =\n\t\t\t\t\t\t\tdvReader.getLeafReaderContext().reader().document(_getDocId());\n\t\t\t\t\t\tBytesRef byteRef = document.getField(\"_source\").binaryValue();\n\t\t\t\t\t\tint valOffset = scanForFeatures(byteRef.bytes, byteRef.offset);\n\t\t\t\t\t\tint finalLength = scanForLength(byteRef, valOffset);\n\t\t\t\t\t\tInputStream inputStream =\n\t\t\t\t\t\t\tnew ByteArrayInputStream(byteRef.bytes, valOffset, finalLength);","sourceCodeStart":92,"sourceCodeEnd":128,"githubUrl":"https://github.com/NationalSecurityAgency/ghidra/blob/d5f144c24d6bc53c9cbf4448c6d11143e7696206/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/VectorCompareScriptFactory.java#L92-L128","documentation":"After the \"features\" key, scanForLength walks the bytes looking for a closing double-quote that delimits the base64 vector value. If it reaches the end of the buffer (finalLength == byteRef.length) without finding one, the value is truncated, so IOException is thrown.","triggerScenarios":"A document whose \"features\" value is unterminated (no closing `\"`) - a truncated source, a length cutoff, or a partial/corrupt write.","commonSituations":"Indexed source truncated by source filtering/excludes or size limits; _source disabled or capped; corrupt document; an indexing tool that failed to close the quoted string.","solutions":["Re-index the offending document with a complete, properly quoted features value.","Disable source truncation or increase Elasticsearch source/_source limits for that index.","Run a repair/re-index of affected docs and validate the stored source before querying."],"exampleFix":"// before: truncated source field\n{ \"features\": \"<partial-base64-without-closing-quote }\n// after: complete, properly closed value\n{ \"features\": \"<full-base64-vector>\" }","handlingStrategy":"validation","validationCode":"// Before querying, ensure the features value is well-formed (quoted, complete):\n// re-index documents whose stored _source has a truncated features value,\n// and disable _source truncation/size limits for the index.","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Disable source truncation or raise _source size limits on the BSim index.","Re-index and validate stored source completeness before running vector queries."],"tags":["elasticsearch","bsim","indexing","data-format"],"backgroundTag":null,"analyzedSha":"d5f144c24d6bc53c9cbf4448c6d11143e7696206","analyzedAt":"2026-08-14T01:00:57.564Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}