NationalSecurityAgency/ghidra · error · IOException

Document does not contain complete "features"

Error message

Document does not contain complete "features"

What it means

After the "features" key, scanForLength walks the bytes looking for a closing double-quote that delimits the base64 vector value. If it reaches the end of the buffer (finalLength == byteRef.length) without finding one, the value is truncated, so IOException is thrown.

Source

Thrown at Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/VectorCompareScriptFactory.java:110

				}
				else {
					throw new IOException("Document is missing \"features\"");
				}
			}
			return offset;
		}

		private static int scanForLength(BytesRef byteRef, int startOffset) throws IOException {
			int finalLength = 0;
			int maxLength = byteRef.length - (startOffset - byteRef.offset);
			while (finalLength < maxLength) {
				if (byteRef.bytes[finalLength + startOffset] == '\"') {
					break;
				}
				finalLength += 1;
			}
			if (finalLength == byteRef.length) {
				throw new IOException("Document does not contain complete \"features\"");
			}
			return finalLength;
		}

		@Override
		public ScoreScript newInstance(DocReader docReader) throws IOException {
			return new ScoreScript(params, lookup, docReader) {
				@Override
				public double execute(ExplanationHolder explanation) {
					try {
						DocValuesDocReader dvReader = (DocValuesDocReader) docReader;
						Document document =
							dvReader.getLeafReaderContext().reader().document(_getDocId());
						BytesRef byteRef = document.getField("_source").binaryValue();
						int valOffset = scanForFeatures(byteRef.bytes, byteRef.offset);
						int finalLength = scanForLength(byteRef, valOffset);
						InputStream inputStream =
							new ByteArrayInputStream(byteRef.bytes, valOffset, finalLength);

View on GitHub (pinned to d5f144c24d)

Solutions

1. Re-index the offending document with a complete, properly quoted features value.
2. Disable source truncation or increase Elasticsearch source/_source limits for that index.
3. Run a repair/re-index of affected docs and validate the stored source before querying.

Example fix

// before: truncated source field
{ "features": "<partial-base64-without-closing-quote }
// after: complete, properly closed value
{ "features": "<full-base64-vector>" }

// Before querying, ensure the features value is well-formed (quoted, complete):
// re-index documents whose stored _source has a truncated features value,
// and disable _source truncation/size limits for the index.

When it happens

Trigger: A document whose "features" value is unterminated (no closing `"`) - a truncated source, a length cutoff, or a partial/corrupt write.

Common situations: Indexed source truncated by source filtering/excludes or size limits; _source disabled or capped; corrupt document; an indexing tool that failed to close the quoted string.

Related errors

• Document is missing "features"
• {getType()}scripts cannot be used for context [{context.name
• Unknown script name {scriptSource}
• Unknown error format
• Mismatch in metaid

AI-assisted analysis of NationalSecurityAgency/ghidra@d5f144c24d (2026-08-14). Data as JSON: /api/errors/ea83c0acd99d4a17. Report an issue: GitHub.