{"record":{"id":"255997f5b5720269","repo":"NationalSecurityAgency/ghidra","slug":"bad-encoding-in-result-document","errorCode":null,"errorMessage":"Bad encoding in result document","messagePattern":"Bad encoding in result document","errorType":"exception","errorClass":"ElasticException","httpStatus":null,"severity":"error","filePath":"Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java","lineNumber":878,"sourceCode":"\t\ttry {\n\t\t\tint returnedHits = hitsArray.size();\n\t\t\tfor (int i = 0; i < returnedHits; ++i) {\n\t\t\t\tJsonObject mainHit = (JsonObject) hitsArray.get(i);\n\t\t\t\tVectorResult vecRes = new VectorResult();\n\t\t\t\tvecRes.vectorid = Base64Lite.decodeLongBase64(mainHit.get(\"_id\").getAsString());\n\t\t\t\tvecRes.hitcount = -1;\t\t// Cannot fill in at this time\n\t\t\t\tvecRes.sim = mainHit.get(\"_score\").getAsDouble();\n\t\t\t\tJsonObject source = (JsonObject) mainHit.get(\"_source\");\n\t\t\t\tStringReader reader = new StringReader(source.get(\"features\").getAsString());\n\t\t\t\tvecRes.vec = vectorFactory.restoreVectorFromBase64(reader, decodeBuffer);\n\t\t\t\tvector.compareCounts(vecRes.vec, vecCompare);\n\t\t\t\tvecCompare.dotproduct = vecRes.sim * vector.getLength() * vecRes.vec.getLength();\n\t\t\t\tvecRes.signif = vectorFactory.calculateSignificance(vecCompare);\n\t\t\t\tlistResult.add(vecRes);\n\t\t\t}\n\t\t}\n\t\tcatch (IOException ex) {\n\t\t\tthrow new ElasticException(\"Bad encoding in result document\");\n\t\t}\n\t\tlong totalCount = 0;\n\t\tIterator<VectorResult> iter1 = listResult.iterator();\n\t\tIterator<VectorResult> iter2 = listResult.iterator();\n\t\twhile (iter1.hasNext()) {\n\t\t\ttotalCount += fetchVectorCounts(iter1, iter2, MAX_VECTORCOUNT_WINDOW);\n\t\t}\n\t\treturn totalCount;\n\t}\n\n\t/**\n\t * Returns the total number of hits in the given list of VectorResults\n\t * \n\t * @param listResult is the list of VectorResults\n\t * @return the total count\n\t */\n\tprivate int getTotalCount(List<VectorResult> listResult) {\n\t\tint count = 0;","sourceCodeStart":860,"sourceCodeEnd":896,"githubUrl":"https://github.com/NationalSecurityAgency/ghidra/blob/d5f144c24d6bc53c9cbf4448c6d11143e7696206/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java#L860-L896","documentation":"Thrown in queryNearestVector when an IOException is caught while decoding a feature vector from the Base64-encoded \"features\" field of a search-result _source document. The vectorFactory.restoreVectorFromBase64 call failed, meaning a stored vector document contains data that cannot be parsed back into an LSHVector.","triggerScenarios":"During queryNearestVector, iterating over the hitsArray from a vector/_search response. For one or more result hits, source.get(\"features\").getAsString() yields a string that restoreVectorFromBase64 cannot decode, raising an IOException that is caught and re-thrown as this ElasticException.","commonSituations":"Database corruption from disk errors or interrupted writes; vector written by an incompatible BSim client version using a different base64 encoding scheme; Elasticsearch reindexing or dynamic-mapping changes that altered the features text field; partial shard recovery serving stale or corrupted segment data.","solutions":["Identify the specific corrupted vector document by examining the hit _id values and query them individually via curl to inspect the raw features field.","Re-ingest the affected executable(s) to overwrite the corrupted vector documents with valid base64 data.","Check Elasticsearch cluster health (_cluster/health) for yellow/red status indicating shard or disk problems, and resolve any underlying storage issues.","If corruption is widespread, drop and recreate the database from the source executables using the generate command."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    database.query(queryNearest);\n} catch (ElasticException e) {\n    if (e.getMessage().contains(\"Bad encoding in result document\")) {\n        // Data corruption — identify and re-ingest the affected executable\n        Msg.error(this, \"Corrupted vector data detected; consider re-ingesting affected executables\");\n    }\n    throw e;\n}","preventionTips":["Monitor Elasticsearch cluster health and disk integrity to prevent segment corruption.","Ensure ingestion completes without interruption; use the same BSim version for ingest and query.","Periodically validate vector documents by sampling and decoding features fields."],"tags":["elasticsearch","bsim","data-corruption","base64","vector","decode"],"backgroundTag":null,"analyzedSha":"d5f144c24d6bc53c9cbf4448c6d11143e7696206","analyzedAt":"2026-08-14T01:00:57.564Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}