{"record":{"id":"c1a125faa73463b9","repo":"NationalSecurityAgency/ghidra","slug":"multi-search-for-exe-records-failed","errorCode":null,"errorMessage":"Multi-search for exe records failed","messagePattern":"Multi-search for exe records failed","errorType":"exception","errorClass":"ElasticException","httpStatus":null,"severity":"error","filePath":"Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java","lineNumber":1131,"sourceCode":"\t\tint count = 0;\n\t\tfor (int i = 0; i < maxDocuments; ++i) {\n\t\t\tString exeId = iter1.next().generateExeIdString();\n\t\t\tbuffer.append(\"{}\\n\");\t\t// Keep default index and type\n\t\t\tbuffer.append(\"{ \\\"query\\\": { \\\"bool\\\": { \\\"filter\\\": { \\\"term\\\": { \\\"_id\\\": \\\"\");\n\t\t\tbuffer.append(exeId);\n\t\t\tbuffer.append(\"\\\" }}}}}\\n\");\n\t\t\tcount += 1;\n\t\t\tif (!iter1.hasNext()) {\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\tJsonObject bulkobj = connection.executeBulk(path, buffer.toString());\n\t\tJsonArray responses = (JsonArray) bulkobj.get(\"responses\");\n\t\tfor (int i = 0; i < count; ++i) {\n\t\t\tJsonObject subquery = (JsonObject) responses.get(i);\n\t\t\tJsonElement hits = subquery.get(\"hits\");\n\t\t\tif (ElasticConnection.isNull(hits)) {\n\t\t\t\tthrow new ElasticException(\"Multi-search for exe records failed\");\n\t\t\t}\n\t\t\tJsonObject totalRec = (JsonObject) ((JsonObject) hits).get(\"total\");\n\t\t\tlong total = totalRec.get(\"value\").getAsLong();\n\t\t\tif (total != 1) {\n\t\t\t\tthrow new ElasticException(\"Could not recover unique executable via id\");\n\t\t\t}\n\t\t}\n\t\tfor (int i = 0; i < count; ++i) {\n\t\t\tJsonObject subquery = (JsonObject) responses.get(i);\n\t\t\tJsonObject hits = (JsonObject) subquery.get(\"hits\");\n\t\t\tJsonArray hitsArray = (JsonArray) hits.get(\"hits\");\n\t\t\thits = (JsonObject) hitsArray.get(0);\n\t\t\tExecutableRecord newExe = makeExecutableRecord(manager, hits);\n\t\t\tRowKey rowKey = iter2.next();\n\t\t\tmanager.cacheExecutableByRow(newExe, rowKey);\n\t\t}\n\t}\n","sourceCodeStart":1113,"sourceCodeEnd":1149,"githubUrl":"https://github.com/NationalSecurityAgency/ghidra/blob/d5f144c24d6bc53c9cbf4448c6d11143e7696206/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java#L1113-L1149","documentation":"Thrown in queryExecutableRecordById when one sub-response in an Elasticsearch _msearch (multi-search) bulk request is missing its \"hits\" field, as detected by ElasticConnection.isNull(hits). Each sub-response in an msearch should contain a hits object; its absence means that individual search failed (e.g., the error envelope replaced the normal hits object).","triggerScenarios":"Called when resolving executable records by document id in bulk via repository_executable/_msearch. After executeBulk returns, each subquery response is checked for a hits key. If a shard is unavailable, the index was deleted, or the document type mapping changed, the sub-response carries an error object instead of hits.","commonSituations":"An executable document was deleted between id collection and the msearch; a shard is relocating or recovering during the query; the executable index mapping was altered; Elasticsearch returns a 4xx-level error for a specific sub-request.","solutions":["Inspect the individual sub-response object (subquery) that triggered the failure to read the embedded error type and reason for the specific root cause.","Ensure no concurrent deletion operations are running against the repository while queries execute.","Check Elasticsearch cluster health and shard allocation; wait for green before retrying.","Retry the query after the transient shard issue resolves."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"int retries = 0;\nwhile (true) {\n    try {\n        database.query(query);\n        break;\n    } catch (ElasticException e) {\n        if (e.getMessage().contains(\"Multi-search for exe records failed\") && retries < 2) {\n            retries++;\n            continue; // transient shard issue — retry after brief pause\n        }\n        throw e;\n    }\n}","preventionTips":["Ensure no concurrent deletions target the executable index while queries run.","Wait for green cluster health before issuing queries to avoid hitting recovering shards.","Log the individual msearch sub-response errors for diagnosis when this occurs."],"tags":["elasticsearch","bsim","msearch","query","response-parsing","shard"],"backgroundTag":null,"analyzedSha":"d5f144c24d6bc53c9cbf4448c6d11143e7696206","analyzedAt":"2026-08-14T01:00:57.564Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}