{"record":{"id":"03851de66088f47f","repo":"NationalSecurityAgency/ghidra","slug":"meta-document-does-not-exist-for-id","errorCode":null,"errorMessage":"meta document does not exist for id=","messagePattern":"meta document does not exist for id=","errorType":"exception","errorClass":"ElasticException","httpStatus":null,"severity":"critical","filePath":"Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java","lineNumber":648,"sourceCode":"\t\t}\n\t\tbuffer.append(\" ] }\");\n\t\tJsonObject resp =\n\t\t\tconnection.executeStatement(ElasticConnection.GET, \"meta/_mget\", buffer.toString());\n\t\tJsonArray docs = (JsonArray) resp.get(\"docs\");\n\t\tfor (int i = 0; i < maxDocuments; ++i) {\n\t\t\tif (!iter2.hasNext()) {\n\t\t\t\tbreak;\n\t\t\t}\n\t\t\tvecRes = iter2.next();\n\t\t\tJsonObject oneResp = (JsonObject) docs.get(i);\n\t\t\tString matchId = oneResp.get(\"_id\").getAsString();\n\t\t\tlong matchIdVal = Base64Lite.decodeLongBase64(matchId);\n\t\t\tif (matchIdVal != vecRes.vectorid) {\n\t\t\t\tthrow new ElasticException(\"Mismatch in metaid\");\n\t\t\t}\n\t\t\tJsonElement source = oneResp.get(\"_source\");\n\t\t\tif (ElasticConnection.isNull(source)) {\n\t\t\t\tthrow new ElasticException(\"meta document does not exist for id=\" + matchId);\n\t\t\t}\n\t\t\tlong count = ((JsonObject) source).get(\"count\").getAsLong();\n\t\t\ttotalCount += count;\n\t\t\tvecRes.hitcount = (int) count;\n\t\t}\n\t\treturn totalCount;\n\t}\n\n\t/**\n\t * Fetch vectors in bulk from the database, given a list of VectorResults with the vector ids\n\t * The vector documents are queried, then the resulting LSHVector objects are filled\n\t * in for the VectorResults by parsing the documents. Two iterators pointing to the same list\n\t * of VectorResults are required, one for building the query, one for filling in the LSHVectors.\n\t * If no exception is thrown, both iterators are advanced the same number of times.\n\t * @param iter1 is the iterator to VectorResults to fill in\n\t * @param iter2 is a copy of the first iterator\n\t * @param maxDocuments is the maximum number of documents to query for\n\t * @throws ElasticException for communication problems with the server","sourceCodeStart":630,"sourceCodeEnd":666,"githubUrl":"https://github.com/NationalSecurityAgency/ghidra/blob/d5f144c24d6bc53c9cbf4448c6d11143e7696206/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java#L630-L666","documentation":"In fetchVectorCounts, after confirming the meta document's _id matches, its _source is checked for null. A null/JsonNull _source means ES returned a hit for the id but the document source is absent: the meta doc was deleted or never indexed while a vector referencing it still exists. This is an index-consistency break.","triggerScenarios":"A vector result references a metaid, the meta/_mget returns a doc object with that _id but _source is null (found:false-style tombstone), so count cannot be read. Happens after a partial delete of meta docs, mid-ingest before meta is written, replication lag returning a stale tombstone, or index corruption.","commonSituations":"Interrupted BSim ingest that wrote vectors but not all meta docs; a delete/rollback that left dangling vector->meta references; replica inconsistency during a node failure.","solutions":["Treat as a transient consistency gap and retry the query after a short delay (replication/refresh may catch up).","If persistent, identify the dangling vector->meta reference and re-ingest the missing meta document, or rebuild the repository index.","Check ES cluster health (unassigned shards, red status) which often accompanies such gaps."],"exampleFix":"// before\nlong c = db.fetchVectorCounts(it1, it2, n);\n// after\ntry { return db.fetchVectorCounts(it1, it2, n); }\ncatch (ElasticException e) {\n  if (e.getMessage().startsWith(\"meta document does not exist\")) { sleep(2000); return db.fetchVectorCounts(it1copy, it2copy, n); }\n  throw e;\n}","handlingStrategy":"retry","validationCode":"// Pre-validate consistency by checking cluster health and the specific meta id presence:\npublic static boolean metaPresent(ElasticConnection c, String b64Id) throws ElasticException {\n    JsonObject r = c.executeStatementExpectFailure(ElasticConnection.GET, \"meta/_doc/\" + b64Id, \"\");\n    JsonElement found = ((JsonObject) r).get(\"found\");\n    return found != null && found.getAsString().equals(\"true\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    return db.fetchVectorCounts(it1, it2, n);\n} catch (ElasticException e) {\n    if (!e.getMessage().startsWith(\"meta document does not exist\")) throw e;\n    Thread.sleep(2000); // allow refresh/replication to converge\n    return db.fetchVectorCounts(copy(it1), copy(it2), n);\n}","preventionTips":["Wait for indexing/replication to settle before querying fresh data.","Monitor cluster health; yellow/red clusters produce such gaps.","Re-ingest dangling meta documents or rebuild the repository if gaps persist."],"tags":["elasticsearch","data-integrity","bsim","consistency"],"backgroundTag":null,"analyzedSha":"d5f144c24d6bc53c9cbf4448c6d11143e7696206","analyzedAt":"2026-08-14T01:00:57.564Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}