{"record":{"id":"78776f45adece778","repo":"NationalSecurityAgency/ghidra","slug":"missing-tokenizer-configuration","errorCode":null,"errorMessage":"Missing tokenizer configuration","messagePattern":"Missing tokenizer configuration","errorType":"exception","errorClass":"ElasticException","httpStatus":null,"severity":"error","filePath":"Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java","lineNumber":2411,"sourceCode":"\t\t\tif (ex.getMessage().contains(\"no such index\")) {\n\t\t\t\tthrow new NoDatabaseException(baseURL);\n\t\t\t}\n\t\t\tthrow ex;\n\t\t}\n\t\tJsonObject repo = (JsonObject) res.get(repository + \"_vector\");\n\t\tJsonObject settings = (JsonObject) repo.get(\"settings\");\n\t\tJsonObject index = (JsonObject) settings.get(\"index\");\n\t\tJsonObject analysis = (JsonObject) index.get(\"analysis\");\n\t\tJsonObject tokenizer = (JsonObject) analysis.get(\"tokenizer\");\n\t\tString tokenizerName = null;\n\t\tfor (String key : tokenizer.keySet()) {\n\t\t\tif (key.startsWith(\"lsh_\")) {\n\t\t\t\ttokenizerName = key;\n\t\t\t\tbreak;\n\t\t\t}\n\t\t}\n\t\tif (tokenizerName == null) {\n\t\t\tthrow new ElasticException(\"Missing tokenizer configuration\");\n\t\t}\n\t\tJsonObject tokenizerSettings = (JsonObject) tokenizer.get(tokenizerName);\n\t\tString idfWeights = tokenizerSettings.get(ElasticUtilities.LSH_WEIGHTS).getAsString();\n\t\tString[] split = idfWeights.split(\" \");\n\t\tdouble[] weightArray = new double[split.length];\n\t\tif (weightArray.length != config.weightfactory.getSize()) {\n\t\t\tthrow new ElasticException(\"weighttable has wrong number of rows\");\n\t\t}\n\t\tfor (int i = 0; i < weightArray.length; ++i) {\n\t\t\tweightArray[i] = Double.parseDouble(split[i]);\n\t\t}\n\t\tconfig.weightfactory.set(weightArray);\n\n\t\tString lookup = tokenizerSettings.get(ElasticUtilities.IDF_CONFIG).getAsString();\n\t\tsplit = lookup.split(\" \");\n\t\tint[] lookupArray = new int[split.length];\n\t\tfor (int i = 0; i < lookupArray.length; ++i) {\n\t\t\tlookupArray[i] = Integer.parseInt(split[i]);","sourceCodeStart":2393,"sourceCodeEnd":2429,"githubUrl":"https://github.com/NationalSecurityAgency/ghidra/blob/d5f144c24d6bc53c9cbf4448c6d11143e7696206/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java#L2393-L2429","documentation":"Thrown in initializeElastic when the vector index settings JSON does not contain an analysis section with a tokenizer whose name starts with \"lsh_\". The custom LSH tokenizer (type lsh_tokenizer) is created during createVectorIndex and is essential for BSim feature-vector comparison; its absence means the vector index was created without the BSim plugin or with an incompatible setup.","triggerScenarios":"During initialize() -> initializeElastic, after reading repository_vector/_settings. The code navigates settings -> index -> analysis -> tokenizer and searches keySet() for any key starting with \"lsh_\". If none is found, the exception is thrown.","commonSituations":"BSim Elasticsearch plugin (providing lsh_tokenizer type) is not installed on the node that created the index; the index was created manually or by a different tool bypassing createVectorIndex; corrupted index settings after a plugin removal or upgrade; querying a non-BSim index that happens to share the name prefix.","solutions":["Ensure the BSim Elasticsearch plugin is installed on all nodes: run bin/elasticsearch-plugin list and confirm the lsh plugin is present.","Restart Elasticsearch after installing the plugin so the lsh_tokenizer type is registered.","Drop and recreate the database using the proper generate command so createVectorIndex runs with the plugin active.","Verify the repository name does not collide with a pre-existing non-BSim index."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Verify the lsh tokenizer exists in vector index settings before initializing\nJsonObject settings = connection.executeURIOnly(ElasticConnection.GET, \"vector/_settings\");\nString settingsStr = settings.toString();\nif (!settingsStr.contains(\"lsh_tokenizer\") && !settingsStr.contains(\"lsh_\")) {\n    throw new IllegalStateException(\n        \"Vector index missing LSH tokenizer — BSim plugin may not be installed\");\n}","typeGuard":null,"tryCatchPattern":"try {\n    database.initialize();\n} catch (ElasticException e) {\n    if (e.getMessage().equals(\"Missing tokenizer configuration\")) {\n        Msg.error(this, \"BSim plugin not installed or vector index misconfigured: \" + e.getMessage());\n        // Reinstall plugin and recreate database\n    }\n    throw e;\n}","preventionTips":["Install the BSim Elasticsearch plugin on all nodes before creating the database.","Restart Elasticsearch after plugin installation to register the lsh_tokenizer type.","Verify plugin presence via bin/elasticsearch-plugin list in deployment scripts."],"tags":["elasticsearch","bsim","plugin","tokenizer","initialization","lsh"],"backgroundTag":null,"analyzedSha":"d5f144c24d6bc53c9cbf4448c6d11143e7696206","analyzedAt":"2026-08-14T01:00:57.564Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}