NationalSecurityAgency/ghidra · error · ElasticException

Missing tokenizer configuration

Error message

Missing tokenizer configuration

What it means

Thrown in initializeElastic when the vector index settings JSON does not contain an analysis section with a tokenizer whose name starts with "lsh_". The custom LSH tokenizer (type lsh_tokenizer) is created during createVectorIndex and is essential for BSim feature-vector comparison; its absence means the vector index was created without the BSim plugin or with an incompatible setup.

Source

Thrown at Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java:2411

			if (ex.getMessage().contains("no such index")) {
				throw new NoDatabaseException(baseURL);
			}
			throw ex;
		}
		JsonObject repo = (JsonObject) res.get(repository + "_vector");
		JsonObject settings = (JsonObject) repo.get("settings");
		JsonObject index = (JsonObject) settings.get("index");
		JsonObject analysis = (JsonObject) index.get("analysis");
		JsonObject tokenizer = (JsonObject) analysis.get("tokenizer");
		String tokenizerName = null;
		for (String key : tokenizer.keySet()) {
			if (key.startsWith("lsh_")) {
				tokenizerName = key;
				break;
			}
		}
		if (tokenizerName == null) {
			throw new ElasticException("Missing tokenizer configuration");
		}
		JsonObject tokenizerSettings = (JsonObject) tokenizer.get(tokenizerName);
		String idfWeights = tokenizerSettings.get(ElasticUtilities.LSH_WEIGHTS).getAsString();
		String[] split = idfWeights.split(" ");
		double[] weightArray = new double[split.length];
		if (weightArray.length != config.weightfactory.getSize()) {
			throw new ElasticException("weighttable has wrong number of rows");
		}
		for (int i = 0; i < weightArray.length; ++i) {
			weightArray[i] = Double.parseDouble(split[i]);
		}
		config.weightfactory.set(weightArray);

		String lookup = tokenizerSettings.get(ElasticUtilities.IDF_CONFIG).getAsString();
		split = lookup.split(" ");
		int[] lookupArray = new int[split.length];
		for (int i = 0; i < lookupArray.length; ++i) {
			lookupArray[i] = Integer.parseInt(split[i]);

View on GitHub (pinned to d5f144c24d)

Solutions

1. Ensure the BSim Elasticsearch plugin is installed on all nodes: run bin/elasticsearch-plugin list and confirm the lsh plugin is present.
2. Restart Elasticsearch after installing the plugin so the lsh_tokenizer type is registered.
3. Drop and recreate the database using the proper generate command so createVectorIndex runs with the plugin active.
4. Verify the repository name does not collide with a pre-existing non-BSim index.

// Verify the lsh tokenizer exists in vector index settings before initializing
JsonObject settings = connection.executeURIOnly(ElasticConnection.GET, "vector/_settings");
String settingsStr = settings.toString();
if (!settingsStr.contains("lsh_tokenizer") && !settingsStr.contains("lsh_")) {
    throw new IllegalStateException(
        "Vector index missing LSH tokenizer — BSim plugin may not be installed");
}

try {
    database.initialize();
} catch (ElasticException e) {
    if (e.getMessage().equals("Missing tokenizer configuration")) {
        Msg.error(this, "BSim plugin not installed or vector index misconfigured: " + e.getMessage());
        // Reinstall plugin and recreate database
    }
    throw e;
}

When it happens

Trigger: During initialize() -> initializeElastic, after reading repository_vector/_settings. The code navigates settings -> index -> analysis -> tokenizer and searches keySet() for any key starting with "lsh_". If none is found, the exception is thrown.

Common situations: BSim Elasticsearch plugin (providing lsh_tokenizer type) is not installed on the node that created the index; the index was created manually or by a different tool bypassing createVectorIndex; corrupted index settings after a plugin removal or upgrade; querying a non-BSim index that happens to share the name prefix.

Related errors

• Could not find hits document
• Database instance does not exist
• Unrecoverable error: Could not find configuration
• Missing critical configuration value:
• weighttable has wrong number of rows

AI-assisted analysis of NationalSecurityAgency/ghidra@d5f144c24d (2026-08-14). Data as JSON: /api/errors/78776f45adece778. Report an issue: GitHub.