languagetool-org/languagetool · error

No ngram data found for:

Error message

No ngram data found for: 

What it means

AggregatedNgramToLucene.indexLine() splits each input line into an ngram and its count, then looks up a LuceneIndex keyed by the ngram's token count (word count). If indexes.get(ngramParts.length) returns null — the aggregated file contains an ngram length for which no index was opened — the line cannot be routed and the tool throws with the offending line parts.

Source

Thrown at languagetool-dev/src/main/java/org/languagetool/dev/bigdata/AggregatedNgramToLucene.java:80

        indexLine(line);
      }
    }
  }

  private void indexLine(String line) throws IOException {
    if (lineCount++ % 250_000 == 0) {
      System.out.printf(Locale.ENGLISH, "Indexing line %d\n", lineCount);
    }
    String[] lineParts = line.split("\t");
    if (lineParts.length != 2) {
      System.err.println("Not 2 parts but " + lineParts.length + ", ignoring: '" + line + "'");
      return;
    }
    String ngram = lineParts[0];
    String[] ngramParts = ngram.split(" ");
    LuceneIndex index = indexes.get(ngramParts.length);
    if (index == null) {
      throw new RuntimeException("No ngram data found for: " + Arrays.toString(lineParts));
    }
    long count = Long.parseLong(lineParts[1]);
    if (ngramParts.length == 1) {
      totalTokenCount += count;
    }
    index.indexWriter.addDocument(getDoc(ngram, count));
  }

  @NotNull
  private Document getDoc(String ngram, long count) {
    Document doc = new Document();
    doc.add(new Field("ngram", ngram, StringField.TYPE_NOT_STORED));  // use StringField.TYPE_STORED for easier debugging with e.g. Luke
    doc.add(getCountField(count));
    return doc;
  }

  @NotNull
  private LongField getCountField(long count) {

View on GitHub (pinned to 2e990059ce)

Solutions

1. Ensure the indexes map is initialized for every ngram length present in the input (check the tool's ngram size configuration/main arguments).
2. Pre-validate the aggregated input: filter or split files by ngram length before indexing.
3. Log and skip unindexed lengths instead of throwing if out-of-range lines are expected noise.
4. Inspect the offending line printed in Arrays.toString(lineParts) to check whether it's malformed rather than a genuine long ngram.

Example fix

// before
LuceneIndex index = indexes.get(ngramParts.length);
if (index == null) {
  throw new RuntimeException("No ngram data found for: " + Arrays.toString(lineParts));
}
// after
LuceneIndex index = indexes.get(ngramParts.length);
if (index == null) {
  System.err.println("WARN: no index for ngram length " + ngramParts.length + ", skipping: " + ngram);
  return;
}

// verify ngram lengths in the input before indexing
int maxLen = indexes.keySet().stream().max(Integer::compare).orElse(0);
for (String line : Files.readAllLines(input)) {
  int len = line.split(" ")[0].trim().split(" ").length;
  if (len < 1 || len > maxLen) {
    System.err.println("Out-of-range ngram length " + len + ": " + line);
  }
}

try {
  indexer.indexInputFile(path);
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("No ngram data found")) {
    System.err.println("Re-run with indexes covering all ngram lengths in input");
  }
}

When it happens

Trigger: Processing an aggregated ngram TSV whose lines contain ngrams longer (or shorter) than the configured set of indexes — e.g. the tool was initialized for 1-5 word ngrams but the input contains 6-grams; also a malformed line where the split produces an unexpected token count.

Common situations: Feeding a mixed or misconfigured aggregated corpus into the indexer; running the tool with a max-ngram-size setting smaller than the data; whitespace/format changes in the aggregation output shifting the field layout.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• No hits for
• More hits than expected for
• Got more than one hit for:
• Directory must contain at least '1grams', '2grams', and '3gr
• Expected at least '1grams', '2grams', and '3grams' sub direc

AI-assisted analysis of languagetool-org/languagetool@2e990059ce (2026-09-06). Data as JSON: /api/errors/efe20ad95d1c8bb4. Report an issue: GitHub.