{"record":{"id":"efe20ad95d1c8bb4","repo":"languagetool-org/languagetool","slug":"no-ngram-data-found-for","errorCode":null,"errorMessage":"No ngram data found for: ","messagePattern":"No ngram data found for: ","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"languagetool-dev/src/main/java/org/languagetool/dev/bigdata/AggregatedNgramToLucene.java","lineNumber":80,"sourceCode":"        indexLine(line);\n      }\n    }\n  }\n\n  private void indexLine(String line) throws IOException {\n    if (lineCount++ % 250_000 == 0) {\n      System.out.printf(Locale.ENGLISH, \"Indexing line %d\\n\", lineCount);\n    }\n    String[] lineParts = line.split(\"\\t\");\n    if (lineParts.length != 2) {\n      System.err.println(\"Not 2 parts but \" + lineParts.length + \", ignoring: '\" + line + \"'\");\n      return;\n    }\n    String ngram = lineParts[0];\n    String[] ngramParts = ngram.split(\" \");\n    LuceneIndex index = indexes.get(ngramParts.length);\n    if (index == null) {\n      throw new RuntimeException(\"No ngram data found for: \" + Arrays.toString(lineParts));\n    }\n    long count = Long.parseLong(lineParts[1]);\n    if (ngramParts.length == 1) {\n      totalTokenCount += count;\n    }\n    index.indexWriter.addDocument(getDoc(ngram, count));\n  }\n\n  @NotNull\n  private Document getDoc(String ngram, long count) {\n    Document doc = new Document();\n    doc.add(new Field(\"ngram\", ngram, StringField.TYPE_NOT_STORED));  // use StringField.TYPE_STORED for easier debugging with e.g. Luke\n    doc.add(getCountField(count));\n    return doc;\n  }\n\n  @NotNull\n  private LongField getCountField(long count) {","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/languagetool-org/languagetool/blob/2e990059ce67d5e2a0f7f7ca5d31160c6709df4b/languagetool-dev/src/main/java/org/languagetool/dev/bigdata/AggregatedNgramToLucene.java#L62-L98","documentation":"AggregatedNgramToLucene.indexLine() splits each input line into an ngram and its count, then looks up a LuceneIndex keyed by the ngram's token count (word count). If indexes.get(ngramParts.length) returns null — the aggregated file contains an ngram length for which no index was opened — the line cannot be routed and the tool throws with the offending line parts.","triggerScenarios":"Processing an aggregated ngram TSV whose lines contain ngrams longer (or shorter) than the configured set of indexes — e.g. the tool was initialized for 1-5 word ngrams but the input contains 6-grams; also a malformed line where the split produces an unexpected token count.","commonSituations":"Feeding a mixed or misconfigured aggregated corpus into the indexer; running the tool with a max-ngram-size setting smaller than the data; whitespace/format changes in the aggregation output shifting the field layout.","solutions":["Ensure the indexes map is initialized for every ngram length present in the input (check the tool's ngram size configuration/main arguments).","Pre-validate the aggregated input: filter or split files by ngram length before indexing.","Log and skip unindexed lengths instead of throwing if out-of-range lines are expected noise.","Inspect the offending line printed in Arrays.toString(lineParts) to check whether it's malformed rather than a genuine long ngram."],"exampleFix":"// before\nLuceneIndex index = indexes.get(ngramParts.length);\nif (index == null) {\n  throw new RuntimeException(\"No ngram data found for: \" + Arrays.toString(lineParts));\n}\n// after\nLuceneIndex index = indexes.get(ngramParts.length);\nif (index == null) {\n  System.err.println(\"WARN: no index for ngram length \" + ngramParts.length + \", skipping: \" + ngram);\n  return;\n}","handlingStrategy":"validation","validationCode":"// verify ngram lengths in the input before indexing\nint maxLen = indexes.keySet().stream().max(Integer::compare).orElse(0);\nfor (String line : Files.readAllLines(input)) {\n  int len = line.split(\" \")[0].trim().split(\" \").length;\n  if (len < 1 || len > maxLen) {\n    System.err.println(\"Out-of-range ngram length \" + len + \": \" + line);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  indexer.indexInputFile(path);\n} catch (RuntimeException e) {\n  if (e.getMessage().startsWith(\"No ngram data found\")) {\n    System.err.println(\"Re-run with indexes covering all ngram lengths in input\");\n  }\n}","preventionTips":["Configure the indexer's ngram size range to match the aggregated input data","Pre-split or filter aggregated TSV files by ngram length","Validate a sample of input lines before a long indexing run","Log-and-skip unindexed lengths if the input is known to contain stragglers"],"tags":["java","lucene","ngram","input-validation"],"backgroundTag":"invalid-argument-value","analyzedSha":"2e990059ce67d5e2a0f7f7ca5d31160c6709df4b","analyzedAt":"2026-09-06T09:20:17.015Z","contentChangedAt":"2026-09-06T09:20:17.015Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}