{"record":{"id":"6f65560b99791ddb","repo":"languagetool-org/languagetool","slug":"no-hits-for","errorCode":null,"errorMessage":"No hits for ","messagePattern":"No hits for ","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"languagetool-dev/src/main/java/org/languagetool/dev/archive/StartTokenCounter.java","lineNumber":64,"sourceCode":"    File dir = new File(\"/data/google-ngram-index/en/2grams\");\n    try (FSDirectory directory = FSDirectory.open(dir.toPath());\n         IndexReader reader = DirectoryReader.open(directory)) {\n      IndexSearcher searcher = new IndexSearcher(reader);\n      Fields fields = MultiFields.getFields(reader);\n      Terms ngrams = fields.terms(\"ngram\");\n      TermsEnum iterator = ngrams.iterator();\n      BytesRef next;\n      int i = 0;\n      while ((next = iterator.next()) != null) {\n        String term = next.utf8ToString();\n        if (term.startsWith(LanguageModel.GOOGLE_SENTENCE_START)) {\n          if (term.matches(\".*_(ADJ|ADV|NUM|VERB|ADP|NOUN|PRON|CONJ|DET|PRT)$\")) {\n            //System.out.println(\"ignore: \" + term);\n            continue;\n          }\n          TopDocs topDocs = searcher.search(new TermQuery(new Term(\"ngram\", term)), 3);\n          if (topDocs.totalHits == 0) {\n            throw new RuntimeException(\"No hits for \" + term + \": \" + topDocs.totalHits);\n          } else if (topDocs.totalHits == 1) {\n            int docId = topDocs.scoreDocs[0].doc;\n            Document document = reader.document(docId);\n            Long count = Long.parseLong(document.get(\"count\"));\n            //System.out.println(term + \" -> \" + count);\n            totalCount += count;\n            if (++i % 10_000 == 0) {\n              System.out.println(i + \" ... \" + totalCount);\n            }\n          } else {\n            throw new RuntimeException(\"More hits than expected for \" + term + \": \" + topDocs.totalHits);\n          }\n        }\n      }\n    }\n    System.out.println(\"==> \" + totalCount);\n  }\n  ","sourceCodeStart":46,"sourceCodeEnd":82,"githubUrl":"https://github.com/languagetool-org/languagetool/blob/2e990059ce67d5e2a0f7f7ca5d31160c6709df4b/languagetool-dev/src/main/java/org/languagetool/dev/archive/StartTokenCounter.java#L46-L82","documentation":"StartTokenCounter counts occurrences of sentence-start tokens in an ngram Lucene index. For every term looked up via searcher.search(new TermQuery(new Term(\"ngram\", term)), 3), it requires at least one hit; a totalHits of 0 means the term is absent from the index, so counting cannot proceed and it throws.","triggerScenarios":"Querying the Lucene ngram index with a term that was never indexed — e.g. the input corpus (parsed TSV, POS-stripped) contains a token whose exact surface form with the trailing _POS suffix does not exist in the index, the index was built from a different/smaller corpus, or the field name \"ngram\" doesn't match the indexed field.","commonSituations":"Building the Lucene index from one ngram corpus (e.g. Google Books) but querying with terms from another (web); index build interrupted so documents are missing; casing or morphological suffix mismatch between query term and indexed ngram strings.","solutions":["Confirm the Lucene index was built completely and from the same corpus the terms come from; rebuild with AggregatedNgramToLucene if needed.","Log and skip unknown terms instead of failing: change the throw to a warning + continue for rare OOV tokens.","Check that the query field name \"ngram\" and term formatting (including any _POS suffix) exactly match what getDoc() indexed.","Open the index with Luke or a small probe program to verify the missing term really is absent."],"exampleFix":"// before\nTopDocs topDocs = searcher.search(new TermQuery(new Term(\"ngram\", term)), 3);\nif (topDocs.totalHits == 0) {\n  throw new RuntimeException(\"No hits for \" + term + \": \" + topDocs.totalHits);\n}\n// after\nTopDocs topDocs = searcher.search(new TermQuery(new Term(\"ngram\", term)), 3);\nif (topDocs.totalHits == 0) {\n  System.err.println(\"WARN: no index hits, skipping term: \" + term);\n  continue;\n}","handlingStrategy":"try-catch","validationCode":"// probe the index before batch counting\ntry (IndexReader reader = DirectoryReader.open(indexDir)) {\n  IndexSearcher s = new IndexSearcher(reader);\n  if (s.search(new TermQuery(new Term(\"ngram\", term)), 1).totalHits == 0) {\n    System.err.println(\"Term not in index: \" + term);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  TopDocs topDocs = searcher.search(new TermQuery(new Term(\"ngram\", term)), 3);\n  if (topDocs.totalHits == 0) { /* log OOV and continue */ }\n} catch (RuntimeException e) {\n  skippedTerms.add(term);\n}","preventionTips":["Build the Lucene index and query terms from the same corpus and tokenization pipeline","Verify index completeness after building (doc count vs expected ngram count)","Normalize casing/POS-suffix formatting identically in index and query","Treat OOV terms as expected in sparse corpora: skip-and-log rather than fail-fast"],"tags":["java","lucene","ngram","search"],"backgroundTag":"empty-result-set","analyzedSha":"2e990059ce67d5e2a0f7f7ca5d31160c6709df4b","analyzedAt":"2026-09-06T09:20:17.015Z","contentChangedAt":"2026-09-06T09:20:17.015Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}