{"record":{"id":"96dee3a55cc0a6f4","repo":"languagetool-org/languagetool","slug":"got-more-than-one-hit-for","errorCode":null,"errorMessage":"Got more than one hit for: ","messagePattern":"Got more than one hit for: ","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"languagetool-dev/src/main/java/org/languagetool/dev/bigdata/CommonCrawlToNgram.java","lineNumber":199,"sourceCode":"    index.searcher = new IndexSearcher(index.reader);\n    for (Map.Entry<String, Long> entry : ngramToCount.entrySet()) {\n      Term ngram = new Term(\"ngram\", entry.getKey());\n      TopDocs topDocs = index.searcher.search(new TermQuery(ngram), 2);\n      //System.out.println(ngram + \" ==> \" + topDocs.totalHits);\n      if (topDocs.totalHits == 0) {\n        Document doc = getDoc(entry.getKey(), entry.getValue());\n        index.indexWriter.addDocument(doc);\n      } else if (topDocs.totalHits == 1) {\n        int docNumber = topDocs.scoreDocs[0].doc;\n        Document document = index.reader.document(docNumber);\n        long oldCount = Long.parseLong(document.getField(\"count\").stringValue());\n        //System.out.println(ngram + \" -> \" + oldCount + \"+\" + entry.getValue());\n        index.indexWriter.deleteDocuments(ngram);\n        index.indexWriter.addDocument(getDoc(entry.getKey(), oldCount + entry.getValue()));\n        // would probably be faster, but we currently rely on the count being a common field:\n        //indexWriter.updateNumericDocValue(ngram, \"count\", oldCount + entry.getValue());\n      } else if (topDocs.totalHits > 1) {\n        throw new RuntimeException(\"Got more than one hit for: \" + ngram);\n      }\n      //System.out.println(\"   \" + entry.getKey() + \" -> \" + entry.getValue());\n    }\n    if (ngramSize == 1) {\n      // TODO: runtime code will crash if there are more than 1000 of these docs, so update instead of delete\n      long total = ngramToCount.values().stream().mapToLong(Number::longValue).sum();\n      System.out.println(\"Adding totalTokenCount doc: \" + total);\n      addTotalTokenCountDoc(total, index.indexWriter);\n    }\n    System.out.println(\"Commit...\");\n    index.indexWriter.commit();\n    System.out.println(\"Commit done, indexing took \" + (System.currentTimeMillis()-startTime) + \"ms\");\n    ngramToCount.clear();\n  }\n\n  @NotNull\n  private Document getDoc(String ngram, long count) {\n    Document doc = new Document();","sourceCodeStart":181,"sourceCodeEnd":217,"githubUrl":"https://github.com/languagetool-org/languagetool/blob/2e990059ce67d5e2a0f7f7ca5d31160c6709df4b/languagetool-dev/src/main/java/org/languagetool/dev/bigdata/CommonCrawlToNgram.java#L181-L217","documentation":"CommonCrawlToNgram.writeToLucene() expects each ngram to correspond to at most one document in the Lucene index. After fetching topDocs for the ngram, if totalHits > 1 the index holds duplicate entries for the same ngram, so the delete-and-update logic (which assumes a single prior doc) cannot determine the correct old count and the tool throws.","triggerScenarios":"Updating counts for an ngram whose term query matches multiple documents — usually after the index was built more than once into the same directory (docs appended without deletion), or when duplicate keys were indexed from overlapping input files.","commonSituations":"Re-running the ngram indexing pipeline over an existing index directory; merging partial indexes with overlapping ngrams; interrupted runs that left partial duplicate documents behind.","solutions":["Rebuild the index from scratch in a clean directory so every ngram maps to exactly one document.","Deduplicate existing indexes: for each duplicate ngram, sum counts, deleteDocuments(ngram), and re-add a single doc.","Ensure every write path deletes the ngram before adding (the code already does deleteDocuments before addDocument on the single-hit path — apply the same on rebuild).","Consider using a single indexed unique field or updateDocument/atomic replace semantics to prevent duplicates structurally."],"exampleFix":"// before\n} else if (topDocs.totalHits > 1) {\n  throw new RuntimeException(\"Got more than one hit for: \" + ngram);\n}\n// after\n} else if (topDocs.totalHits > 1) {\n  long summed = 0;\n  for (ScoreDoc sd : topDocs.scoreDocs) {\n    summed += Long.parseLong(reader.document(sd.doc).get(\"count\"));\n  }\n  index.indexWriter.deleteDocuments(ngram);\n  index.indexWriter.addDocument(getDoc(ngram, summed));\n}","handlingStrategy":"validation","validationCode":"// check uniqueness of ngram docs before incremental updates\nTopDocs td = searcher.search(new TermQuery(new Term(\"ngram\", ngram)), 2);\nif (td.totalHits > 1) {\n  throw new IllegalStateException(\"Duplicate ngram docs in index: \" + ngram);\n}","typeGuard":null,"tryCatchPattern":"try {\n  writer.writeAndEvaluate(...);\n} catch (RuntimeException e) {\n  if (e.getMessage().startsWith(\"Got more than one hit\")) {\n    rebuildIndexInCleanDir();\n  }\n}","preventionTips":["Rebuild indexes from scratch instead of appending to existing directories","Always deleteDocuments(ngram) before addDocument when updating counts","Run a post-build duplicate check over sampled terms","Use atomic update (delete-then-add in one commit) semantics for count updates"],"tags":["java","lucene","ngram","duplicate-data"],"backgroundTag":"internal-invariant-violation","analyzedSha":"2e990059ce67d5e2a0f7f7ca5d31160c6709df4b","analyzedAt":"2026-09-06T09:20:17.015Z","contentChangedAt":"2026-09-06T09:20:17.015Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}