{"record":{"id":"4b8125519d20f231","repo":"languagetool-org/languagetool","slug":"could-not-index","errorCode":null,"errorMessage":"Could not index ","messagePattern":"Could not index ","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"languagetool-dev/src/main/java/org/languagetool/dev/bigdata/FrequencyIndexCreator.java","lineNumber":159,"sourceCode":"          indexLinesFromGoogleFile(globalDataWriter, file, totalBytes, hiveMode);\n        } else {\n          try (DataWriter dw = new TextDataWriter(indexDir)) {\n            indexLinesFromGoogleFile(dw, file, totalBytes, hiveMode);\n          }\n          markIndexAsComplete(indexDir);\n        }\n      } else {\n        if (globalDataWriter != null) {\n          indexLinesFromGoogleFile(globalDataWriter, file, totalBytes, hiveMode);\n        } else {\n          try (DataWriter dw = new LuceneDataWriter(indexDir)) {\n            indexLinesFromGoogleFile(dw, file, totalBytes, hiveMode);\n          }\n          markIndexAsComplete(indexDir);\n        }\n      }\n    } catch (Exception e) {\n      throw new RuntimeException(\"Could not index \" + file, e);\n    }\n    bytesProcessed.addAndGet(file.length());\n  }\n\n  private void markIndexAsComplete(File directory) throws IOException {\n    try (FileWriter fw = new FileWriter(new File(directory, LT_COMPLETE_MARKER))) {\n      fw.write(new Date().toString());\n    }\n  }\n\n  private boolean isIndexComplete(File directory) {\n    return new File(directory, LT_COMPLETE_MARKER).exists();\n  }\n\n  private void indexLinesFromGoogleFile(DataWriter writer, File inputFile, long totalBytes, boolean hiveMode) throws IOException {\n    float progress = (float)bytesProcessed.get() / totalBytes * 100;\n    System.out.printf(\"==== Working on \" + inputFile + \" (%.2f%%) ====\\n\", progress);\n    try (","sourceCodeStart":141,"sourceCodeEnd":177,"githubUrl":"https://github.com/languagetool-org/languagetool/blob/2e990059ce67d5e2a0f7f7ca5d31160c6709df4b/languagetool-dev/src/main/java/org/languagetool/dev/bigdata/FrequencyIndexCreator.java#L141-L177","documentation":"FrequencyIndexCreator.index wraps the whole indexing of one input file in a try-catch and rethrows any failure as 'Could not index <file>', preserving the cause. It indicates the specific input file could not be converted into the Lucene index, e.g. unreadable or malformed Google n-gram data, or an I/O failure writing the index.","triggerScenarios":"indexLinesFromGoogleFile throwing on a corrupt/truncated/uncompressed .gz file; IOException writing to the index directory; disk full; Lucene writer failing mid-run.","commonSituations":"Incomplete dataset downloads producing truncated gzip files; read-protected files; index directory on a full or read-only filesystem; unexpected file format version.","solutions":["Inspect the wrapped cause (e.getCause()) to find the underlying failure and fix it","Verify the input file is a valid, complete gzip/Google n-gram file (gunzip -t)","Check disk space and write permissions on the index directory","Re-download the corrupt input file and re-run"],"exampleFix":"// diagnosis\ntry { creator.run(inputDir, indexDir); }\ncatch (RuntimeException e) { e.getCause().printStackTrace(); /* real reason */ }\n// fix corrupt input\ngunzip -t file.gz || wget -c <url>/file.gz","handlingStrategy":"try-catch","validationCode":"if (!file.canRead() || file.length() == 0) {\n  throw new IllegalStateException(\"Unreadable/empty input: \" + file);\n}","typeGuard":null,"tryCatchPattern":"try {\n  creator.run(inputDir, indexBaseDir);\n} catch (RuntimeException e) {\n  Throwable cause = e.getCause();\n  log.error(\"Failed file: \" + e.getMessage() + \" reason: \" + cause, cause);\n}","preventionTips":["Validate gzip integrity (gunzip -t) of all input files before indexing","Monitor free disk space in the index directory","Always log e.getCause() to identify the real failure"],"tags":["indexing","io","wrapped-exception"],"backgroundTag":"file-read-failed","analyzedSha":"2e990059ce67d5e2a0f7f7ca5d31160c6709df4b","analyzedAt":"2026-09-06T09:20:17.015Z","contentChangedAt":"2026-09-06T09:20:17.015Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}