languagetool-org/languagetool · error

Could not index

Error message

Could not index 

What it means

FrequencyIndexCreator.index wraps the whole indexing of one input file in a try-catch and rethrows any failure as 'Could not index <file>', preserving the cause. It indicates the specific input file could not be converted into the Lucene index, e.g. unreadable or malformed Google n-gram data, or an I/O failure writing the index.

Source

Thrown at languagetool-dev/src/main/java/org/languagetool/dev/bigdata/FrequencyIndexCreator.java:159

          indexLinesFromGoogleFile(globalDataWriter, file, totalBytes, hiveMode);
        } else {
          try (DataWriter dw = new TextDataWriter(indexDir)) {
            indexLinesFromGoogleFile(dw, file, totalBytes, hiveMode);
          }
          markIndexAsComplete(indexDir);
        }
      } else {
        if (globalDataWriter != null) {
          indexLinesFromGoogleFile(globalDataWriter, file, totalBytes, hiveMode);
        } else {
          try (DataWriter dw = new LuceneDataWriter(indexDir)) {
            indexLinesFromGoogleFile(dw, file, totalBytes, hiveMode);
          }
          markIndexAsComplete(indexDir);
        }
      }
    } catch (Exception e) {
      throw new RuntimeException("Could not index " + file, e);
    }
    bytesProcessed.addAndGet(file.length());
  }

  private void markIndexAsComplete(File directory) throws IOException {
    try (FileWriter fw = new FileWriter(new File(directory, LT_COMPLETE_MARKER))) {
      fw.write(new Date().toString());
    }
  }

  private boolean isIndexComplete(File directory) {
    return new File(directory, LT_COMPLETE_MARKER).exists();
  }

  private void indexLinesFromGoogleFile(DataWriter writer, File inputFile, long totalBytes, boolean hiveMode) throws IOException {
    float progress = (float)bytesProcessed.get() / totalBytes * 100;
    System.out.printf("==== Working on " + inputFile + " (%.2f%%) ====\n", progress);
    try (

View on GitHub (pinned to 2e990059ce)

Solutions

1. Inspect the wrapped cause (e.getCause()) to find the underlying failure and fix it
2. Verify the input file is a valid, complete gzip/Google n-gram file (gunzip -t)
3. Check disk space and write permissions on the index directory
4. Re-download the corrupt input file and re-run

Example fix

// diagnosis
try { creator.run(inputDir, indexDir); }
catch (RuntimeException e) { e.getCause().printStackTrace(); /* real reason */ }
// fix corrupt input
gunzip -t file.gz || wget -c <url>/file.gz

if (!file.canRead() || file.length() == 0) {
  throw new IllegalStateException("Unreadable/empty input: " + file);
}

try {
  creator.run(inputDir, indexBaseDir);
} catch (RuntimeException e) {
  Throwable cause = e.getCause();
  log.error("Failed file: " + e.getMessage() + " reason: " + cause, cause);
}

When it happens

Trigger: indexLinesFromGoogleFile throwing on a corrupt/truncated/uncompressed .gz file; IOException writing to the index directory; disk full; Lucene writer failing mid-run.

Common situations: Incomplete dataset downloads producing truncated gzip files; read-protected files; index directory on a full or read-only filesystem; unexpected file format version.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

• Could not get rules of language <language>
• Could not load remote rules.
• Could not load coherency data from " + path
• Cannot load or parse input stream of '${filename}'
• Error analyzing sentence: '${sentence}'

AI-assisted analysis of languagetool-org/languagetool@2e990059ce (2026-09-06). Data as JSON: /api/errors/4b8125519d20f231. Report an issue: GitHub.