languagetool-org/languagetool · error · UnsupportedOperationException

Too many values (" + valueCount + "), the storage needs adju

Error message

Too many values (" + valueCount + "), the storage needs adjusting

What it means

Capacity guard in the ManualTagger binary storage builder: the total number of lemma+POS values across the whole manual tagging dictionary exceeds MAX_OFFSET, the largest value encodable in the bit-shifted offset field. This is a hard internal data-size limit of the packed String[] representation, hit only by pathologically large manual dictionaries.

Source

Thrown at languagetool-core/src/main/java/org/languagetool/tagging/ManualTagger.java:69

  private static final int MAX_OFFSET = (1 << 32 - OFFSET_SHIFT) - 1;
  private static final int ENTRY_SIZE = 2;
  private final String[] data;

  /** A map from inflected forms to encoded lemma+POS pair offsets in {@link #data} */
  private final Object2IntMap<String> map;

  public ManualTagger(InputStream inputStream) throws IOException {
    this(inputStream, false);
  }

  public ManualTagger(InputStream inputStream, boolean internTags) throws IOException {
    Map<String, List<TaggedWord>> mapping = loadMapping(inputStream, internTags);
    map = new Object2IntOpenHashMap<>(mapping.size());
    int valueCount = mapping.values().stream().mapToInt(v -> v.size()).sum();
    int firstIndex = ENTRY_SIZE; // skip an entry, as 0 means an absent value in TObjectIntHashMap
    data = new String[valueCount * ENTRY_SIZE + firstIndex];
    if (valueCount > MAX_OFFSET) {
      throw new UnsupportedOperationException("Too many values (" + valueCount + "), the storage needs adjusting");
    }
    int index = firstIndex;
    for (Map.Entry<String, List<TaggedWord>> entry : mapping.entrySet()) {
      List<TaggedWord> value = entry.getValue();
      if (value.size() > MAX_LENGTH) {
        throw new UnsupportedOperationException(
          "Too many lemmas (" + value.size() + " > " + MAX_LENGTH + " for " + entry.getKey() + "), the storage needs adjusting");
      }
      map.put(entry.getKey(), ((index / ENTRY_SIZE) << OFFSET_SHIFT) | value.size());
      for (TaggedWord tw : value) {
        data[index++] = intern(tw.getLemma());
        data[index++] = intern(tw.getPosTag());
      }
    }
  }

  private static Map<String, List<TaggedWord>> loadMapping(InputStream inputStream, boolean internTags) throws IOException {
    Map<String, List<TaggedWord>> map = new HashMap<>();

View on GitHub (pinned to 2e990059ce)

Solutions

1. Split the manual additions file into smaller dictionaries or reduce its number of entries
2. Increase OFFSET_SHIFT/MAX_OFFSET in ManualTagger and rebuild to widen the offset storage

When it happens

Trigger: Thrown at languagetool-core/src/main/java/org/languagetool/tagging/ManualTagger.java:69 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of languagetool-org/languagetool@2e990059ce (2026-09-06). Data as JSON: /api/errors/c9ab7a382fcecd2f. Report an issue: GitHub.