stanfordnlp/CoreNLP · error · IllegalArgumentException

eolString cannot be null

Error message

eolString cannot be null

What it means

TokenizerAdapter.setEolString enforces a non-null end-of-line string. A null eolString would make it impossible to distinguish line ends from end-of-file during tokenization, so null is rejected with IllegalArgumentException.

Solutions

1. Pass an explicit string such as "\n" instead of null
2. Default the value: eolString = maybeNull != null ? maybeNull : "\n"
3. If the value comes from properties, check containsKey before reading

Example fix

// before
tok.setEolString(props.getProperty("eolString"));
// after
tok.setEolString(props.getProperty("eolString", "\n"));

String eol = props.getProperty("eolString");
if (eol == null) eol = "\n";

try {
  tokenizerAdapter.setEolString(eol);
} catch (IllegalArgumentException e) {
  tokenizerAdapter.setEolString("\n");
}

When it happens

Trigger: Calling setEolString(null) on a TokenizerAdapter (e.g. one wrapping a WhitespaceTokenizer or Lexer) — often by passing an unset config variable or a null result of getProperty().

Common situations: Building a tokenizer from properties where the 'eolString' option is absent; wiring a null through factory code that should have defaulted to "\n".

Related errors

• adjustFinalToken: Unexpected final char: |
• Cannot make a Lemmatize with no nodeName
• Cannot make an EditNode with no nodeName
• Could not load tokenizer factory
• Expected HasOffsets from the DocumentPreprocessor

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/89813be61747f62f. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/process/TokenizerAdapter.java:75

      }
    } catch (IOException ioe) {
      // do nothing, return null
      return null;
    }
  }


  /**
   * Set the <code>String</code> returned when the inner tokenizer
   * returns an end-of-line token.  This will only happen if the
   * inner tokenizer has been set to <code>eolIsSignificant(true)</code>.
   *
   * @param eolString The String used to represent eol.  It is not allowed
   *                  to be <code>null</code> (which would confuse line ends and file end)
   */
  public void setEolString(String eolString) {
    if (eolString == null) {
      throw new IllegalArgumentException("eolString cannot be null");
    }
    this.eolString = eolString;
  }


  /**
   * Say whether the <code>String</code> is the end-of-line token for
   * this tokenizer.
   *
   * @param str The String being tested
   * @return Whether it is the end-of-line token
   */
  public boolean isEol(String str) {
    return eolString.equals(str);
  }

}

View on GitHub (pinned to 1b7edd19c4)