stanfordnlp/CoreNLP · error · RuntimeException

Word ( ) mapped to null

Error message

Word (%s) mapped to null

What it means

IBMArabicEscaper.apply escapes/strips an Arabic token (removing diacritics, normalization markers, clitic annotations) and throws this RuntimeException when the transformation produces an empty string — meaning the whole word vanished during escaping. The message says "mapped to null" because an empty token is treated as unusable output for downstream processing.

Solutions

1. Filter out empty/punctuation-only/diacritic-only tokens before calling apply.
2. Pre-validate input: strip the same characters yourself and drop words that would become empty.
3. If the token is meaningful, keep it unescaped or map it to a placeholder token rather than passing it through.
4. Review the tokenizer settings so such degenerate tokens are never produced upstream.

Example fix

// before
for (String w : tokens) result.add(escaper.apply(w));
// after
for (String w : tokens) {
  if (w == null || w.trim().isEmpty()) continue;
  String escaped = escaper.apply(w); // may still throw for diacritic-only words
  if (!escaped.isEmpty()) result.add(escaped);
}

// java: drop tokens that escaping would empty
static boolean survivesEscaping(String w, boolean annotationsOnly) {
  String stripped = annotationsOnly
      ? IBMArabicEscaper.stripAnnotationsAndClassing(w)
      : IBMArabicEscaper.escapeString(w);
  return stripped != null && !stripped.isEmpty();
}

// java
if (w == null || w.isEmpty()) continue; // skip degenerate input
if (!survivesEscaping(w, escaperIsAnnotationsOnly)) continue;

try {
  escaped = escaper.apply(w);
} catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Word (")) {
    LOG.warn("Token vanished during escaping, skipping: " + w);
    continue;
  } else throw e;
}

When it happens

Trigger: Calling apply(w) (or feeding text through the escaper in an Arabic pipeline) with a word consisting solely of strippable characters — only diacritics/tatweel, or only annotation/punctuation characters when annotationsAndClassingOnly is set.

Common situations: Preprocessing corpora containing tokens made entirely of diacritics or separators; segmented text where a segment reduced to a bare punctuation or diacritic-only token; running raw IBM-style Arabic output with stray markers through the escaper before deduplication.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• this.getClass().getName() + ": Case is presently…
• Arabic does not support feature type: " + feat.toString()
• : Inconsistent u2b/b2u arrays.
• : Parser grammar does not exist
• : No 1best segmentation available

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/e47880c2ee52ebb8. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/arabic/IBMArabicEscaper.java:185

    return newSentence;
  }

  /**
   * Applies escaping to a single word. Interns the escaped string.
   *
   * @param w The word
   * @return The escaped word
   * @throws RuntimeException If a word is nullified (which is really bad for the parser and
   * for MT)
   */
  public String apply(String w) {

    String escapedWord = (annotationsAndClassingOnly) ?
        stripAnnotationsAndClassing(w) : escapeString(w);

    if (escapedWord.isEmpty()) {
      throw new RuntimeException(String.format("Word (%s) mapped to null", w));
    }

    return escapedWord.intern();
  }

  /** This main method preprocesses one-sentence-per-line input, making the
   *  same changes as the Function.  By default it writes the output to files
   *  with the same name as the files passed in on the command line but with
   *  {@code .sent} appended to their names.  If you give the flag
   *  {@code -f} then output is instead sent to stdout.  Input and output
   *  is always in UTF-8.
   *
   *  @param args A list of filenames.  The files must be UTF-8 encoded.
   *  @throws IOException If there are any issues
   */
  public static void main(String[] args) throws IOException {
    IBMArabicEscaper escaper = new IBMArabicEscaper();
    boolean printToStdout = false;

View on GitHub (pinned to 1b7edd19c4)