{"record":{"id":"e47880c2ee52ebb8","repo":"stanfordnlp/CoreNLP","slug":"word-s-mapped-to-null","errorCode":null,"errorMessage":"Word (%s) mapped to null","messagePattern":"Word \\((.+?)\\) mapped to null","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"src/edu/stanford/nlp/international/arabic/IBMArabicEscaper.java","lineNumber":185,"sourceCode":"\n    return newSentence;\n  }\n\n  /**\n   * Applies escaping to a single word. Interns the escaped string.\n   *\n   * @param w The word\n   * @return The escaped word\n   * @throws RuntimeException If a word is nullified (which is really bad for the parser and\n   * for MT)\n   */\n  public String apply(String w) {\n\n    String escapedWord = (annotationsAndClassingOnly) ?\n        stripAnnotationsAndClassing(w) : escapeString(w);\n\n    if (escapedWord.isEmpty()) {\n      throw new RuntimeException(String.format(\"Word (%s) mapped to null\", w));\n    }\n\n    return escapedWord.intern();\n  }\n\n  /** This main method preprocesses one-sentence-per-line input, making the\n   *  same changes as the Function.  By default it writes the output to files\n   *  with the same name as the files passed in on the command line but with\n   *  {@code .sent} appended to their names.  If you give the flag\n   *  {@code -f} then output is instead sent to stdout.  Input and output\n   *  is always in UTF-8.\n   *\n   *  @param args A list of filenames.  The files must be UTF-8 encoded.\n   *  @throws IOException If there are any issues\n   */\n  public static void main(String[] args) throws IOException {\n    IBMArabicEscaper escaper = new IBMArabicEscaper();\n    boolean printToStdout = false;","sourceCodeStart":167,"sourceCodeEnd":203,"githubUrl":"https://github.com/stanfordnlp/CoreNLP/blob/1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a/src/edu/stanford/nlp/international/arabic/IBMArabicEscaper.java#L167-L203","documentation":"IBMArabicEscaper.apply escapes/strips an Arabic token (removing diacritics, normalization markers, clitic annotations) and throws this RuntimeException when the transformation produces an empty string — meaning the whole word vanished during escaping. The message says \"mapped to null\" because an empty token is treated as unusable output for downstream processing.","triggerScenarios":"Calling apply(w) (or feeding text through the escaper in an Arabic pipeline) with a word consisting solely of strippable characters — only diacritics/tatweel, or only annotation/punctuation characters when annotationsAndClassingOnly is set.","commonSituations":"Preprocessing corpora containing tokens made entirely of diacritics or separators; segmented text where a segment reduced to a bare punctuation or diacritic-only token; running raw IBM-style Arabic output with stray markers through the escaper before deduplication.","solutions":["Filter out empty/punctuation-only/diacritic-only tokens before calling apply.","Pre-validate input: strip the same characters yourself and drop words that would become empty.","If the token is meaningful, keep it unescaped or map it to a placeholder token rather than passing it through.","Review the tokenizer settings so such degenerate tokens are never produced upstream."],"exampleFix":"// before\nfor (String w : tokens) result.add(escaper.apply(w));\n// after\nfor (String w : tokens) {\n  if (w == null || w.trim().isEmpty()) continue;\n  String escaped = escaper.apply(w); // may still throw for diacritic-only words\n  if (!escaped.isEmpty()) result.add(escaped);\n}","handlingStrategy":"validation","validationCode":"// java: drop tokens that escaping would empty\nstatic boolean survivesEscaping(String w, boolean annotationsOnly) {\n  String stripped = annotationsOnly\n      ? IBMArabicEscaper.stripAnnotationsAndClassing(w)\n      : IBMArabicEscaper.escapeString(w);\n  return stripped != null && !stripped.isEmpty();\n}","typeGuard":"// java\nif (w == null || w.isEmpty()) continue; // skip degenerate input\nif (!survivesEscaping(w, escaperIsAnnotationsOnly)) continue;","tryCatchPattern":"try {\n  escaped = escaper.apply(w);\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().startsWith(\"Word (\")) {\n    LOG.warn(\"Token vanished during escaping, skipping: \" + w);\n    continue;\n  } else throw e;\n}","preventionTips":["Filter diacritic-only and punctuation-only tokens before preprocessing.","Log-and-skip rather than crash when corpora contain degenerate tokens.","Tune upstream tokenization so pure-marker tokens are never emitted."],"tags":["java","arabic","preprocessing","empty-token"],"backgroundTag":"empty-required-field","analyzedSha":"1b7edd19c4d0d7b1f13a2591425b9b60a0b1af7a","analyzedAt":"2026-09-10T02:24:07.274Z","contentChangedAt":"2026-09-10T02:24:07.274Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}