What causes "Unknown token in (line )%n" in stanfordnlp/CoreNLP?

Open the config file at the reported line number and fix or remove the unknown token

stanfordnlp/CoreNLP · error · IllegalArgumentException

Unknown token in (line )%n

Error message

Unknown token in  (line )%n

What it means

ConfigParser.parse() reads a whitespace/section-delimited config file for dataset parsing parameters. When a line's token matches none of the known keywords, it reports 'Unknown token' with the file and line number and throws IllegalArgumentException, aborting config parsing.

Solutions

Open the config file at the reported line number and fix or remove the unknown token
Check the spelling against valid tokens for your CoreNLP version (option names change between releases)
Remove options copied from incompatible tool versions or move them to the correct section
Quote/split arguments correctly so stray characters do not form bogus tokens

Example fix

// before (config file)
dataset train
tokenzie_per_line true   // typo
// after
dataset train
tokenize_per_line true

Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-scan config lines against a whitelist of known tokens
Set<String> known = Set.of("dataset", "tokenize_per_line", ...);
for (String line : Files.readAllLines(configFile)) {
  String tok = line.trim().split("\\s+")[0];
  if (!tok.isEmpty() && !known.contains(tok)) log.warn("Unknown config token: " + tok);
}

Try / catch

try {
  parser.parse();
} catch (IllegalArgumentException e) {
  System.err.println(e.getMessage()); // includes file and line number
  // fix the config at the reported line, then retry or abort
}

Prevention

Copy config examples only from the docs matching your CoreNLP version
Lint config files with a whitelist before running the pipeline
Avoid hand-editing configs in editors with autocorrect/whitespace transforms

When it happens

Trigger: A config file contains an unrecognized keyword/token (typo, option from a different tool version, or a token meant for a section the parser does not know), encountered while parse() scans lines via main().

Common situations: Copy-pasting configuration examples from documentation for a different CoreNLP version; typos like 'tokenzie' or custom options; shell-quoting issues that merge tokens; editing the config by hand.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/510c929cf2d1f9af. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/trees/treebank/ConfigParser.java:189

            String actualParam = tokens[0].trim();
            String paramValue = tokens[1].trim();
            if(paramTemplate.second != null) {
              paramToken = paramTemplate.second.matcher(paramValue);
              if(paramToken.matches()) {
                paramsForDataset.setProperty(actualParam, paramValue);
              } else {
                System.err.printf("%s: Skipping illegal parameter value in %s (line %d)%n", this.getClass().getName(), configFile,reader.getLineNumber());
                break;
              }
            } else {
              paramsForDataset.setProperty(actualParam, paramValue);
            }
          }
        }
        if (!matched) {
          String error = this.getClass().getName() + ": Unknown token in " + configFile + " (line " + reader.getLineNumber() + ")%n";
          System.err.printf(error);
          throw new IllegalArgumentException(error);
        }
      }

      if(paramsForDataset != null) datasetList.add(paramsForDataset);

      reader.close();

    } catch (FileNotFoundException e) {
      System.err.printf("%s: Cannot open file %s%n", this.getClass().getName(), configFile);
    } catch (IOException e) {
      System.err.printf("%s: Error reading %s (line %d)%n", this.getClass().getName(), configFile, lineNum);
    }
  }

  @Override
  public String toString() {
    final int numDatasets = datasetList.size();
    StringBuilder sb = new StringBuilder(String.format("Loaded %d datasets: %n",numDatasets));

View on GitHub (pinned to 1b7edd19c4)