stanfordnlp/CoreNLP · error · RuntimeIOException

Bad character encoding

Error message

Bad character encoding

What it means

SpanishTokenizer.main (the command-line tokenizer entry point) wraps its output loop in a try/catch; an UnsupportedEncodingException thrown while constructing the output writer with the requested encoding is rethrown as a RuntimeIOException with the message 'Bad character encoding'. This means the -encoding value given on the command line is not a supported charset name for the JVM.

Solutions

1. Use a standard charset name exactly: 'UTF-8' or 'ISO-8859-1'.
2. Run Charset.availableCharsets() (or check JVM docs) to list charsets supported by your JVM and pick a valid one.
3. Omit the -encoding option to use the platform default encoding.
4. Ensure the file is not read with a mismatched encoding that cascades into writer construction.

Example fix

// before
java ... SpanishTokenizer -encoding utf-16leish -file in.txt
// after
java ... SpanishTokenizer -encoding UTF-8 -file in.txt

String enc = cmdLineEncoding;
if (enc != null && !Charset.isSupported(enc)) {
  throw new IllegalArgumentException("Unsupported encoding: " + enc);
}

try {
  SpanishTokenizer.main(args);
} catch (RuntimeIOException e) {
  if ("Bad character encoding".equals(e.getMessage())) {
    System.err.println("Fix the -encoding flag; use UTF-8 or ISO-8859-1");
  } else throw e;
}

When it happens

Trigger: Running java edu.stanford.nlp.international.spanish.process.SpanishTokenizer with an -encoding option whose value is not a valid charset name (e.g. 'utf8' misspellings unsupported by the JVM, 'ISO-8859-2' unavailable, or an empty/garbled value).

Common situations: Command-line scripts with a typo in the encoding flag value, running on a JVM with a restricted charset provider, or copying an encoding string from a locale that differs from the machine's installed charsets.

Understand the failure class

Background: "is not a valid" / "Invalid ... value" environment variable errors: how libraries validate env vars and what to do when they reject yours — this error's family across 48 libraries.

Related errors

• adjustFinalToken: Unexpected final char: |
• args: treebankPath trainNums testNums
• Bad process cp1252
• Bad serialized file:
• Cannot find or open + sentFileName

AI-assisted analysis of stanfordnlp/CoreNLP@1b7edd19c4 (2026-09-10). Data as JSON: /api/errors/c64cbb1d0bfb96ef. Report an issue: GitHub.

Appendix: source

Thrown at src/edu/stanford/nlp/international/spanish/process/SpanishTokenizer.java:496

            writer.newLine();
            printSpace = false;
          }
        } else {
          String outputToken = toLower ? word.toLowerCase(es) : word;
          if (onePerLine) {
            writer.write(outputToken);
            writer.newLine();
          } else {
            if (printSpace) {
              writer.write(" ");
            }
            writer.write(outputToken);
            printSpace = true;
          }
        }
      }
    } catch (UnsupportedEncodingException e) {
      throw new RuntimeIOException("Bad character encoding", e);
    } catch (IOException e) {
      throw new RuntimeIOException(e);
    }
    long elapsedTime = System.nanoTime() - startTime;
    double linesPerSec = (double) nLines / (elapsedTime / 1e9);
    System.err.printf("Done! Tokenized %d lines (%d tokens) at %.2f lines/sec%n", nLines, nTokens, linesPerSec);
  } // end main()

}

View on GitHub (pinned to 1b7edd19c4)