apache/hadoop · error · IOException

Unexpected: " + type

Error message

Unexpected: " + type

What it means

The join-expression lexer (Parser.Lexer over java.io.StreamTokenizer) accepts identifiers (letters, digits, '_', '$', and '.' inside words), numbers, double-quoted strings, commas, and parentheses. Lexer.next() throws IOException("Unexpected: <char code>") for any other character, where the number is the character code (e.g. 58 for ':'). Note that an unquoted '/' does not throw: StreamTokenizer treats it as a comment character and silently swallows the rest of the line.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/join/Parser.java:176

        case StreamTokenizer.TT_EOF:
        case StreamTokenizer.TT_EOL:
          return null;
        case StreamTokenizer.TT_NUMBER:
          return new NumToken(tok.nval);
        case StreamTokenizer.TT_WORD:
          return new StrToken(TType.IDENT, tok.sval);
        case '"':
          return new StrToken(TType.QUOT, tok.sval);
        default:
          switch (type) {
            case ',':
              return new Token(TType.COMMA);
            case '(':
              return new Token(TType.LPAREN);
            case ')':
              return new Token(TType.RPAREN);
            default:
              throw new IOException("Unexpected: " + type);
          }
      }
    }
  }

  @InterfaceAudience.Public
  @InterfaceStability.Evolving
  public abstract static class Node implements ComposableInputFormat {
    /**
     * Return the node type registered for the particular identifier.
     * By default, this is a CNode for any composite node and a WNode
     * for "wrapped" nodes. User nodes will likely be composite
     * nodes.
     * @see #addIdentifier(java.lang.String, java.lang.Class[], java.lang.Class, java.lang.Class)
     * @see CompositeInputFormat#setFormat(org.apache.hadoop.mapred.JobConf)
     */
    static Node forIdent(String ident) throws IOException {
      try {

View on GitHub (pinned to 2add963021)

Solutions

1. Wrap every path in double quotes: tbl(<class>,\"hdfs://nn/data\")
2. Let CompositeInputFormat.compose() generate and quote paths for you
3. Keep the expression to identifiers, numbers, commas, parens, and double-quoted strings only

Example fix

// before: unquoted URI — ':' is not in the lexer grammar
job.set("mapred.join.expr", "inner(tbl(F,hdfs://nn/a),tbl(F,hdfs://nn/b))");

// after: paths double-quoted (compose() does this for you)
job.set("mapred.join.expr",
    CompositeInputFormat.compose("inner", SequenceFileInputFormat.class,
        "hdfs://nn/a", "hdfs://nn/b"));

String expr = CompositeInputFormat.compose("inner",
    SequenceFileInputFormat.class, "hdfs://nn/a", "hdfs://nn/b");
job.set("mapred.join.expr", expr);
try {
  new CompositeInputFormat<Text>().setFormat(job); // lexes + parses client-side
} catch (IOException e) {
  throw new IllegalArgumentException("join expression contains a character outside the grammar", e);
}

try {
  new CompositeInputFormat<Object>().setFormat(job);
} catch (IOException e) {
  throw new IllegalArgumentException(
      "unexpected character in mapred.join.expr; quote all paths and URIs", e);
}

When it happens

Trigger: Unquoted characters outside the grammar in mapred.join.expr: an unquoted URI tbl(Fmt,hdfs://nn/data) hits ':'; property-style syntax inner(a=b); a single-quoted path 'x'; stray brackets or semicolons from shell editing.

Common situations: Pasting HDFS URIs or globs into the expression without double quotes; hand-editing the expression in job config XML and introducing stray characters; quotes stripped by shell one-liners.

Related errors

• Expected nodetype
• Parse error
• Expected ','
• Expected numtype
• Expected strtype

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b2e0561bc819440e. Report an issue: GitHub.