jamiebuilds/the-super-tiny-compiler · error · TypeError

I dont know what this character is: ' + char

Error message

I dont know what this character is: ' + char

What it means

This error is thrown by the tokenizer when it encounters a character that does not match any token pattern: whitespace, semicolon, parenthesis, number, string, or name. It is the tokenizer's catch-all guard meaning the input source contains a character the compiler's grammar does not recognize. The character is appended to the message so you can identify the offending byte.

Source

Thrown at the-super-tiny-compiler.js:533

    if (LETTERS.test(char)) {
      let value = '';

      // Again we're just going to loop through all the letters pushing them to
      // a value.
      while (LETTERS.test(char)) {
        value += char;
        char = input[++current];
      }

      // And pushing that value as a token with the type `name` and continuing.
      tokens.push({ type: 'name', value });

      continue;
    }

    // Finally if we have not matched a character by now, we're going to throw
    // an error and completely exit.
    throw new TypeError('I dont know what this character is: ' + char);
  }

  // Then at the end of our `tokenizer` we simply return the tokens array.
  return tokens;
}

/**
 * ============================================================================
 *                                 ヽ/❀o ل͜ o\ﾉ
 *                                THE PARSER!!!
 * ============================================================================
 */

/**
 * For our parser we're going to take our array of tokens and turn it into an
 * AST.
 *
 *   [{ type: 'paren', value: '(' }, ...]   =>   { type: 'Program', body: [...] }

View on GitHub (pinned to d8d4013045)

Solutions

1. Remove or replace the offending character shown in the message — the compiler only accepts parens, names, numbers, double-quoted strings, whitespace, and semicolons.
2. Strip or sanitize input before tokenizing (e.g. remove commas/operators) if you control the input format.
3. If you need a real JS parser, switch to a full parser such as acorn, espree, or @babel/parser.

Example fix

// before (comma is unsupported)
tokenizer('(add 1, 2)');
// throws TypeError: I dont know what this character is: ,

// after (whitespace-separated arguments)
tokenizer('(add 1 2)');

const VALID = /^[\s();()a-zA-Z0-9"]+$/; // rough grammar surface
function isTokenizable(input) {
  return typeof input === 'string' && !/[^\s();()a-zA-Z0-9"]/.test(input.replace(/"[^"]*"/g, '""'));
}
if (!isTokenizable(src)) throw new Error('Input contains unsupported characters');
const tokens = tokenizer(src);

function isTokenizableSource(input) {
  return typeof input === 'string' && /^[\s();()a-zA-Z0-9"]*$/.test(input.replace(/"[^"]*"/g, '""'));
}

try { tokens = tokenizer(src); } catch (e) { if (e instanceof TypeError && /I dont know what this character is/.test(e.message)) { /* report offending char: e.message.split(': ')[1] */ } else throw e; }

When it happens

Trigger: Calling tokenizer(input) on a string containing characters outside the supported grammar, e.g. commas, operators (+ - * / = < >), brackets [ ] { }, quotes with unusual escaping, comments, or non-ASCII characters. Any one unrecognized char aborts tokenization immediately.

Common situations: Assuming the super-tiny-compiler is a general JS parser and feeding it arbitrary JavaScript (e.g. '(add 1, 2)' with a comma, or 'x = 5' with an equals sign). It only supports the tiny Lisp-like grammar: parens, names, numbers, double-quoted strings, whitespace, and semicolons.

Related errors

• token.type

AI-assisted analysis of jamiebuilds/the-super-tiny-compiler@d8d4013045 (2026-08-28). Data as JSON: /api/errors/edb0e3e732d984b9. Report an issue: GitHub.