jamiebuilds/the-super-tiny-compiler · error · TypeError

I dont know what this character is: ' + char

Error message

I dont know what this character is: ' + char

What it means

This error is thrown by the tokenizer when it encounters a character that does not match any token pattern: whitespace, semicolon, parenthesis, number, string, or name. It is the tokenizer's catch-all guard meaning the input source contains a character the compiler's grammar does not recognize. The character is appended to the message so you can identify the offending byte.

Solutions

1. Remove or replace the offending character shown in the message — the compiler only accepts parens, names, numbers, double-quoted strings, whitespace, and semicolons.
2. Strip or sanitize input before tokenizing (e.g. remove commas/operators) if you control the input format.
3. If you need a real JS parser, switch to a full parser such as acorn, espree, or @babel/parser.

Example fix

// before (comma is unsupported)
tokenizer('(add 1, 2)');
// throws TypeError: I dont know what this character is: ,

// after (whitespace-separated arguments)
tokenizer('(add 1 2)');

const VALID = /^[\s();()a-zA-Z0-9"]+$/; // rough grammar surface
function isTokenizable(input) {
  return typeof input === 'string' && !/[^\s();()a-zA-Z0-9"]/.test(input.replace(/"[^"]*"/g, '""'));
}
if (!isTokenizable(src)) throw new Error('Input contains unsupported characters');
const tokens = tokenizer(src);

function isTokenizableSource(input) {
  return typeof input === 'string' && /^[\s();()a-zA-Z0-9"]*$/.test(input.replace(/"[^"]*"/g, '""'));
}

try { tokens = tokenizer(src); } catch (e) { if (e instanceof TypeError && /I dont know what this character is/.test(e.message)) { /* report offending char: e.message.split(': ')[1] */ } else throw e; }

When it happens

Trigger: Calling tokenizer(input) on a string containing characters outside the supported grammar, e.g. commas, operators (+ - * / = < >), brackets [ ] { }, quotes with unusual escaping, comments, or non-ASCII characters. Any one unrecognized char aborts tokenization immediately.

Common situations: Assuming the super-tiny-compiler is a general JS parser and feeding it arbitrary JavaScript (e.g. '(add 1, 2)' with a comma, or 'x = 5' with an equals sign). It only supports the tiny Lisp-like grammar: parens, names, numbers, double-quoted strings, whitespace, and semicolons.

Related errors

• token.type

AI-assisted analysis of jamiebuilds/the-super-tiny-compiler@d8d4013045 (2026-08-28). Data as JSON: /api/errors/edb0e3e732d984b9. Report an issue: GitHub.

Appendix: source

Thrown at the-super-tiny-compiler.js:533

    if (LETTERS.test(char)) {
      let value = '';

      // Again we're just going to loop through all the letters pushing them to
      // a value.
      while (LETTERS.test(char)) {
        value += char;
        char = input[++current];
      }

      // And pushing that value as a token with the type `name` and continuing.
      tokens.push({ type: 'name', value });

      continue;
    }

    // Finally if we have not matched a character by now, we're going to throw
    // an error and completely exit.
    throw new TypeError('I dont know what this character is: ' + char);
  }

  // Then at the end of our `tokenizer` we simply return the tokens array.
  return tokens;
}

/**
 * ============================================================================
 *                                 ヽ/❀o ل͜ o\ﾉ
 *                                THE PARSER!!!
 * ============================================================================
 */

/**
 * For our parser we're going to take our array of tokens and turn it into an
 * AST.
 *
 *   [{ type: 'paren', value: '(' }, ...]   =>   { type: 'Program', body: [...] }

View on GitHub (pinned to d8d4013045)