{"record":{"id":"1a8f1abb2d809f65","repo":"stride3d/stride","slug":"bad-token-tok","errorCode":null,"errorMessage":"Bad token {tok}","messagePattern":"Bad token (.+?)","errorType":"exception","errorClass":"Exception","httpStatus":null,"severity":"error","filePath":"sources/shaders/Stride.Shaders.Parsers/CppNet/Preprocessor.cs","lineNumber":1943,"sourceCode":"\t\t\t\t\t\treturn tok;\n\t\t\t\t\tif (source.isExpanding(m))\n\t\t\t\t\t\treturn tok;\n\t\t\t\t\tif (macro(m, tok))\n\t\t\t\t\t\tbreak;\n\t\t\t\t\treturn tok;\n\n\t\t\t\tcase Token.P_LINE:\n\t\t\t\t\tif (getFeature(Feature.LINEMARKERS))\n\t\t\t\t\t\treturn tok;\n\t\t\t\t\tbreak;\n\n\t\t\t\tcase Token.INVALID:\n\t\t\t\t\tif (getFeature(Feature.CSYNTAX))\n\t\t\t\t\t\terror(tok, (String)tok.getValue());\n\t\t\t\t\treturn tok;\n\n\t\t\t\tdefault:\n                    throw new Exception(\"Bad token \" + tok);\n\t\t\t\t\t// break;\n\n\t\t\t\tcase Token.HASH:\n\t\t\t\t\ttok = source_token_nonwhite();\n\t\t\t\t\t// (new Exception(\"here\")).printStackTrace();\n\t\t\t\t\tswitch (tok.getType()) {\n\t\t\t\t\t\tcase Token.NL:\n\t\t\t\t\t\t\tgoto BREAK_LEX;\t/* Some code has #\\n */\n\t\t\t\t\t\tcase Token.IDENTIFIER:\n\t\t\t\t\t\t\tbreak;\n\t\t\t\t\t\tdefault:\n\t\t\t\t\t\t\terror(tok,\n\t\t\t\t\t\t\t\t\"Preprocessor directive not a word \" +\n\t\t\t\t\t\t\t\ttok.getText());\n\t\t\t\t\t\t\treturn source_skipline(false);\n\t\t\t\t\t}\n\t\t\t\t\tint\t_ppcmd = ppcmds[tok.getText()];\n\t\t\t\t\tif (_ppcmd == null) {","sourceCodeStart":1925,"sourceCodeEnd":1961,"githubUrl":"https://github.com/stride3d/stride/blob/96fad776d210c221682aac1ccdf4c79dc046fc38/sources/shaders/Stride.Shaders.Parsers/CppNet/Preprocessor.cs#L1925-L1961","documentation":"The main token() loop of the Preprocessor encountered a token type it has no case for, so it throws a generic Exception(\"Bad token \" + tok). This is a fall-through default in the tokenizer state machine and indicates a token kind the preprocessor cannot handle at top level.","triggerScenarios":"A Lexer produces a token type outside the set handled in the switch (e.g. Token.INVALID when CSYNTAX feature handling paths diverge, or a custom lexer emitting unexpected token kinds) — reached via the public Preprocessor token stream API.","commonSituations":"Custom/patched lexers feeding the preprocessor, corrupted or non-standard shader source producing odd tokens, version drift between lexer and preprocessor token enums.","solutions":["Inspect the offending token text in the message and fix the source at that position","Update/align the Lexer and Preprocessor versions so token enums match","Check for binary/corrupted characters in the shader file (encoding issues)","Catch Exception and log tok for diagnosis"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"if (tok.Type == TokenType.INVALID) error(tok.Line, tok.Column, tok.Value as string);","typeGuard":"bool isHandled(TokenType t) => t is not TokenType.INVALID && Enum.IsDefined(t);","tryCatchPattern":"try { foreach (var tok in pp) Process(tok); } catch (Exception e) when (e.Message.StartsWith(\"Bad token\")) { log.Error($\"Unhandled token in source: {e.Message}\"); }","preventionTips":["Keep Lexer and Preprocessor token enums in sync","Sanitize shader files for stray/invalid characters before preprocessing"],"tags":["shader","preprocessor","tokenizer"],"backgroundTag":"internal-invariant-violation","analyzedSha":"96fad776d210c221682aac1ccdf4c79dc046fc38","analyzedAt":"2026-09-14T02:59:31.279Z","contentChangedAt":"2026-09-14T02:59:31.279Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}