{"record":{"id":"74cf2f480fc50ddf","repo":"HandyOrg/HandyControl","slug":"tokenizerhelperextradataencountered","errorCode":null,"errorMessage":"TokenizerHelperExtraDataEncountered","messagePattern":"TokenizerHelperExtraDataEncountered","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"src/Shared/HandyControl_Shared/Tools/Helper/TokenizerHelper.cs","lineNumber":172,"sourceCode":"            throw new InvalidOperationException(\"TokenizerHelperEmptyToken\");\n        }\n\n        return true;\n    }\n\n    private void ScanToNextToken(char separator)\n    {\n        // if already at end of the string don't bother\n        if (_charIndex >= _strLen) return;\n\n        var currentChar = _str[_charIndex];\n\n        // check that the currentChar is a space or the separator.  If not\n        // we have an error. this can happen in the quote case\n        // that the char after the quotes string isn't a char.\n        if (currentChar != separator && !char.IsWhiteSpace(currentChar))\n        {\n            throw new InvalidOperationException(\"TokenizerHelperExtraDataEncountered\");\n        }\n\n        // loop until hit a character that isn't\n        // an argument separator or whitespace.\n        // !!!Todo: if more than one argSet throw an exception\n        var argSepCount = 0;\n        while (_charIndex < _strLen)\n        {\n            currentChar = _str[_charIndex];\n\n            if (currentChar == separator)\n            {\n                FoundSeparator = true;\n                ++argSepCount;\n                _charIndex++;\n\n                if (argSepCount > 1)\n                {","sourceCodeStart":154,"sourceCodeEnd":190,"githubUrl":"https://github.com/HandyOrg/HandyControl/blob/2c0875ebd67326e0c67282967e3e809c69282fee/src/Shared/HandyControl_Shared/Tools/Helper/TokenizerHelper.cs#L154-L190","documentation":"Sentinel validation error in ScanToNextToken (called from NextToken): after finishing a quoted token, the very next character is neither the separator nor whitespace. The quoted string is followed immediately by more data — e.g. '\"a\"b' — which the tokenizer grammar rejects, because a valid token must be terminated by a separator or whitespace before the next one begins.","triggerScenarios":"Thrown at src/Shared/HandyControl_Shared/Tools/Helper/TokenizerHelper.cs:172 when the library encounters an invalid state.","commonSituations":"See trigger scenarios.","solutions":["Correct the input format: quote-delimited tokens must be followed by whitespace or the separator, e.g. change '\"a\"b,c' to '\"a\" b,c' or 'a b,c'","Pre-validate the string with a regex that enforces quote-then-separator boundaries before handing it to TokenizerHelper","Catch InvalidOperationException around the parse loop and report the raw string and position so users can fix the malformed entry"],"exampleFix":null,"handlingStrategy":"validation","validationCode":null,"typeGuard":null,"tryCatchPattern":null,"preventionTips":[],"tags":[],"backgroundTag":null,"analyzedSha":"2c0875ebd67326e0c67282967e3e809c69282fee","analyzedAt":"2026-09-14T14:45:29.754Z","contentChangedAt":"2026-09-14T14:45:29.754Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}