HandyOrg/HandyControl · error · InvalidOperationException

TokenizerHelperExtraDataEncountered

Error message

TokenizerHelperExtraDataEncountered

What it means

Sentinel validation error in ScanToNextToken (called from NextToken): after finishing a quoted token, the very next character is neither the separator nor whitespace. The quoted string is followed immediately by more data — e.g. '"a"b' — which the tokenizer grammar rejects, because a valid token must be terminated by a separator or whitespace before the next one begins.

Solutions

1. Correct the input format: quote-delimited tokens must be followed by whitespace or the separator, e.g. change '"a"b,c' to '"a" b,c' or 'a b,c'
2. Pre-validate the string with a regex that enforces quote-then-separator boundaries before handing it to TokenizerHelper
3. Catch InvalidOperationException around the parse loop and report the raw string and position so users can fix the malformed entry

When it happens

Trigger: Thrown at src/Shared/HandyControl_Shared/Tools/Helper/TokenizerHelper.cs:172 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of HandyOrg/HandyControl@2c0875ebd6 (2026-09-14). Data as JSON: /api/errors/74cf2f480fc50ddf. Report an issue: GitHub.

Appendix: source

Thrown at src/Shared/HandyControl_Shared/Tools/Helper/TokenizerHelper.cs:172

            throw new InvalidOperationException("TokenizerHelperEmptyToken");
        }

        return true;
    }

    private void ScanToNextToken(char separator)
    {
        // if already at end of the string don't bother
        if (_charIndex >= _strLen) return;

        var currentChar = _str[_charIndex];

        // check that the currentChar is a space or the separator.  If not
        // we have an error. this can happen in the quote case
        // that the char after the quotes string isn't a char.
        if (currentChar != separator && !char.IsWhiteSpace(currentChar))
        {
            throw new InvalidOperationException("TokenizerHelperExtraDataEncountered");
        }

        // loop until hit a character that isn't
        // an argument separator or whitespace.
        // !!!Todo: if more than one argSet throw an exception
        var argSepCount = 0;
        while (_charIndex < _strLen)
        {
            currentChar = _str[_charIndex];

            if (currentChar == separator)
            {
                FoundSeparator = true;
                ++argSepCount;
                _charIndex++;

                if (argSepCount > 1)
                {

View on GitHub (pinned to 2c0875ebd6)