HandyOrg/HandyControl · error · InvalidOperationException

TokenizerHelperEmptyToken

Error message

TokenizerHelperEmptyToken

What it means

Sentinel validation error at the end of NextToken: tokenization completed but produced a zero-length token (_currentTokenLength < 1). This happens when the input has leading/consecutive separators or whitespace-only content where a token was expected — the helper treats an empty token as malformed input and throws instead of returning an empty string.

Solutions

1. Sanitize the input string before tokenizing: trim leading/trailing separators and whitespace, and collapse repeated separators so no empty tokens can be produced
2. Skip empty positions in the consuming loop: call NextToken only while LastSeparator/position indicates real content remains, or use a tokenizer API variant that skips empties
3. Wrap NextToken in try/catch for InvalidOperationException and abort parsing with a descriptive error naming the offending input

When it happens

Trigger: Thrown at src/Shared/HandyControl_Shared/Tools/Helper/TokenizerHelper.cs:154 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of HandyOrg/HandyControl@2c0875ebd6 (2026-09-14). Data as JSON: /api/errors/30d714497f60c10b. Report an issue: GitHub.

Appendix: source

Thrown at src/Shared/HandyControl_Shared/Tools/Helper/TokenizerHelper.cs:154

            ++newTokenLength;
        }

        // if quoteCount isn't zero we hit the end of the string
        // before the ending quote
        if (quoteCount > 0)
        {
            throw new InvalidOperationException("TokenizerHelperMissingEndQuote");
        }

        ScanToNextToken(separator); // move so at the start of the nextToken for next call

        // finally made it, update the _currentToken values
        _currentTokenIndex = newTokenIndex;
        _currentTokenLength = newTokenLength;

        if (_currentTokenLength < 1)
        {
            throw new InvalidOperationException("TokenizerHelperEmptyToken");
        }

        return true;
    }

    private void ScanToNextToken(char separator)
    {
        // if already at end of the string don't bother
        if (_charIndex >= _strLen) return;

        var currentChar = _str[_charIndex];

        // check that the currentChar is a space or the separator.  If not
        // we have an error. this can happen in the quote case
        // that the char after the quotes string isn't a char.
        if (currentChar != separator && !char.IsWhiteSpace(currentChar))
        {
            throw new InvalidOperationException("TokenizerHelperExtraDataEncountered");

View on GitHub (pinned to 2c0875ebd6)