dotnet/wpf · error · InvalidOperationException

SR.Format(SR.TokenizerHelperEmptyToken, _charIndex, _str)

Error message

SR.Format(SR.TokenizerHelperEmptyToken, _charIndex, _str)

What it means

TokenizerHelper.NextToken throws InvalidOperationException with TokenizerHelperEmptyToken when the token it extracted has length < 1, i.e. two separators appeared with nothing between them. The message includes the character index and full string.

Solutions

  1. Remove empty elements (double separators) from the input string
  2. Filter out null/empty values before joining values into the string
  3. Validate the token list format before calling the converter
  4. Catch InvalidOperationException and report the index from the message

Example fix

// before
pc.Parse("10,20,,30,40");
// after
pc.Parse("10,20,30,40");
Defensive patterns

Strategy: validation

Validate before calling

static bool NoDoubleSeparators(string s, char sep) =>
    !s.Contains(new string(sep, 2));

Try / catch

try { helper.NextToken(false); }
catch (InvalidOperationException ex) { /* empty token at index from message */ }

Prevention

When it happens

Trigger: Parsing a token list containing consecutive separators like '1,,2' or a separator immediately followed by end-of-string, so GetCurrentToken yields an empty token.

Common situations: Hand-edited XAML PointCollection/DoubleCollection strings with doubled commas; programmatically joined values where an element was null/empty.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

AI-assisted analysis of dotnet/wpf@81131a70a4 (2026-09-14). Data as JSON: /api/errors/9e3f37166f02e431. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.DotNet.Wpf/src/Shared/MS/Internal/TokenizerHelper.cs:240

                ++newTokenLength;
            }

            // if quoteCount isn't zero we hit the end of the string
            // before the ending quote
            if (quoteCount > 0)
            {
                throw new System.InvalidOperationException(SR.Format(SR.TokenizerHelperMissingEndQuote, _str));                
            }

            ScanToNextToken(separator); // move so at the start of the nextToken for next call

            // finally made it, update the _currentToken values
            _currentTokenIndex = newTokenIndex;
            _currentTokenLength = newTokenLength;

            if (_currentTokenLength < 1)
            {
                throw new System.InvalidOperationException(SR.Format(SR.TokenizerHelperEmptyToken, _charIndex, _str));
            }

            return true;
        }

        // helper to move the _charIndex to the next token or to the end of the string
        private void ScanToNextToken(char separator)
        {
            // if already at end of the string don't bother
            if (_charIndex < _strLen)
            {
                char currentChar = _str[_charIndex];

                // check that the currentChar is a space or the separator.  If not
                // we have an error. this can happen in the quote case
                // that the char after the quotes string isn't a char.
                if (!(currentChar == separator) &&
                    !Char.IsWhiteSpace(currentChar))

View on GitHub (pinned to 81131a70a4)