dotnet/wpf · error · InvalidOperationException

SR.Format(SR.TokenizerHelperExtraDataEncountered…

Error message

SR.Format(SR.TokenizerHelperExtraDataEncountered, _charIndex, _str)

What it means

TokenizerHelper.LastTokenRequired is called after parsing the final expected token of a formatted string (via Parse) and asserts that the entire input was consumed; any leftover non-whitespace text throws InvalidOperationException(SR.Format(SR.TokenizerHelperExtraDataEncountered, _charIndex, _str)) with the character index and full string. WPF uses this when parsing XAML attribute values such as Point, Size, Color, or matrix strings.

Solutions

Remove any extra tokens/characters so the string contains exactly the expected number of comma/space-separated values for the type (e.g. Point needs exactly 2 numbers).
Check for accidental doubled separators (",," or " ,") and trailing separators before parsing.
Trim and normalize the string (collapse whitespace) before calling Parse.
Wrap Parse in try/catch for InvalidOperationException and surface a friendly message including the offending position.
Use TryParse-style alternatives or pre-validate the token count with str.Split(',') where available.

Example fix

// before
var p = (Point)TypeConverter Point.Parse("10,20,30"); // extra token
// after
string s = "10,20,30";
var parts = s.Split(new[] { ',', ' ' }, StringSplitOptions.RemoveEmptyEntries);
if (parts.Length != 2)
    throw new FormatException($"Point requires exactly 2 values, got {parts.Length}: '{s}'");
var p = Point.Parse(string.Join(",", parts.Take(2)));

Defensive patterns

Strategy: validation

Validate before calling

// ensure exactly the expected number of tokens before parsing
string[] tokens = str.Split(new[] { ',', ' ' }, StringSplitOptions.RemoveEmptyEntries);
if (tokens.Length != expectedTokenCount)
    throw new FormatException($"Expected {expectedTokenCount} values, got {tokens.Length}: '{str}'");
var value = Parse(str);

Type guard

static bool HasExactTokens(string s, int n) =>
    s.Split(new[] { ',', ' ' }, StringSplitOptions.RemoveEmptyEntries).Length == n;

Try / catch

try
{
    var point = Point.Parse("10,20,30");
}
catch (InvalidOperationException ex)
{
    // extra data after the last expected token
    Console.WriteLine($"Bad formatted value: {ex.Message}");
}

Prevention

Verify token counts against the type's expected arity (Point=2, Size=2, Matrix=6, etc.).
Reject doubled or trailing separators before parsing.
Trim/collapse whitespace in user-supplied or concatenated format strings.
Never hand-edit XAML geometry attribute strings without re-parsing them in a test.

When it happens

Trigger: Passing a string with extra trailing content to a Parse method that uses TokenizerHelper — e.g. "1,2 3,4 extra", a doubled comma like "10,,20", or an extra coordinate in a Point/Vector/Matrix string.

Common situations: Hand-edited XAML attribute values; strings built by concatenation that leave a trailing separator; locale-independent parsing of user-entered geometry data with more tokens than the type accepts.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

AI-assisted analysis of dotnet/wpf@81131a70a4 (2026-09-14). Data as JSON: /api/errors/4d053b8230fbad08. Report an issue: GitHub.

Appendix: source

Thrown at src/Microsoft.DotNet.Wpf/src/Shared/MS/Internal/TokenizerHelper.cs:102

        internal string GetCurrentToken()
        {
            // if no current token, return null
            if (_currentTokenIndex < 0)
            {
                return null;
            }

            return _str.Substring(_currentTokenIndex,_currentTokenLength);
        }
        
        /// <summary>
        /// Throws an exception if there is any non-whitespace left un-parsed.
        /// </summary>
        internal void LastTokenRequired()
        {
            if (_charIndex != _strLen)
            {
                throw new System.InvalidOperationException(SR.Format(SR.TokenizerHelperExtraDataEncountered, _charIndex, _str));
            }
        }

        /// <summary>
        /// Advances to the NextToken
        /// </summary>
        /// <returns>true if next token was found, false if at end of string</returns>
        internal bool NextToken()
        {
            return NextToken(false);
        }

        /// <summary>
        /// Advances to the NextToken, throwing an exception if not present
        /// </summary>
        /// <returns>The next token found</returns>
        internal string NextTokenRequired()
        {

View on GitHub (pinned to 81131a70a4)