LykosAI/StabilityMatrix · error · PromptSyntaxError

Invalid Token

Error message

Invalid Token

What it means

The prompt tokenizer (Sublime-syntax based) flags malformed extra-network syntax with scopes like invalid.illegal; GetExtraNetworks inspects each token and throws PromptSyntaxError with the token's start/end offsets when found. It is a parse-time validation error pointing at the offending text region in the prompt.

Solutions

1. Use the StartIndex/EndIndex from PromptSyntaxError to locate and fix the malformed token in the prompt string.
2. Ensure every extra-network tag has balanced < > and correct <type:name:weight> form.
3. Run ValidateExtraNetworks() on user input before submitting inference.
4. Escape or remove characters that the syntax definition marks illegal.

Example fix

// before
prompt = "a photo <lora:detail_tuned:0.8";   // missing '>'
// after
prompt = "a photo <lora:detail_tuned:0.8>";

// pre-check prompt text for balanced extra-network tags
foreach (var tag in Regex.Matches(text, "<[^>]*$"))
    throw new FormatException("Unclosed extra-network tag");

bool PromptWellFormed(string text) =>
    text.Count(c => c == '<') == text.Count(c => c == '>');

try { prompt.Process(tokenizersPath); }
catch (PromptSyntaxError ex)
{
    editor.Select(ex.StartIndex, ex.EndIndex - ex.StartIndex);
    ShowSyntaxHint("Invalid token at selection");
}

When it happens

Trigger: A prompt token contains invalid extra-network syntax — e.g. unbalanced brackets like <lora:model or stray < > — causing the tokenizer to mark scopes containing invalid.illegal.

Common situations: Hand-edited prompts with truncated tags; copy-pasted prompts with corrupted angle brackets; programmatic prompt assembly missing a closing bracket; wildcard/escaped characters confusing the tokenizer.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

• Authentication and authorization failures — expired tokens, bad credentials, and missing scopes.

Related errors

• Fields line not found
• Unable to locate starting marker of last line
• Length cannot be null when latentType is Hunyuan
• Model file name must contain a valid file name.
• Prompt extensions not installed

AI-assisted analysis of LykosAI/StabilityMatrix@af93d6ef57 (2026-09-12). Data as JSON: /api/errors/6471c05df5facfa1. Report an issue: GitHub.

Appendix: source

Thrown at StabilityMatrix.Avalonia/Models/Inference/Prompt.cs:133

        using var tokens = TokenizeResult.Tokens.Cast<IToken>().GetEnumerator();

        // Maintain both token and text stacks for validation
        var outputTokens = new Stack<IToken>();
        var outputText = new Stack<string>();
        var wildcardStack = new Stack<StringBuilder>();

        // Store extra networks
        var promptExtraNetworks = new List<PromptExtraNetwork>();

        while (tokens.MoveNext())
        {
            var currentToken = tokens.Current;

            // For any invalid syntax, throw
            if (currentToken.Scopes.Any(s => s.Contains("invalid.illegal")))
            {
                // Generic
                throw new PromptSyntaxError(
                    "Invalid Token",
                    currentToken.StartIndex,
                    GetSafeEndIndex(currentToken.EndIndex)
                );
            }

            // Comments - ignore
            if (currentToken.Scopes.Any(s => s.Contains("comment.line")))
            {
                continue;
            }

            // Handle wildcard start
            if (
                processWildcards
                && currentToken.Scopes.Contains("punctuation.definition.wildcard.begin.prompt")
            )
            {

View on GitHub (pinned to af93d6ef57)