antonmedv/fx · error

Unknown token type: %v

Error message

Unknown token type: %v

What it means

Catch-all in Lexer.Next for a token type that is neither Word nor Comment. Under normal operation the tokenizer only emits word, comment, and EOF states, so this fires only on an internal inconsistency or an unhandled token type from a modified/extended tokenizer — effectively an unreachable-by-design guard.

Source

Thrown at internal/shlex/shlex.go:162

func NewLexer(r io.Reader) *Lexer {
	return (*Lexer)(NewTokenizer(r))
}

// Next returns the next word, or an error. If there are no more words,
// the error will be io.EOF.
func (l *Lexer) Next() (string, error) {
	for {
		token, err := (*Tokenizer)(l).Next()
		if err != nil {
			return "", err
		}
		switch token.tokenType {
		case WordToken:
			return token.value, nil
		case CommentToken:
			// skip comments
		default:
			return "", fmt.Errorf("Unknown token type: %v", token.tokenType)
		}
	}
}

// Tokenizer turns an input stream into a sequence of typed tokens
type Tokenizer struct {
	input      bufio.Reader
	classifier tokenClassifier
}

// NewTokenizer creates a new tokenizer from an input stream.
func NewTokenizer(r io.Reader) *Tokenizer {
	input := bufio.NewReader(r)
	classifier := newDefaultClassifier()
	return &Tokenizer{
		input:      *input,
		classifier: classifier}
}

View on GitHub (pinned to 4f31cd3a0c)

Solutions

1. Inspect the token stream for non-word tokens the lexer does not handle
2. Update the switch to handle the reported token type
3. Report as a bug if triggered by ordinary shell-like input

When it happens

Trigger: Thrown at internal/shlex/shlex.go:162 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of antonmedv/fx@4f31cd3a0c (2026-09-02). Data as JSON: /api/errors/6195dabee5f2f82e. Report an issue: GitHub.