golang/go · info

final token

Error message

final token

What it means

ErrFinalToken is a sentinel error a SplitFunc returns to tell the Scanner to stop scanning cleanly, optionally delivering one final token (scan.go:128, handled at scan.go:152-159). Despite living among error sentinels, it is not a failure: when the Scanner sees it, it sets done=true, returns the accompanying token (if non-nil) as the last token, and Scanner.Err() reports nil. It exists to enable early termination or to emit a final empty token that a plain nil-token return cannot express.

Source

Thrown at src/bufio/scan.go:128

}

// Text returns the most recent token generated by a call to [Scanner.Scan]
// as a newly allocated string holding its bytes.
func (s *Scanner) Text() string {
	return string(s.token)
}

// ErrFinalToken is a special sentinel error value. It is intended to be
// returned by a Split function to indicate that the scanning should stop
// with no error. If the token being delivered with this error is not nil,
// the token is the last token.
//
// The value is useful to stop processing early or when it is necessary to
// deliver a final empty token (which is different from a nil token).
// One could achieve the same behavior with a custom error value but
// providing one here is tidier.
// See the emptyFinalToken example for a use of this value.
var ErrFinalToken = errors.New("final token")

// Scan advances the [Scanner] to the next token, which will then be
// available through the [Scanner.Bytes] or [Scanner.Text] method. It returns false when
// there are no more tokens, either by reaching the end of the input or an error.
// After Scan returns false, the [Scanner.Err] method will return any error that
// occurred during scanning, except that if it was [io.EOF], [Scanner.Err]
// will return nil.
// Scan panics if the split function returns too many empty
// tokens without advancing the input. This is a common error mode for
// scanners.
func (s *Scanner) Scan() bool {
	if s.done {
		return false
	}
	s.scanCalled = true
	// Loop until we have a token.
	for {
		// See if we can get a token with what we already have.

View on GitHub (pinned to b6b368adc5)

Solutions

1. If you are seeing this surface as an unexpected error, you likely returned ErrFinalToken by accident from a SplitFunc — return nil error for normal tokens and reserve ErrFinalToken for explicit stop-with-final-token intent.
2. To use it intentionally: return (advance, finalTokenBytes, bufio.ErrFinalToken) and the Scanner will deliver finalTokenBytes once and then stop.
3. To deliver a final empty token specifically, return (0, []byte{}, bufio.ErrFinalToken) — this is the documented use case.
4. Do not wrap or compare with errors.Is unless you understand it short-circuits Scan; check == directly per the source.

Example fix

// intent: stop after a TERMINATOR line, deliver it as the last token
split := func(data []byte, atEOF bool) (int, []byte, error) {
    if bytes.HasPrefix(data, []byte("TERMINATOR\n")) {
        return len("TERMINATOR\n"), data[:len("TERMINATOR\n")], bufio.ErrFinalToken
    }
    // normal line splitting otherwise...
    return ScanLines(data, atEOF)
}

// Return ErrFinalToken intentionally to stop scanning with an optional final token.
stopSplit := func(data []byte, atEOF bool) (int, []byte, error) {
    if /* termination condition */ false {
        return 0, []byte{}, bufio.ErrFinalToken // deliver a final empty token, then stop
    }
    return bufio.ScanLines(data, atEOF)
}

// Detect whether a SplitFunc ever returns ErrFinalToken by wrapping it.
func logsFinalToken(split bufio.SplitFunc) bufio.SplitFunc {
    return func(data []byte, atEOF bool) (int, []byte, error) {
        adv, tok, err := split(data, atEOF)
        if err == bufio.ErrFinalToken {
            log.Println("split signalled final token")
        }
        return adv, tok, err
    }
}

// After Scan returns false, Err() reports nil when ErrFinalToken was used.
for sc.Scan() {
    process(sc.Bytes())
}
if err := sc.Err(); err != nil {
    // ErrFinalToken does NOT appear here; the Scanner consumes it as a stop signal
}

When it happens

Trigger: Triggered only when a SplitFunc deliberately returns ErrFinalToken (scan.go:150-152). Built-in split functions never return it; it is purely an opt-in mechanism for custom splitting logic that knows when to stop before EOF.

Common situations: A custom SplitFunc that wants to stop at a sentinel record (e.g., a terminator line) without scanning the rest. Implementations that must deliver a trailing empty token to signal 'end of stream' to downstream consumers. Early-exit parsers for streaming protocols with an explicit end marker.

Related errors

• bufio.Scanner: token too long
• bufio.Scanner: SplitFunc returns negative advance count
• bufio.Scanner: SplitFunc returns advance count beyond input
• bufio: reader returned negative count from Read
• bufio: writer returned negative count from Write

AI-assisted analysis of golang/go@b6b368adc5 (2026-08-12). Data as JSON: /api/errors/0c2407ce37c16e27. Report an issue: GitHub.