What causes "bufio.Scanner: token too long" in golang/go?

Call scanner.Buffer(make([]byte, 0, 64*1024), ) before scanning to raise maxTokenSize to fit your largest expected token.

golang/go · error

bufio.Scanner: token too long

Error message

bufio.Scanner: token too long

What it means

bufio.Scanner sets ErrTooLong when a token (e.g., a single line) exceeds the scanner's maxTokenSize, which defaults to MaxScanTokenSize = 64 * 1024 bytes (scan.go:82, scan.go:200-202). The scanner doubles its internal buffer until it reaches maxTokenSize; if a complete token still does not fit, scanning halts and Scanner.Err() returns this error. It is a returned error (not a panic), surfaced after Scan() returns false.

Source

Thrown at src/bufio/scan.go:71

// immediately stops the scanning.
//
// Otherwise, the [Scanner] advances the input. If the token is not nil,
// the [Scanner] returns it to the user. If the token is nil, the
// Scanner reads more data and continues scanning; if there is no more
// data--if atEOF was true--the [Scanner] returns. If the data does not
// yet hold a complete token, for instance if it has no newline while
// scanning lines, a [SplitFunc] can return (0, nil, nil) to signal the
// [Scanner] to read more data into the slice and try again with a
// longer slice starting at the same point in the input.
//
// The function is never called with an empty data slice unless atEOF
// is true. If atEOF is true, however, data may be non-empty and,
// as always, holds unprocessed text.
type SplitFunc func(data []byte, atEOF bool) (advance int, token []byte, err error)

// Errors returned by Scanner.
var (
	ErrTooLong         = errors.New("bufio.Scanner: token too long")
	ErrNegativeAdvance = errors.New("bufio.Scanner: SplitFunc returns negative advance count")
	ErrAdvanceTooFar   = errors.New("bufio.Scanner: SplitFunc returns advance count beyond input")
	ErrBadReadCount    = errors.New("bufio.Scanner: Read returned impossible count")
)

const (
	// MaxScanTokenSize is the maximum size used to buffer a token
	// unless the user provides an explicit buffer with [Scanner.Buffer].
	// The actual maximum token size may be smaller as the buffer
	// may need to include, for instance, a newline.
	MaxScanTokenSize = 64 * 1024

	startBufSize = 4096 // Size of initial allocation for buffer.
)

// NewScanner returns a new [Scanner] to read from r.
// The split function defaults to [ScanLines].
func NewScanner(r io.Reader) *Scanner {

View on GitHub (pinned to b6b368adc5)

Solutions

Call scanner.Buffer(make([]byte, 0, 64*1024), <maxBytes>) before scanning to raise maxTokenSize to fit your largest expected token.
Switch to bufio.Reader.ReadString('\n') or ReadBytes for line-oriented parsing with no hard token cap.
If the input genuinely has no delimiter, pre-split it (e.g., stream chunking) before scanning.
Lower maxTokenSize deliberately only if you want to reject oversized tokens as a safety bound.

Example fix

// before — default 64 KB cap, panics on long lines
scanner := bufio.NewReader(file)
// using NewScanner with default max
sc := bufio.NewScanner(file)
for sc.Scan() {} // ErrTooLong on >64KB line

// after — raise the cap explicitly
sc := bufio.NewScanner(file)
sc.Buffer(make([]byte, 0, 1024*1024), 10*1024*1024) // up to 10 MB tokens
for sc.Scan() {
    line := sc.Text()
}

Defensive patterns

Strategy: validation

Validate before calling

// Configure the scanner buffer BEFORE the first Scan call.
sc := bufio.NewScanner(r)
sc.Buffer(make([]byte, 0, 64*1024), maxExpectedTokenBytes)
for sc.Scan() {
    _ = sc.Bytes()
}
if err := sc.Err(); err != nil {
    if err == bufio.ErrTooLong {
        // token exceeded maxExpectedTokenBytes; handle oversized input
    }
}

Try / catch

// After Scan returns false, distinguish ErrTooLong from other failures.
for sc.Scan() {
    process(sc.Bytes())
}
if err := sc.Err(); err != nil {
    if errors.Is(err, bufio.ErrTooLong) {
        // switch to a streaming reader or raise the cap
    } else {
        return err
    }
}

Prevention

Always call Scanner.Buffer with an explicit max when input token size is unbounded.
For line parsing of arbitrary-length input, prefer bufio.Reader.ReadString over Scanner.
Document the chosen max token size where the scanner is constructed.

When it happens

Trigger: Triggered when the buffer is full at s.end == len(s.buf), len(s.buf) >= s.maxTokenSize, and the split function still has not produced a token (scan.go:197-203). Common with ScanLines on input containing a single line longer than 64 KB. Raise the limit with Scanner.Buffer(buf, max) or switch to bufio.Reader.ReadString.

Common situations: Scanning unbounded log lines, minified JSON, or CSV fields with very long values. Parsing machine-generated files with no newline for long stretches. Default Go toolchain builds where developers assumed Scan() handles arbitrarily large lines.

Related errors

AI-assisted analysis of golang/go@b6b368adc5 (2026-08-12). Data as JSON: /api/errors/f1402d9b6b7d4f22. Report an issue: GitHub.