{"record":{"id":"f1402d9b6b7d4f22","repo":"golang/go","slug":"bufio-scanner-token-too-long","errorCode":null,"errorMessage":"bufio.Scanner: token too long","messagePattern":"bufio\\.Scanner: token too long","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/bufio/scan.go","lineNumber":71,"sourceCode":"// immediately stops the scanning.\n//\n// Otherwise, the [Scanner] advances the input. If the token is not nil,\n// the [Scanner] returns it to the user. If the token is nil, the\n// Scanner reads more data and continues scanning; if there is no more\n// data--if atEOF was true--the [Scanner] returns. If the data does not\n// yet hold a complete token, for instance if it has no newline while\n// scanning lines, a [SplitFunc] can return (0, nil, nil) to signal the\n// [Scanner] to read more data into the slice and try again with a\n// longer slice starting at the same point in the input.\n//\n// The function is never called with an empty data slice unless atEOF\n// is true. If atEOF is true, however, data may be non-empty and,\n// as always, holds unprocessed text.\ntype SplitFunc func(data []byte, atEOF bool) (advance int, token []byte, err error)\n\n// Errors returned by Scanner.\nvar (\n\tErrTooLong         = errors.New(\"bufio.Scanner: token too long\")\n\tErrNegativeAdvance = errors.New(\"bufio.Scanner: SplitFunc returns negative advance count\")\n\tErrAdvanceTooFar   = errors.New(\"bufio.Scanner: SplitFunc returns advance count beyond input\")\n\tErrBadReadCount    = errors.New(\"bufio.Scanner: Read returned impossible count\")\n)\n\nconst (\n\t// MaxScanTokenSize is the maximum size used to buffer a token\n\t// unless the user provides an explicit buffer with [Scanner.Buffer].\n\t// The actual maximum token size may be smaller as the buffer\n\t// may need to include, for instance, a newline.\n\tMaxScanTokenSize = 64 * 1024\n\n\tstartBufSize = 4096 // Size of initial allocation for buffer.\n)\n\n// NewScanner returns a new [Scanner] to read from r.\n// The split function defaults to [ScanLines].\nfunc NewScanner(r io.Reader) *Scanner {","sourceCodeStart":53,"sourceCodeEnd":89,"githubUrl":"https://github.com/golang/go/blob/b6b368adc57c96c3151d224d172029f233ead2c3/src/bufio/scan.go#L53-L89","documentation":"bufio.Scanner sets ErrTooLong when a token (e.g., a single line) exceeds the scanner's maxTokenSize, which defaults to MaxScanTokenSize = 64 * 1024 bytes (scan.go:82, scan.go:200-202). The scanner doubles its internal buffer until it reaches maxTokenSize; if a complete token still does not fit, scanning halts and Scanner.Err() returns this error. It is a returned error (not a panic), surfaced after Scan() returns false.","triggerScenarios":"Triggered when the buffer is full at s.end == len(s.buf), len(s.buf) >= s.maxTokenSize, and the split function still has not produced a token (scan.go:197-203). Common with ScanLines on input containing a single line longer than 64 KB. Raise the limit with Scanner.Buffer(buf, max) or switch to bufio.Reader.ReadString.","commonSituations":"Scanning unbounded log lines, minified JSON, or CSV fields with very long values. Parsing machine-generated files with no newline for long stretches. Default Go toolchain builds where developers assumed Scan() handles arbitrarily large lines.","solutions":["Call scanner.Buffer(make([]byte, 0, 64*1024), <maxBytes>) before scanning to raise maxTokenSize to fit your largest expected token.","Switch to bufio.Reader.ReadString('\\n') or ReadBytes for line-oriented parsing with no hard token cap.","If the input genuinely has no delimiter, pre-split it (e.g., stream chunking) before scanning.","Lower maxTokenSize deliberately only if you want to reject oversized tokens as a safety bound."],"exampleFix":"// before — default 64 KB cap, panics on long lines\nscanner := bufio.NewReader(file)\n// using NewScanner with default max\nsc := bufio.NewScanner(file)\nfor sc.Scan() {} // ErrTooLong on >64KB line\n\n// after — raise the cap explicitly\nsc := bufio.NewScanner(file)\nsc.Buffer(make([]byte, 0, 1024*1024), 10*1024*1024) // up to 10 MB tokens\nfor sc.Scan() {\n    line := sc.Text()\n}","handlingStrategy":"validation","validationCode":"// Configure the scanner buffer BEFORE the first Scan call.\nsc := bufio.NewScanner(r)\nsc.Buffer(make([]byte, 0, 64*1024), maxExpectedTokenBytes)\nfor sc.Scan() {\n    _ = sc.Bytes()\n}\nif err := sc.Err(); err != nil {\n    if err == bufio.ErrTooLong {\n        // token exceeded maxExpectedTokenBytes; handle oversized input\n    }\n}","typeGuard":null,"tryCatchPattern":"// After Scan returns false, distinguish ErrTooLong from other failures.\nfor sc.Scan() {\n    process(sc.Bytes())\n}\nif err := sc.Err(); err != nil {\n    if errors.Is(err, bufio.ErrTooLong) {\n        // switch to a streaming reader or raise the cap\n    } else {\n        return err\n    }\n}","preventionTips":["Always call Scanner.Buffer with an explicit max when input token size is unbounded.","For line parsing of arbitrary-length input, prefer bufio.Reader.ReadString over Scanner.","Document the chosen max token size where the scanner is constructed."],"tags":["go","bufio","scanner","token","io"],"analyzedSha":"b6b368adc57c96c3151d224d172029f233ead2c3","analyzedAt":"2026-08-12T00:22:02.250Z","schemaVersion":2},"datasetVersion":"2026-08-12T08:17:17.861Z"}