{"record":{"id":"3a3186013edf84f2","repo":"alibaba/open-code-review","slug":"ocr-warning-skipping-s-d-bytes-exceeds-d-by","errorCode":null,"errorMessage":"[ocr] WARNING: skipping %s (%d bytes exceeds %d-byte scan limit; raise MaxTokens if the real concern is token budget, not memory)\n","messagePattern":"\\[ocr\\] WARNING: skipping (.+?) \\((.+?) bytes exceeds (.+?)-byte scan limit; raise MaxTokens if the real concern is token budget, not memory\\)\n","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"internal/scan/provider.go","lineNumber":116,"sourceCode":"\t\t\treturn nil, err\n\t\t}\n\t\tif rel == \"\" {\n\t\t\tcontinue\n\t\t}\n\t\tif diff.IsPathExcluded(p.repoDir, rel, gitignorePatterns) {\n\t\t\tcontinue\n\t\t}\n\t\tfull := filepath.Join(p.repoDir, rel)\n\t\tinfo, err := os.Lstat(full)\n\t\tif err != nil {\n\t\t\tfmt.Fprintf(os.Stderr, \"[ocr] WARNING: cannot stat %s: %v\\n\", rel, err)\n\t\t\tcontinue\n\t\t}\n\t\tif !info.Mode().IsRegular() {\n\t\t\tcontinue\n\t\t}\n\t\tif info.Size() > p.maxFileSizeBytes {\n\t\t\tfmt.Fprintf(os.Stderr, \"[ocr] WARNING: skipping %s (%d bytes exceeds %d-byte scan limit; raise MaxTokens if the real concern is token budget, not memory)\\n\",\n\t\t\t\trel, info.Size(), p.maxFileSizeBytes)\n\t\t\tcontinue\n\t\t}\n\t\tbinary, err := isBinaryFile(full)\n\t\tif err != nil {\n\t\t\tfmt.Fprintf(os.Stderr, \"[ocr] WARNING: cannot sniff %s: %v\\n\", rel, err)\n\t\t\tcontinue\n\t\t}\n\t\tif binary {\n\t\t\t// Emit placeholder so preview can display [B], but do not\n\t\t\t// read the file body — saves memory on large binaries.\n\t\t\tout = append(out, model.ScanItem{\n\t\t\t\tPath:     rel,\n\t\t\t\tIsBinary: true,\n\t\t\t})\n\t\t\tcontinue\n\t\t}\n\t\tcontent, err := os.ReadFile(full)","sourceCodeStart":98,"sourceCodeEnd":134,"githubUrl":"https://github.com/alibaba/open-code-review/blob/5cf97d0d15cbd41b602513c4be3bfec3cee5bf7f/internal/scan/provider.go#L98-L134","documentation":"The scan provider's file enumeration (Enumerate, used by Run and preview) skips any regular file larger than maxFileSizeBytes and prints this warning to stderr. It is a budget guard: oversized files would blow the token limit / memory, so they are excluded from review rather than failing the scan.","triggerScenarios":"Calling scanner.Run or preview on a repository containing a regular file whose os.Stat size exceeds p.maxFileSizeBytes (derived from MaxTokens).","commonSituations":"Repo containing vendored minified JS, generated lockfiles (package-lock.json, go.sum at extreme size), datasets, or binaries > the size cap; lowering MaxTokens via config shrinks the cap and newly triggers skips.","solutions":["Raise the MaxTokens setting in your config if token budget, not memory, is the concern (the message explicitly suggests this).","Add the oversized file to exclude/ignore rules if it never needs review, silencing the repeated warning.","Split or minify the offending file, or point the scan at a narrower directory that excludes it.","Accept the skip if the file is unreviewable anyway (generated artifacts); the warning is informational and the scan continues."],"exampleFix":"// before\nprovider := scan.NewProvider(scan.Options{})\n// after\nprovider := scan.NewProvider(scan.Options{MaxTokens: 200000}) // raises maxFileSizeBytes so large files are scanned","handlingStrategy":"validation","validationCode":"info, err := os.Stat(path)\nif err == nil && info.Mode().IsRegular() && info.Size() > maxFileSizeBytes {\n    fmt.Fprintf(os.Stderr, \"pre-check: %s (%d bytes) will be skipped\\n\", path, info.Size())\n}","typeGuard":"func scannable(info os.FileInfo, maxFileSizeBytes int64) bool {\n    return info.Mode().IsRegular() && info.Size() <= maxFileSizeBytes\n}","tryCatchPattern":null,"preventionTips":["Set MaxTokens high enough that expected file sizes fall under the cap","Exclude vendored/generated/minified paths so the warning does not fire on unreviewable files","Pre-check repo for outliers: find . -type f -size +1M","Treat the stderr warning as expected behavior — the scan continues past skipped files"],"tags":["scan","file-size","go"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"5cf97d0d15cbd41b602513c4be3bfec3cee5bf7f","analyzedAt":"2026-09-02T02:08:09.116Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}