{"record":{"id":"d33c202147a5dc3b","repo":"JuliusBrussee/caveman","slug":"cacheengine-cumulative-token-count-overflow","errorCode":null,"errorMessage":"cacheengine: cumulative token count overflow","messagePattern":"cacheengine: cumulative token count overflow","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cacheengine/engine.go","lineNumber":369,"sourceCode":"\tvar lengths [8]byte\n\tbinary.BigEndian.PutUint32(lengths[:4], uint32(len(name)))\n\tbinary.BigEndian.PutUint32(lengths[4:], uint32(len(content)))\n\tdst = append(dst, lengths[:]...)\n\tdst = append(dst, name...)\n\treturn append(dst, content...)\n}\n\nfunc breakpointCandidates(segments []Segment, defaultCalls int, profile Profile) ([]Breakpoint, bool, bool, error) {\n\tvar candidates []Breakpoint\n\tvar prefix []byte\n\tcumulativeTokens := 0\n\tpreviousCalls := math.MaxInt\n\tbelowMinimum := false\n\tnegative := false\n\tfor index, segment := range segments {\n\t\tprefix = appendFrame(prefix, segment.Name, segment.Content)\n\t\tif segment.Tokens > math.MaxInt-cumulativeTokens {\n\t\t\treturn nil, false, false, errors.New(\"cacheengine: cumulative token count overflow\")\n\t\t}\n\t\tcumulativeTokens += segment.Tokens\n\t\tcalls := segment.ExpectedCalls\n\t\tif calls == 0 {\n\t\t\tcalls = defaultCalls\n\t\t}\n\t\tif calls > previousCalls {\n\t\t\treturn nil, false, false, errors.New(\"cacheengine: longer prefix cannot have higher expected reuse\")\n\t\t}\n\t\tpreviousCalls = calls\n\t\tif calls < 2 {\n\t\t\tcontinue\n\t\t}\n\t\tif cumulativeTokens > 0 && cumulativeTokens < profile.MinPrefixTokens {\n\t\t\tbelowMinimum = true\n\t\t\tcontinue\n\t\t}\n\t\tnet := 0.0","sourceCodeStart":351,"sourceCodeEnd":387,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/engine.go#L351-L387","documentation":"Thrown by breakpointCandidates when adding segment.Tokens to the running cumulativeTokens total would overflow int. It is a hard stop before the cumulative count corrupts breakpoint token math.","triggerScenarios":"A segment list whose summed Tokens exceeds math.MaxInt (~9.2e18 on 64-bit, ~2.1e9 on 32-bit builds); on 32-bit targets, token counts in the low billions are enough to trigger it.","commonSituations":"Running the engine on a 32-bit architecture (GOARCH=386/arm) with large token counts; a tokenizer returning garbage (huge counts) for adversarial input; unit tests with MaxInt-valued token values to probe overflow behavior.","solutions":["Validate each segment's Tokens is a plausible small number (< 10^9) before planning","Rebuild/run on a 64-bit target if you are on 386/arm and totals legitimately approach 2^31","Fix the tokenizer if it emits absurd counts for pathological input"],"exampleFix":"// before\nsegments := []cacheengine.Segment{{Name: \"a\", Stable: true, Cacheable: true, Content: b, Tokens: math.MaxInt}}\n\n// after\nsegments := []cacheengine.Segment{{Name: \"a\", Stable: true, Cacheable: true, Content: b, Tokens: 120000}}","handlingStrategy":"validation","validationCode":"var total int\nfor _, s := range segs {\n    if s.Tokens < 0 || s.Tokens > 1<<40 { return errors.New(\"implausible token count\") }\n    total += s.Tokens\n    if total < 0 { return errors.New(\"token total overflow\") }\n}","typeGuard":"// n/a","tryCatchPattern":null,"preventionTips":["Sanity-cap per-segment token counts","Prefer 64-bit builds for large-corpus workloads"],"tags":["cacheengine","overflow","tokens","go"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}