{"record":{"id":"a469d185f857b964","repo":"JuliusBrussee/caveman","slug":"cachebench-observations-exceed-record-limit-d","errorCode":null,"errorMessage":"cachebench: observations exceed record limit %d","messagePattern":"cachebench: observations exceed record limit (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cacheengine/cachebench/observed.go","lineNumber":113,"sourceCode":"func ReadObservationJSONLWithLimits(reader io.Reader, limits ObservationReadLimits) ([]ObservationRecord, error) {\n\tif limits.MaxLineBytes <= 0 || limits.MaxLineBytes > 64<<20 || limits.MaxRecords <= 0 || limits.MaxRecords > 1_000_000 {\n\t\treturn nil, errors.New(\"cachebench: invalid observation read limits\")\n\t}\n\tscanner := bufio.NewScanner(reader)\n\tinitial := 64 * 1024\n\tif limits.MaxLineBytes < initial {\n\t\tinitial = limits.MaxLineBytes\n\t}\n\tscanner.Buffer(make([]byte, initial), limits.MaxLineBytes)\n\tseen := map[string]bool{}\n\tvar records []ObservationRecord\n\tfor line := 1; scanner.Scan(); line++ {\n\t\traw := bytes.TrimSpace(scanner.Bytes())\n\t\tif len(raw) == 0 {\n\t\t\tcontinue\n\t\t}\n\t\tif len(records) >= limits.MaxRecords {\n\t\t\treturn nil, fmt.Errorf(\"cachebench: observations exceed record limit %d\", limits.MaxRecords)\n\t\t}\n\t\tif !validUniqueJSONObject(raw) {\n\t\t\treturn nil, fmt.Errorf(\"cachebench: observation line %d: duplicate or invalid JSON\", line)\n\t\t}\n\t\tdecoder := json.NewDecoder(bytes.NewReader(raw))\n\t\tdecoder.DisallowUnknownFields()\n\t\tvar record ObservationRecord\n\t\tif err := decoder.Decode(&record); err != nil {\n\t\t\treturn nil, fmt.Errorf(\"cachebench: observation line %d: %w\", line, err)\n\t\t}\n\t\tvar trailing any\n\t\tif err := decoder.Decode(&trailing); err != io.EOF {\n\t\t\treturn nil, fmt.Errorf(\"cachebench: observation line %d: trailing JSON\", line)\n\t\t}\n\t\tif record.Schema != ObservationSchema {\n\t\t\treturn nil, fmt.Errorf(\"cachebench: observation line %d: schema %q\", line, record.Schema)\n\t\t}\n\t\tif strings.TrimSpace(record.RequestID) == \"\" || seen[record.RequestID] {","sourceCodeStart":95,"sourceCodeEnd":131,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/cacheengine/cachebench/observed.go#L95-L131","documentation":"The JSONL observation reader caps how many records it will return: once limits.MaxRecords records have been parsed, the next non-empty line triggers this error instead of silently truncating. It fires mid-scan, so a partial slice is discarded (nil returned).","triggerScenarios":"Loading an observations file whose non-empty line count exceeds ReaderLimits.MaxRecords — the (MaxRecords+1)-th populated line errors.","commonSituations":"Long replays producing more observations than the configured budget; raising replay length without raising MaxRecords; concatenating observation files.","solutions":["Raise limits.MaxRecords to at least the file's record count (count non-empty lines).","Split the observation file and evaluate shards in separate runs.","Subsample the replay so observations stay under the cap."],"exampleFix":"// before\nlimits.MaxRecords = 50_000\n// after\nlimits.MaxRecords = 500_000","handlingStrategy":"validation","validationCode":"func countRecords(path string) (int, error) {\n\tf, err := os.Open(path)\n\tif err != nil {\n\t\treturn 0, err\n\t}\n\tdefer f.Close()\n\tn := 0\n\tscanner := bufio.NewScanner(f)\n\tfor scanner.Scan() {\n\t\tif len(bytes.TrimSpace(scanner.Bytes())) > 0 {\n\t\t\tn++\n\t\t}\n\t}\n\treturn n, scanner.Err()\n}\n// if n > limits.MaxRecords { limits.MaxRecords = n }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Pre-count observation lines and size MaxRecords accordingly.","Split long observation files per epoch.","Remember MaxLineBytes too: oversized lines fail the scanner before the record cap is reached."],"tags":["go","limits","observations","jsonl"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}