{"record":{"id":"43ca00ea02f1b9a4","repo":"JuliusBrussee/caveman","slug":"sanitize-row-d-events-exceed-d-bytes","errorCode":null,"errorMessage":"sanitize row %d: events exceed %d bytes","messagePattern":"sanitize row (.+?): events exceed (.+?) bytes","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"shared/platform/importers/importers.go","lineNumber":99,"sourceCode":"\t\trows, err = parseOTLP(data, opts)\n\tcase FormatCavemanJSONL:\n\t\trows, err = parseCavemanJSONL(data, opts)\n\tcase FormatLangfuse:\n\t\trows, err = parseLangfuse(data, opts)\n\tcase FormatHelicone:\n\t\trows, err = parseHelicone(data, opts)\n\tcase FormatGeneric:\n\t\trows, err = parseGeneric(data, opts)\n\tdefault:\n\t\t// fail-closed: unknown formats are an error, never a silent pass-through.\n\t\treturn nil, Summary{}, fmt.Errorf(\"unknown import format %q: supported formats are otlp, caveman-jsonl, langfuse, helicone, generic\", format)\n\t}\n\tif err != nil {\n\t\treturn nil, Summary{}, err\n\t}\n\tfor i := range rows {\n\t\tif len(rows[i].EventsJSON) > telemetry.MaxEventsBytes {\n\t\t\treturn nil, Summary{}, fmt.Errorf(\"sanitize row %d: events exceed %d bytes\", i, telemetry.MaxEventsBytes)\n\t\t}\n\t\tsanitized, sanitizeErr := telemetry.SanitizeAttributes(rows[i].Attributes)\n\t\tif sanitizeErr != nil {\n\t\t\treturn nil, Summary{}, fmt.Errorf(\"sanitize row %d: %w\", i, sanitizeErr)\n\t\t}\n\t\trows[i].Attributes = sanitized\n\t}\n\treturn rows, Summary{Format: format, RowCount: len(rows)}, nil\n}\n\n// ---------------------------------------------------------------------------\n// Shared helpers\n// ---------------------------------------------------------------------------\n\n// chTimeLayout is the ClickHouse DateTime64 textual layout the gateway writer\n// uses; importers must produce the same shape (\"2006-01-02 15:04:05.000\").\nconst chTimeLayout = \"2006-01-02 15:04:05.000\"\n","sourceCodeStart":81,"sourceCodeEnd":117,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/shared/platform/importers/importers.go#L81-L117","documentation":"After format-specific parsing, Import enforces a global cap: each row's EventsJSON may not exceed telemetry.MaxEventsBytes (64 KiB). EventsJSON is the serialized event payload attached to a span; a single oversized event set aborts the entire import with the offending row index. This is a resource guard before anything reaches storage.","triggerScenarios":"A span carrying hundreds of large events (e.g. full request/response bodies logged as events); an OTLP or langfuse record with megabyte-scale attribute arrays that the mapper folds into events; machine-generated spans that embed stack traces or base64 payloads per event.","commonSituations":"Verbose LLM tracing that logs full prompts and completions as events; importing debug-level traces captured with event dumping on; a single trace with thousands of tiny events accumulating past 64 KiB.","solutions":["Reduce event verbosity at the source: sample events or truncate large payloads before capture","Split oversized spans or drop non-essential events in pre-processing so EventsJSON stays under 65536 bytes","Check row index in the error to find which span blew the limit, then re-export that trace with fewer events","If large payloads are required, store them out-of-band and reference them from attributes instead"],"exampleFix":"// before\nevent := map[string]any{\"type\": \"completion\", \"body\": fullResponseBody /* 200KB */}\n\n// after\nconst maxBody = 32 << 10\nif len(fullResponseBody) > maxBody {\n    fullResponseBody = fullResponseBody[:maxBody]\n}\nevent := map[string]any{\"type\": \"completion\", \"body\": fullResponseBody}","handlingStrategy":"validation","validationCode":"const maxEvents = telemetry.MaxEventsBytes\nfunc eventsWithinLimit(eventsJSON []byte) bool { return len(eventsJSON) <= maxEvents }","typeGuard":null,"tryCatchPattern":"if err != nil && strings.Contains(err.Error(), \"events exceed\") { /* find row index, drop or split its events, re-import */ }","preventionTips":["Cap event payload size at capture time (truncate bodies to a fixed budget)","Sample high-frequency events before they attach to spans","Alert when any span's serialized events approach 64 KiB"],"tags":["go","import","telemetry","size-limit","events"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}