{"record":{"id":"670b4e2df80ce2d8","repo":"thanos-io/thanos","slug":"err-error-670b4e","errorCode":null,"errorMessage":"err.Error()","messagePattern":"err\\.Error\\(\\)","errorType":"http","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"pkg/receive/handler_otlp.go","lineNumber":153,"sourceCode":"\t\t\twreq:   &wreq,\n\t\t},\n\t})\n\tif err != nil {\n\t\tlevel.Debug(tLogger).Log(\"msg\", \"failed to handle request\", \"err\", err.Error())\n\t\tswitch errors.Cause(err) {\n\t\tcase errNotReady:\n\t\t\tresponseStatusCode = http.StatusServiceUnavailable\n\t\tcase errUnavailable:\n\t\t\tresponseStatusCode = http.StatusServiceUnavailable\n\t\tcase errConflict:\n\t\t\tresponseStatusCode = http.StatusConflict\n\t\tcase errBadReplica:\n\t\t\tresponseStatusCode = http.StatusBadRequest\n\t\tdefault:\n\t\t\tlevel.Error(tLogger).Log(\"err\", err, \"msg\", \"internal server error\")\n\t\t\tresponseStatusCode = http.StatusInternalServerError\n\t\t}\n\t\thttp.Error(w, err.Error(), responseStatusCode)\n\t}\n\n\tfor tenant, stats := range tenantStats {\n\t\th.writeTimeseriesTotal.WithLabelValues(strconv.Itoa(responseStatusCode), tenant).Observe(float64(stats.timeseries))\n\t\th.writeSamplesTotal.WithLabelValues(strconv.Itoa(responseStatusCode), tenant).Observe(float64(stats.totalSamples))\n\t}\n\n}\n\nfunc (h *Handler) convertToPrometheusFormat(ctx context.Context, pmetrics pmetric.Metrics) ([]tprompb.TimeSeries, []tprompb.MetricMetadata, error) {\n\tconverter := otlptranslator.NewPrometheusConverter()\n\tsettings := otlptranslator.Settings{\n\t\tAddMetricSuffixes:         true,\n\t\tDisableTargetInfo:         !h.options.OtlpEnableTargetInfo,\n\t\tPromoteResourceAttributes: h.options.OtlpResourceAttributes,\n\t}\n\n\tannots, err := converter.FromMetrics(ctx, pmetrics, settings)","sourceCodeStart":135,"sourceCodeEnd":171,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/receive/handler_otlp.go#L135-L171","documentation":"This is the generic internal-server-error path (HTTP 500) of the OTLP ingest handler: after h.writer.Write / forwarding fails with an error that is not errBadReplica or another recognized case, the handler logs 'internal server error' and returns err.Error() with status 500. It surfaces any write/forward failure from the underlying store or replication fan-out.","triggerScenarios":"Write to the local TSDB head failed (storage errors, out-of-order/duplicate handling disabled), or forwarding to other receive nodes failed (hashring lookup failure, all replicas down, network errors) — anything not classified as a bad replica.","commonSituations":"TSDB storage full or corrupted; remote receive peers unreachable due to network policies; hashring config mismatch between receive nodes; head block compaction issues.","solutions":["Check the receive server logs — the log line includes the underlying err with full detail.","Verify storage health: disk space, TSDB head block, and storage.tsdb.path writability.","Check network connectivity and DNS between receive nodes for forwarding failures.","Confirm all receive nodes share a consistent hashring configuration.","Retry the write; for forwarding failures receive retries internally, but persistent 500s indicate an infrastructure problem."],"exampleFix":"# diagnose\nkubectl logs -l app=thanos-receive | grep 'internal server error'\n# check disk\ndf -h /var/thanos/store\n# verify peers\nthanos tools bucket verify 2>/dev/null || curl -s http://receive:10902/-/ready","handlingStrategy":"retry","validationCode":"resp, err := http.Post(url, ct, body)\nif err != nil || resp.StatusCode >= 500 { scheduleRetry(body) }","typeGuard":null,"tryCatchPattern":"if resp.StatusCode >= 500 { backoff.Retry(func() error { return resend(body) }) } else { failPermanent(resp) }","preventionTips":["Monitor receive disk usage and TSDB health","Alert on 5xx rates from receive endpoints","Keep hashring configs synchronized across nodes","Configure exporter retry with exponential backoff for 5xx"],"tags":["http","thanos","storage","internal-error"],"backgroundTag":"http-error-response","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}