{"record":{"id":"415e87145e910ce1","repo":"jaegertracing/jaeger","slug":"failed-executing-metrics-query-w-415e87","errorCode":null,"errorMessage":"failed executing metrics query: %w","messagePattern":"failed executing metrics query: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"internal/storage/metricstore/prometheus/metricstore/reader.go","lineNumber":272,"sourceCode":"func (m MetricsReader) executeQuery(ctx context.Context, p metricsQueryParams) (*metrics.MetricFamily, error) {\n\tif p.GroupByOperation {\n\t\tp.metricName = strings.Replace(p.metricName, \"service\", \"service_operation\", 1)\n\t\tp.metricDesc += \" & operation\"\n\t}\n\tpromQuery := m.buildPromQuery(p)\n\n\tctx, span := startSpanForQuery(ctx, p.metricName, promQuery, m.tracer)\n\tdefer span.End()\n\n\tqueryRange := promapi.Range{\n\t\tStart: p.EndTime.Add(-1 * *p.Lookback),\n\t\tEnd:   *p.EndTime,\n\t\tStep:  *p.Step,\n\t}\n\n\tmv, warnings, err := m.client.QueryRange(ctx, promQuery, queryRange)\n\tif err != nil {\n\t\terr = fmt.Errorf(\"failed executing metrics query: %w\", err)\n\t\tlogErrorToSpan(span, err)\n\t\treturn &metrics.MetricFamily{}, err\n\t}\n\tif len(warnings) > 0 {\n\t\tm.logger.Warn(\"Warnings detected on Prometheus query\", zap.Any(\"warnings\", warnings), zap.String(\"query\", promQuery), zap.Any(\"range\", queryRange))\n\t}\n\n\tm.logger.Debug(\"Prometheus query results\", zap.String(\"results\", mv.String()), zap.String(\"query\", promQuery), zap.Any(\"range\", queryRange))\n\n\treturn m.metricsTranslator.ToDomainMetricsFamily(\n\t\tp.metricName,\n\t\tp.metricDesc,\n\t\tmv,\n\t)\n}\n\nfunc (m MetricsReader) buildPromQuery(metricsParams metricsQueryParams) string {\n\tgroupBy := []string{\"service_name\"}","sourceCodeStart":254,"sourceCodeEnd":290,"githubUrl":"https://github.com/jaegertracing/jaeger/blob/806f4447841ecdb60519f408b004a599d515f437/internal/storage/metricstore/prometheus/metricstore/reader.go#L254-L290","documentation":"executeQuery runs a Prometheus RangeQuery (client.QueryRange) for latencies, call rates, or error rates. When the query execution fails, the error is wrapped with this prefix, logged to the active tracing span, and returned. It is the single choke point for all Prometheus metric-store query failures in this reader.","triggerScenarios":"Any of GetLatencies, GetCallRates, or GetErrorRates reaching executeQuery where m.client.QueryRange returns an error: HTTP failure, non-2xx from the Prometheus API, malformed promql, or context timeout.","commonSituations":"Prometheus unreachable (wrong URL/DNS), TLS misconfiguration, Prometheus 422/400 for invalid queries, timeouts on large time ranges with small steps.","solutions":["Inspect the wrapped error for the exact Prometheus client cause (URL, status code, query).","Verify --prometheus.server-url, TLS, and auth settings against your Prometheus deployment.","Reduce the query range or increase the step to avoid timeouts on heavy queries.","Test the generated promql manually against Prometheus (it is logged on warnings and available via the span)."],"exampleFix":"// before: retrying blindly\nresp, err := getMetrics()\n// after: fail fast and surface the underlying cause\nif err != nil {\n    var apiErr *promapi.Error\n    if errors.As(err, &apiErr) {\n        logger.Error(\"prometheus rejected query\", zap.String(\"query\", promQuery), zap.Error(apiErr))\n    }\n    return nil, err\n}","handlingStrategy":"retry","validationCode":"if resp, err := http.Get(promURL + \"/api/v1/status/buildinfo\"); err != nil || resp.StatusCode != 200 {\n    return errors.New(\"prometheus API unavailable\")\n}","typeGuard":null,"tryCatchPattern":"mv, warnings, err := m.client.QueryRange(ctx, promQuery, queryRange)\nif err != nil {\n    if ctx.Err() != nil { return ctx.Err() } // don't retry cancelled contexts\n    err = fmt.Errorf(\"failed executing metrics query: %w\", err)\n    logErrorToSpan(span, err)\n    return &metrics.MetricFamily{}, err\n}","preventionTips":["Retry only transient failures (timeouts, 5xx) with backoff; never retry 4xx query errors.","Bound the query range/step to avoid oversized queries.","Log the promql with the error so failures are debuggable."],"tags":["prometheus","metrics","query","network"],"backgroundTag":"prometheus-query-failed","analyzedSha":"806f4447841ecdb60519f408b004a599d515f437","analyzedAt":"2026-09-01T02:39:22.140Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}