What causes "failed to get metrics" in apache/beam?

Retry the job results fetch — the job may have succeeded even though metrics retrieval failed.

failed to get metrics

Error message

failed to get metrics

What it means

newUniversalPipelineResult fetches a completed job's metrics from the runner's JobService via gRPC GetJobMetrics. If that RPC fails, the error is wrapped as "failed to get metrics" and returned with an empty metrics result. The pipeline ran, but its counters/monitoring info could not be retrieved.

Source

Thrown at sdks/go/pkg/beam/runners/universal/runnerlib/execute.go:155

		Path:   worker,
		Sha256: hash,
	})
	if err := graphx.UpdateDefaultEnvWorkerType(graphx.URNArtifactFileType, pyld, p); err != nil {
		return err
	}
	return nil
}

type universalPipelineResult struct {
	jobID   string
	metrics *metrics.Results
}

func newUniversalPipelineResult(ctx context.Context, jobID string, client jobpb.JobServiceClient, p *pipepb.Pipeline) (*universalPipelineResult, error) {
	request := &jobpb.GetJobMetricsRequest{JobId: jobID}
	response, err := client.GetJobMetrics(ctx, request)
	if err != nil {
		return &universalPipelineResult{jobID, nil}, errors.Wrap(err, "failed to get metrics")
	}

	monitoredStates := response.GetMetrics()
	metrics := metricsx.FromMonitoringInfos(p, monitoredStates.Attempted, monitoredStates.Committed)
	return &universalPipelineResult{jobID, metrics}, nil
}

func (pr universalPipelineResult) Metrics() metrics.Results {
	return *pr.metrics
}

func (pr universalPipelineResult) JobID() string {
	return pr.jobID
}

View on GitHub (pinned to 12126d8942)

Solutions

Retry the job results fetch — the job may have succeeded even though metrics retrieval failed.
Check connectivity/endpoint to the runner's job service (host, port, TLS).
Verify the job still exists on the runner (it may have been garbage-collected).
Increase the gRPC/context timeout used when creating the pipeline result.

Defensive patterns

Strategy: retry

Validate before calling

// Go: probe the job service before/after the run
conn, err := grpc.Dial(endpoint, grpc.WithInsecure())
if err != nil { /* endpoint unreachable; fix config first */ }

Try / catch

res, err := beam.Run(ctx, runner, p)
if err != nil && strings.Contains(err.Error(), "failed to get metrics") {
    // job likely succeeded; retry fetching metrics or treat as non-fatal
}

Prevention

Treat metrics retrieval failure as non-fatal if the job itself completed.
Verify runner endpoint/TLS before submitting.
Increase context timeouts for large jobs whose metrics fetch may be slow.
Ensure the runner doesn't purge jobs before metrics are fetched.

When it happens

Trigger: client.GetJobMetrics returns a transport or server error while constructing the universal pipeline result after WaitForCompletion, called from Execute.

Common situations: Runner service restarted or unreachable after job completion; network interruption between client and runner; job purged immediately after completion so metrics are unavailable; gRPC deadline exceeded.

Understand the failure class

Background: "API request failed": what wrapped HTTP errors from external APIs mean and how to find the real cause — this error's family across 29 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1fa2c1f3fb9677a0. Report an issue: GitHub.