{"record":{"id":"0e665b75541f9a69","repo":"apache/beam","slug":"could-not-find-the-internal-step-name","errorCode":null,"errorMessage":"could not find the internal step name","messagePattern":"could not find the internal step name","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"sdks/go/pkg/beam/runners/dataflow/dataflowlib/metrics.go","lineNumber":76,"sourceCode":"\t\t\tif err != nil {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tcounters[key] = v\n\t\t} else if metric.Distribution != nil {\n\t\t\tv, err := extractDistributionValue(metric.Distribution)\n\t\t\tif err != nil {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tdistributions[key] = v\n\t\t}\n\t}\n\treturn counters, distributions\n}\n\nfunc extractKey(metric *df.MetricUpdate, p *pipepb.Pipeline) (metrics.StepKey, error) {\n\tstepName, ok := metric.Name.Context[\"step\"]\n\tif !ok {\n\t\treturn metrics.StepKey{}, fmt.Errorf(\"could not find the internal step name\")\n\t}\n\tuserStepName := \"\"\n\n\tfor k, transform := range p.GetComponents().GetTransforms() {\n\t\tif k == stepName {\n\t\t\tuserStepName = transform.GetUniqueName()\n\t\t\tbreak\n\t\t}\n\t}\n\tif userStepName == \"\" {\n\t\treturn metrics.StepKey{}, fmt.Errorf(\"could not translate the internal step name %v\", stepName)\n\t}\n\n\tnamespace := metric.Name.Context[\"namespace\"]\n\tif namespace == \"\" {\n\t\tnamespace = \"dataflow/v1b3\"\n\t}\n","sourceCodeStart":58,"sourceCodeEnd":94,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/go/pkg/beam/runners/dataflow/dataflowlib/metrics.go#L58-L94","documentation":"When translating Dataflow metric updates into Beam user metrics, extractKey requires the metric update's context map to carry the internal Dataflow step name under the \"step\" key. This error means a MetricUpdate arrived without that context, so the runner cannot attribute the metric to any pipeline step. It indicates an unexpected or malformed metric payload from the Dataflow service.","triggerScenarios":"groupByType processes a *df.MetricUpdate whose Name.Context map has no \"step\" entry — e.g. job-level or system metrics returned by the Dataflow metrics API that are not tied to a specific transform step.","commonSituations":"Polling job metrics that include service-level aggregates (like system metrics or job-level counters) which have no step context; Dataflow API changes or unusual metric shapes from the service.","solutions":["Filter MetricUpdates before calling groupByType/extractKey, skipping those without a \"step\" context entry","Check the Dataflow API version/SDK response shape for changed metric context fields","Wrap the metrics extraction loop so per-metric errors are logged and skipped instead of failing the whole metrics fetch"],"exampleFix":"// before\nkey, err := extractKey(mu, p)\nif err != nil { return err }\n// after\nkey, err := extractKey(mu, p)\nif err != nil {\n    log.Printf(\"skipping metric %v: %v\", mu.Name, err)\n    continue\n}","handlingStrategy":"validation","validationCode":"if _, ok := mu.Name.Context[\"step\"]; !ok {\n    // skip: not a step-scoped metric\n    continue\n}","typeGuard":"func hasStepContext(mu *df.MetricUpdate) bool {\n    _, ok := mu.Name.Context[\"step\"]\n    return ok\n}","tryCatchPattern":"key, err := extractKey(mu, pipeline)\nif err != nil {\n    log.Printf(\"skipping metric %v: %v\", mu.Name.Name, err)\n    continue\n}","preventionTips":["Filter MetricUpdates for step-scoped metrics before extraction","Log-and-skip per-metric errors rather than failing the whole metrics pass","Pin Dataflow API versions and review changelogs for metric context changes"],"tags":["dataflow","go","metrics"],"backgroundTag":"missing-required-argument","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}