apache/beam · error

failed to deduce Step from MonitoringInfo: %v

Error message

failed to deduce Step from MonitoringInfo: %v

What it means

extractKey derives a metrics.StepKey from a MonitoringInfo: it uses the transform label, overridden by a PCollection-to-transform mapping. If the resulting step name is empty — no transform label and no PCollection match — the StepKey cannot be built and this error is returned identifying the MonitoringInfo.

Source

Thrown at sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go:156

		default:
			log.Println("unknown metric type", minfo.GetUrn())
		}
	}
	if len(errs) > 0 {
		slog.Debug("errors during metrics processing", "count", len(errs), "errors", errs)
	}
	return counters, distributions, gauges, msecs, pcols
}

func extractKey(mi *pipepb.MonitoringInfo, pcolToTransform map[string]string) (metrics.StepKey, error) {
	labels := newLabels(mi.GetLabels())
	stepName := labels.Transform()

	if v, ok := pcolToTransform[labels.PCollection()]; ok {
		stepName = v
	}
	if stepName == "" {
		return metrics.StepKey{}, fmt.Errorf("failed to deduce Step from MonitoringInfo: %v", mi)
	}
	return metrics.StepKey{Step: stepName, Name: labels.Name(), Namespace: labels.Namespace()}, nil
}

func extractCounterValue(reader *bytes.Reader) (int64, error) {
	value, err := coder.DecodeVarInt(reader)
	if err != nil {
		return -1, err
	}
	return value, nil
}

func extractMsecValue(reader *bytes.Reader) (time.Duration, error) {
	value, err := coder.DecodeVarInt(reader)
	if err != nil {
		return 0, err
	}
	return time.Duration(value) * time.Millisecond, nil

View on GitHub (pinned to 12126d8942)

Solutions

1. Ensure the MonitoringInfo carries a valid transform (PTransform) label.
2. Populate pcolToTransform for every PCollection referenced by monitoring infos before calling extractKey.
3. Log/inspect the MonitoringInfo in the error message to identify which PCollection is unmapped.
4. Upgrade SDK/runner together to keep monitoring-info label URNs consistent.

Example fix

// before
key, err := extractKey(mi, nil) // no PCollection mapping
// after
key, err := extractKey(mi, map[string]string{"pcoll-1": "step-Extract"})

func hasStepIdentity(mi *monpb.MonitoringInfo, pcolToTransform map[string]string) bool {
	return mi.GetLabels()["PTRANSFORM"] != "" ||
		pcolToTransform[mi.GetLabels()["PCOLLECTION"]] != ""
}

key, err := extractKey(mi, pcolToTransform)
if err != nil {
	log.Printf("skipping unmappable monitoring info: %v", err)
	continue
}

When it happens

Trigger: Processing monitoring data (via groupByType) whose labels lack a transform and whose PCollection ID is absent from pcolToTransform — e.g. monitoring infos from unknown/unmapped PCollections or hand-crafted metrics.

Common situations: Runner-emitted monitoring infos for synthetic/edge transforms, metrics for PCollections not registered in the mapping, or version skew where label URNs changed.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• invalid bundle processing state: %d
• failed to get metrics
• namespace and name are required to be non-empty, got %q and
• metric name %s being reused for a different metric type in a
• Type interface{} isn't a supported PCollection type

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/aac8789650a5f147. Report an issue: GitHub.