apache/beam · warning · ValueError

Could not find element

Error message

Could not find element

What it means

`_get_match` filters a protobuf sequence with a predicate and requires exactly one match; zero matches raises ValueError('Could not find element'). In dataflow_metrics it's used to look up pipeline steps by name when translating internal step names.

Source

Thrown at sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py:52

from apache_beam.metrics.execution import MetricKey
from apache_beam.metrics.execution import MetricResult
from apache_beam.metrics.metric import MetricResults
from apache_beam.metrics.metricbase import MetricName
from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import PipelineOptions

_LOGGER = logging.getLogger(__name__)


def _get_match(proto, filter_fn):
  """Finds and returns the first element that matches a query.

  If no element matches the query, it throws ValueError.
  If more than one element matches the query, it returns only the first.
  """
  query = [elm for elm in proto if filter_fn(elm)]
  if len(query) == 0:
    raise ValueError('Could not find element')
  elif len(query) > 1:
    raise ValueError('Too many matches')

  return query[0]


# V1b3 MetricStructuredName keys to accept and copy to the MetricKey labels.
STRUCTURED_NAME_LABELS = set(
    ['execution_step', 'original_name', 'output_user_name'])


class DataflowMetrics(MetricResults):
  """Implementation of MetricResults class for the Dataflow runner."""
  def __init__(self, dataflow_client=None, job_result=None, job_graph=None):
    """Initialize the Dataflow metrics object.

    Args:
      dataflow_client: apiclient.DataflowApplicationClient to interact with the

View on GitHub (pinned to 12126d8942)

Solutions

1. Catch ValueError from _get_match; dataflow_metrics itself catches and falls through to 'Could not translate' handling in the caller.
2. Refresh/re-fetch the job graph and metrics from the same job_id so names match.
3. Check that you're querying the correct job's metrics object (DataflowMetrics for this specific job result).

Example fix

# before
step = metrics._translate_step_name(name)
# after
try:
    step = metrics._translate_step_name(name)
except ValueError as e:
    logging.warning('step name lookup failed: %s', e)
    step = name  # fall back to internal name

step_names = {s.name for s in job_graph.proto.steps}
assert internal_name in step_names, f'{internal_name} not in job graph'

try:
    user_name = metrics._translate_step_name(internal_name)
except ValueError:
    user_name = internal_name

When it happens

Trigger: `_translate_step_name` calls `_get_match(job_graph.proto.steps, lambda x: x.name == internal_name)` and no step in the job graph has that internal name (e.g. stale metric keys from a different job, or job graph mismatch).

Common situations: Querying Dataflow metrics for a job whose graph doesn't correspond to the metric step names — e.g. metrics from a retried/updated job or when the harness emits step names absent from the submitted proto.

Understand the failure class

Background: Record Not Found Errors: "not found", RecordNotFound, and "was not found" — what they mean and how to fix them — this error's family across 28 libraries.

Related errors

• Too many matches
• Could not translate the internal step name %r since job grap
• Could not translate the internal step name %r.
• Can not query metrics. Job id is unknown.
• NotImplementedError

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/de61408b0c6e24d0. Report an issue: GitHub.