apache/beam · warning · RuntimeError

Unable to find the job id or job name from envvar.

Error message

Unable to find the job id or job name from envvar.

What it means

When Google Cloud Profiler is enabled on a Dataflow worker, googlecloudprofiler.start requires a service name and version derived from the job id/job name environment variables. If neither is present, the worker raises 'Unable to find the job id or job name from envvar.'; the surrounding handler logs a warning so profiling is skipped, not fatal.

Solutions

  1. Disable Google Cloud Profiler unless running on Dataflow (remove --dataflow_enable_google_cloud_profiler)
  2. Ensure JOB_ID/JOB_NAME environment variables are set in the worker environment
  3. Set gcp_profiler_service_name/gcp_profiler_service_version explicitly in sdk_worker_main so envvar lookup is bypassed
  4. Check the worker logs: this error is caught and logged as a warning, profiling is simply skipped

Example fix

// before
# harness started with --dataflow_enable_google_cloud_profiler but no JOB_ID
// after
export JOB_ID=my-job-id
export JOB_NAME=my-job-name
# or omit the profiler flag for local runs
Defensive patterns

Strategy: validation

Validate before calling

import os
profiler_ready = os.environ.get('JOB_ID') or os.environ.get('JOB_NAME')
if not profiler_ready:
  print('Cloud Profiler disabled: JOB_ID/JOB_NAME not set')

Type guard

def can_start_profiler(env): return bool(env.get('JOB_ID') and env.get('JOB_NAME'))

Try / catch

try:
  _start_profiler(options)
except RuntimeError as e:
  log.warning('profiler skipped: %s', e)  # non-fatal, mirroring sdk_worker_main

Prevention

When it happens

Trigger: Enabling cloud profiling (e.g. --dataflow_enable_google_cloud_profiler or setting the profiler flag) on a runner/environments where JOB_ID/JOB_NAME env vars are not exported to the worker.

Common situations: Running the harness locally or on non-Dataflow runners with profiling enabled; custom container images that drop Dataflow env vars; older runner versions not exporting job metadata.

Understand the failure class

Background: "environment variable is not set" and "Missing keys in environment" errors: what missing required env var messages mean and how to fix them — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/31b5f9f8e6bd58cb. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/worker/sdk_worker_main.py:220

      data_sampler=data_sampler,
      deferred_exception=deferred_exception,
      runner_capabilities=runner_capabilities,
      element_processing_timeout_minutes=sdk_pipeline_options.view_as(
          WorkerOptions).element_processing_timeout_minutes)
  return fn_log_handler, sdk_harness, sdk_pipeline_options


def _start_profiler(gcp_profiler_service_name, gcp_profiler_service_version):
  try:
    import googlecloudprofiler
    if gcp_profiler_service_name and gcp_profiler_service_version:
      googlecloudprofiler.start(
          service=gcp_profiler_service_name,
          service_version=gcp_profiler_service_version,
          verbose=1)
      _LOGGER.info('Turning on Google Cloud Profiler.')
    else:
      raise RuntimeError('Unable to find the job id or job name from envvar.')
  except Exception as e:  # pylint: disable=broad-except
    _LOGGER.warning(
        'Unable to start google cloud profiler due to error: %s. For how to '
        'enable Cloud Profiler with Dataflow see '
        'https://cloud.google.com/dataflow/docs/guides/profiling-a-pipeline.'
        'For troubleshooting tips with Cloud Profiler see '
        'https://cloud.google.com/profiler/docs/troubleshooting.' % e)


def _get_gcp_profiler_name_if_enabled(sdk_pipeline_options):
  gcp_profiler_service_name = sdk_pipeline_options.view_as(
      GoogleCloudOptions).get_cloud_profiler_service_name()

  return gcp_profiler_service_name


def main(unused_argv):
  """Main entry point for SDK Fn Harness."""

View on GitHub (pinned to 12126d8942)