openai/openai-python · error · ValueError

Expected a non-empty value for `eval_id` but received {eval_

Error message

Expected a non-empty value for `eval_id` but received {eval_id!r}

What it means

The sync EvalRun.create() method validates that eval_id is truthy before POSTing to /evals/{eval_id}/runs. The id is a path parameter, so an empty value would produce an invalid URL; the SDK raises ValueError before any request is sent.

Source

Thrown at src/openai/resources/evals/runs/runs.py:104

          metadata: Set of 16 key-value pairs that can be attached to an object. This can be useful
              for storing additional information about the object in a structured format, and
              querying for objects via API or the dashboard.

              Keys are strings with a maximum length of 64 characters. Values are strings with
              a maximum length of 512 characters.

          name: The name of the run.

          extra_headers: Send extra headers

          extra_query: Add additional query parameters to the request

          extra_body: Add additional JSON properties to the request

          timeout: Override the client-level default timeout for this request, in seconds
        """
        if not eval_id:
            raise ValueError(f"Expected a non-empty value for `eval_id` but received {eval_id!r}")
        return self._post(
            path_template("/evals/{eval_id}/runs", eval_id=eval_id),
            body=maybe_transform(
                {
                    "data_source": data_source,
                    "metadata": metadata,
                    "name": name,
                },
                run_create_params.RunCreateParams,
            ),
            options=make_request_options(
                extra_headers=extra_headers,
                extra_query=extra_query,
                extra_body=extra_body,
                timeout=timeout,
                security={"bearer_auth": True},
            ),
            cast_to=RunCreateResponse,

View on GitHub (pinned to 9917c6e28e)

Solutions

Ensure eval_id is the id of an existing eval (from client.evals.create(...).id or client.evals.retrieve(...).id).
Verify the earlier eval-creation call succeeded before starting a run.
Validate config/env values at startup.
Add a guard before create().

Example fix

# before
run = client.evals.runs.create(eval_id=eval_id, data_source=data_source)

# after
if not eval_id:
    raise ValueError(f"eval_id is required, got {eval_id!r}")
run = client.evals.runs.create(eval_id=eval_id, data_source=data_source)

Defensive patterns

Strategy: validation

Validate before calling

if not isinstance(eval_id, str) or not eval_id.strip():
    raise ValueError(f"eval_id must be a non-empty string, got {eval_id!r}")

Type guard

def is_valid_eval_id(eval_id: object) -> bool:
    return isinstance(eval_id, str) and bool(eval_id.strip())

Try / catch

try:
    run = client.evals.runs.create(eval_id=eval_id, data_source=data_source)
except ValueError as e:
    if 'eval_id' in str(e):
        raise ValueError(f'Cannot start run: {e}') from e
    raise

Prevention

Create the eval first and store its id before starting runs.
Validate config-sourced eval ids at startup.
Never pass placeholder strings like '<eval_id>'.

When it happens

Trigger: Calling client.evals.runs.create(eval_id='', data_source=..., ...) (or with None) on the sync client.

Common situations: Eval id read from env/config that is unset, previous eval-creation step failed so the variable stayed empty, or placeholder text left in copied example code.

Related errors

AI-assisted analysis of openai/openai-python@9917c6e28e (2026-08-28). Data as JSON: /api/errors/d1f5abab55ded964. Report an issue: GitHub.