t8y2/dbx · error · ValueError

BENCH_ORDER must contain exactly

Error message

BENCH_ORDER must contain exactly: {','.join(commands)}

What it means

benchmark_order parses the BENCH_ORDER environment variable, which must list exactly the same benchmark names as the available commands (by default jdbc,go). If the comma-separated list doesn't match the command set exactly, it raises ValueError naming the required set. This exists to let users reorder benchmarks without silently dropping or duplicating one.

Solutions

1. Set BENCH_ORDER to exactly the required names in the desired order, e.g. BENCH_ORDER=go,jdbc.
2. Unset BENCH_ORDER to use the default 'jdbc,go'.
3. Check the valid keys by inspecting the commands dict / error message listing the expected set.
4. Update stale CI configs or docs to include any newly added benchmark names.

Example fix

# before
BENCH_ORDER=jdbc python run.py
# after
BENCH_ORDER=jdbc,go python run.py

import os
names = [n.strip() for n in os.getenv("BENCH_ORDER", "jdbc,go").split(",") if n.strip()]
assert set(names) == {"jdbc", "go"}, f"BENCH_ORDER must be a permutation of jdbc,go, got {names}"

try:
    order = benchmark_order(commands)
except ValueError as e:
    print(e); order = list(commands)  # fall back to default order

When it happens

Trigger: Setting BENCH_ORDER to something other than a permutation of the command keys — e.g. BENCH_ORDER=jdbc (missing go), BENCH_ORDER="jdbc, go, extra" (extra name), duplicates, or empty entries leading to count mismatch.

Common situations: Typo in a benchmark name; copying BENCH_ORDER from an older script version when a new benchmark was added; trailing/extra commas producing empty names; whitespace handled but misspelled names not.

Understand the failure class

Background: "is not a valid" / "Invalid ... value" environment variable errors: how libraries validate env vars and what to do when they reject yours — this error's family across 48 libraries.

Related errors

• must be positive
• BENCH_CANDIDATES selected no candidates
• FileNotFoundError(path)
• must be a boolean
• must be positive

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/3d087e8926e8c5a6. Report an issue: GitHub.

Appendix: source

Thrown at agents/drivers/iotdb/bench/run.py:227

        "rounds": rounds,
    }


def percentile(values: list[float], fraction: float) -> float:
    index = max(0, min(len(values) - 1, int(len(values) * fraction + 0.999999) - 1))
    return values[index]


def ensure_artifacts() -> None:
    for path in (JDBC_JAR, GO_BINARY):
        if not path.is_file():
            raise FileNotFoundError(path)


def benchmark_order(commands: dict[str, list[str]]) -> list[str]:
    names = [name.strip() for name in os.getenv("BENCH_ORDER", "jdbc,go").split(",") if name.strip()]
    if len(names) != len(commands) or set(names) != set(commands):
        raise ValueError(f"BENCH_ORDER must contain exactly: {','.join(commands)}")
    return names


def env_int(name: str, fallback: int) -> int:
    value = int(os.getenv(name, str(fallback)))
    if value <= 0:
        raise ValueError(f"{name} must be positive")
    return value


def env_float(name: str, fallback: float) -> float:
    value = float(os.getenv(name, str(fallback)))
    if value <= 0:
        raise ValueError(f"{name} must be positive")
    return value


def env_bool(name: str, fallback: bool) -> bool:

View on GitHub (pinned to c0390bff16)