oracle/graal · error · RuntimeError

Suite runs only a single benchmark, got: {benchmarks}

Error message

Suite runs only a single benchmark, got: {benchmarks}

What it means

Same single-benchmark constraint as error 803, but raised when more than one benchmark was supplied. DaCapo runs one benchmark per JVM invocation, so a list like ['fop','lusearch'] cannot be turned into one command line.

Source

Thrown at sdk/mx.sdk/mx_sdk_benchmark.py:2640

    def dataLocation(self):
        if self.version() == "23.11-MR2-chopin":
            basePath = self.daCapoPath()
            subdir = "dacapo-23.11-MR2-chopin"
            if self.minimalArchive():
                subdir += "-minimal"
            return os.path.join(basePath, subdir)
        else:
            raise RuntimeError(f"data location is not supported for suite version '{self.version()}'")

    def minimalArchive(self):
        return False

    def createCommandLineArgs(self, benchmarks, bmSuiteArgs):
        if benchmarks is None:
            raise RuntimeError(
                "Suite runs only a single benchmark.")
        if len(benchmarks) != 1:
            raise RuntimeError(
                f"Suite runs only a single benchmark, got: {benchmarks}")

        benchmark = benchmarks[0]
        runArgs = self.postprocessRunArgs(benchmark, self.runArgs(bmSuiteArgs))
        if runArgs is None:
            return None

        jarPath = self.jarPath(benchmark)
        return self.vmArgs(bmSuiteArgs) + ["-jar"] + [jarPath, benchmark] + runArgs

    def benchmarkList(self, bmSuiteArgs):
        missing_sizes = set(self.daCapoIterations().keys()).difference(set(self.daCapoSizes().keys()))
        if len(missing_sizes) > 0:
            mx.abort(f"Missing size definitions for benchmark(s): {missing_sizes}")
        return [b for b, it in self.daCapoIterations().items()
                if self.workloadSize() in self.daCapoSizes().get(b, []) and it != -1]

    def successPatterns(self):

View on GitHub (pinned to a66e9ccd1d)

Solutions

Split into one mx benchmark invocation per benchmark.
Use mx's bench-suite-level repetition ('--repeat' or the benchmark runner's loop) rather than bundling benchmarks into one call.
Check for trailing commas or empty entries in the selector that may parse as two benchmarks.

Example fix

# before
mx benchmark dacapo-chopin:fop,lusearch

# after
mx benchmark dacapo-chopin:fop && mx benchmark dacapo-chopin:lusearch

Defensive patterns

Strategy: validation

Validate before calling

assert len(benchmarks) == 1, (
    f"DaCapo runs one benchmark per JVM: got {benchmarks}; "
    "split into separate invocations.")

Prevention

Loop over benchmarks one mx invocation at a time instead of passing lists.
Reject multi-value selectors early in wrapper scripts.

When it happens

Trigger: Passing multiple benchmarks to the DaCapo suite, e.g. 'mx benchmark dacapo-chopin:fop,lusearch' or a script that forwards a benchmark list.

Common situations: Users used to suites that accept comma-separated benchmark lists; harnesses that aggregate several benchmarks into one mx call.

Related errors

AI-assisted analysis of oracle/graal@a66e9ccd1d (2026-08-14). Data as JSON: /api/errors/58b007cca6d904e9. Report an issue: GitHub.