tursodatabase/turso · error · ValueError

: each run must contain the same series and all six query…

Error message

{path}: each run must contain the same series and all six query cases

What it means

Within read_runs, every run must expose the same set of series and each series must cover all six entries in the QUERIES constant. This ValueError is raised when a run's series set deviates from the first run's, or any series is missing query cases, so per-run bars/lines would be incomplete or inconsistent.

Solutions

Rerun the affected benchmark file so every run contains all series and all six query cases.
Remove the incomplete run(s) from the CSV before plotting.
Ensure the benchmark harness always writes the full engine/mode x query matrix per run.

Example fix

# before: csv has runs missing the 'regex' engine series
# after: regenerate so every run has every series for all six QUERIES
python bench-fts.py --engines tursodefault,tursofts,regex --queries all

Defensive patterns

Strategy: validation

Validate before calling

import csv
rows = list(csv.DictReader(open('results/run1.csv')))
runs = {}
for row in rows:
    runs.setdefault(row['run'], {}).setdefault(f"{row['engine']}/{row['mode']}", set()).add(row['query'])
for run, series in runs.items():
    assert len({frozenset(s) for s in series.values()}) == 1, f'run {run}: series cover different query sets'
    assert all(s == set(QUERIES) for s in series.values()), f'run {run}: missing query cases'

Try / catch

try:
    config, samples = read_runs(paths)
except ValueError as e:
    if 'same series and all six query cases' in str(e):
        print(f'Fix: {e} - rerun or drop the incomplete benchmark file')
    else:
        raise

Prevention

Let interrupted benchmark runs write to a temp file and only publish complete runs.
Keep the engine/mode list fixed for all runs in one results file.
Validate each results CSV for the full series x query matrix before plotting.

When it happens

Trigger: A benchmark CSV whose rows for a run omit some engine/mode series or miss one or more of the six query cases; partially completed benchmark runs appended to a CSV.

Common situations: A benchmark run interrupted midway leaving missing query rows; engine renamed/removed between runs in the same file; hand-edited CSV dropping rows.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of tursodatabase/turso@8d4a589f8d (2026-09-20). Data as JSON: /api/errors/d50557ca8f1071d6. Report an issue: GitHub.

Appendix: source

Thrown at perf/fts/plot/plot-fts.py:82

        raise ValueError("no sweep results found")
    return configuration, dict(sorted(points.items()))


def read_runs(paths, percentile="p50"):
    configuration = None
    samples = {}
    expected_series = None
    for path in paths:
        current, runs = read_file(path, percentile, samples)
        if configuration is None:
            configuration = current
        if configuration != current:
            raise ValueError("plot only one configuration at a time")
        for run in runs.values():
            if expected_series is None:
                expected_series = set(run)
            if set(run) != expected_series or any(seen != set(QUERIES) for seen in run.values()):
                raise ValueError(f"{path}: each run must contain the same series and all six query cases")
    if configuration is None:
        raise ValueError("no results found")
    configuration = dict(zip(CONFIGURATION, configuration))
    configuration["percentile"] = percentile
    return configuration, samples


def read_file(path, percentile, samples):
    configuration = None
    runs = {}
    with path.open(newline="") as stream:
        reader = csv.DictReader(stream)
        required = set(CONFIGURATION) | {"engine", "mode", "run", "query", "queries"}
        if not required <= set(reader.fieldnames or []):
            raise ValueError(f"{path}: missing benchmark columns")
        for row in reader:
            if row.get("profiled", "false") != "false":
                raise ValueError("profiled timings must not be used for benchmark comparisons")

View on GitHub (pinned to 8d4a589f8d)