{"record":{"id":"d50557ca8f1071d6","repo":"tursodatabase/turso","slug":"path-each-run-must-contain-the-same-series-and-all-six-query","errorCode":null,"errorMessage":"{path}: each run must contain the same series and all six query cases","messagePattern":"(.+?): each run must contain the same series and all six query cases","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"perf/fts/plot/plot-fts.py","lineNumber":82,"sourceCode":"        raise ValueError(\"no sweep results found\")\n    return configuration, dict(sorted(points.items()))\n\n\ndef read_runs(paths, percentile=\"p50\"):\n    configuration = None\n    samples = {}\n    expected_series = None\n    for path in paths:\n        current, runs = read_file(path, percentile, samples)\n        if configuration is None:\n            configuration = current\n        if configuration != current:\n            raise ValueError(\"plot only one configuration at a time\")\n        for run in runs.values():\n            if expected_series is None:\n                expected_series = set(run)\n            if set(run) != expected_series or any(seen != set(QUERIES) for seen in run.values()):\n                raise ValueError(f\"{path}: each run must contain the same series and all six query cases\")\n    if configuration is None:\n        raise ValueError(\"no results found\")\n    configuration = dict(zip(CONFIGURATION, configuration))\n    configuration[\"percentile\"] = percentile\n    return configuration, samples\n\n\ndef read_file(path, percentile, samples):\n    configuration = None\n    runs = {}\n    with path.open(newline=\"\") as stream:\n        reader = csv.DictReader(stream)\n        required = set(CONFIGURATION) | {\"engine\", \"mode\", \"run\", \"query\", \"queries\"}\n        if not required <= set(reader.fieldnames or []):\n            raise ValueError(f\"{path}: missing benchmark columns\")\n        for row in reader:\n            if row.get(\"profiled\", \"false\") != \"false\":\n                raise ValueError(\"profiled timings must not be used for benchmark comparisons\")","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/tursodatabase/turso/blob/8d4a589f8d13ac184700d2a8f724f27e1995be3b/perf/fts/plot/plot-fts.py#L64-L100","documentation":"Within read_runs, every run must expose the same set of series and each series must cover all six entries in the QUERIES constant. This ValueError is raised when a run's series set deviates from the first run's, or any series is missing query cases, so per-run bars/lines would be incomplete or inconsistent.","triggerScenarios":"A benchmark CSV whose rows for a run omit some engine/mode series or miss one or more of the six query cases; partially completed benchmark runs appended to a CSV.","commonSituations":"A benchmark run interrupted midway leaving missing query rows; engine renamed/removed between runs in the same file; hand-edited CSV dropping rows.","solutions":["Rerun the affected benchmark file so every run contains all series and all six query cases.","Remove the incomplete run(s) from the CSV before plotting.","Ensure the benchmark harness always writes the full engine/mode x query matrix per run."],"exampleFix":"# before: csv has runs missing the 'regex' engine series\n# after: regenerate so every run has every series for all six QUERIES\npython bench-fts.py --engines tursodefault,tursofts,regex --queries all","handlingStrategy":"validation","validationCode":"import csv\nrows = list(csv.DictReader(open('results/run1.csv')))\nruns = {}\nfor row in rows:\n    runs.setdefault(row['run'], {}).setdefault(f\"{row['engine']}/{row['mode']}\", set()).add(row['query'])\nfor run, series in runs.items():\n    assert len({frozenset(s) for s in series.values()}) == 1, f'run {run}: series cover different query sets'\n    assert all(s == set(QUERIES) for s in series.values()), f'run {run}: missing query cases'","typeGuard":null,"tryCatchPattern":"try:\n    config, samples = read_runs(paths)\nexcept ValueError as e:\n    if 'same series and all six query cases' in str(e):\n        print(f'Fix: {e} - rerun or drop the incomplete benchmark file')\n    else:\n        raise","preventionTips":["Let interrupted benchmark runs write to a temp file and only publish complete runs.","Keep the engine/mode list fixed for all runs in one results file.","Validate each results CSV for the full series x query matrix before plotting."],"tags":["python","benchmarking","data-completeness"],"backgroundTag":"schema-validation-failed","analyzedSha":"8d4a589f8d13ac184700d2a8f724f27e1995be3b","analyzedAt":"2026-09-20T13:18:14.658Z","contentChangedAt":"2026-09-20T13:18:14.658Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}