{"record":{"id":"bf4e184c3e582f12","repo":"tursodatabase/turso","slug":"sweep-files-must-match-benchmark-sample-budget-fixture-and","errorCode":null,"errorMessage":"sweep files must match benchmark, sample budget, fixture and engine/mode series","messagePattern":"sweep files must match benchmark, sample budget, fixture and engine/mode series","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"perf/fts/plot/plot-fts.py","lineNumber":61,"sourceCode":"        if args.sweep:\n            plot_sweep(configuration, samples, output, args.relative)\n        else:\n            plot(configuration, samples, output)\n        print(f\"wrote {output}\")\n\n\ndef read_sweep(paths, percentile=\"p50\"):\n    configuration = None\n    points = {}\n    for path in paths:\n        current, samples = read_runs([path], percentile)\n        connections = int(current.pop(\"connections\"))\n        if connections <= 0 or connections in points:\n            raise ValueError(\"each sweep file must have a distinct positive connection count\")\n        if configuration is None:\n            configuration = current\n        if current != configuration or (points and set(samples) != set(next(iter(points.values())))):\n            raise ValueError(\"sweep files must match benchmark, sample budget, fixture and engine/mode series\")\n        points[connections] = samples\n    if not points:\n        raise ValueError(\"no sweep results found\")\n    return configuration, dict(sorted(points.items()))\n\n\ndef read_runs(paths, percentile=\"p50\"):\n    configuration = None\n    samples = {}\n    expected_series = None\n    for path in paths:\n        current, runs = read_file(path, percentile, samples)\n        if configuration is None:\n            configuration = current\n        if configuration != current:\n            raise ValueError(\"plot only one configuration at a time\")\n        for run in runs.values():\n            if expected_series is None:","sourceCodeStart":43,"sourceCodeEnd":79,"githubUrl":"https://github.com/tursodatabase/turso/blob/8d4a589f8d13ac184700d2a8f724f27e1995be3b/perf/fts/plot/plot-fts.py#L43-L79","documentation":"After reading each sweep file, read_sweep compares its remaining configuration tuple and sample series against the first file's. It raises this ValueError when files disagree on benchmark configuration (benchmark, sample budget, fixture) or on the set of engine/mode series, since a combined connection-sweep plot would be meaningless across different setups.","triggerScenarios":"Calling read_sweep with CSVs generated by different plot-fts configurations (different benchmark, percentile/sample budget, fixture) or containing different engine/mode columns than the first file.","commonSituations":"Mixing result files from before and after a plot-fts/benchmark change; plotting sweep files generated against different database fixtures; renaming engines in one run but not others.","solutions":["Regenerate all sweep files with the same benchmark, sample budget, fixture, and engine/mode set.","Remove the sweep files that came from a different configuration.","Align the series by rerunning the missing engine/mode combinations so every file has identical series."],"exampleFix":"# before\npython plot-fts.py sweep-old-fixture/c8.csv sweep-new-fixture/c16.csv\n# after\npython plot-fts.py sweep-v2/c8.csv sweep-v2/c16.csv","handlingStrategy":"validation","validationCode":"import csv\ndef config_of(p):\n    r = csv.DictReader(open(p))\n    return tuple(next(r)[k] for k in CONFIGURATION)\nassert len({config_of(p) for p in paths}) == 1, 'all sweep files must share one configuration'","typeGuard":null,"tryCatchPattern":"try:\n    config, points = read_sweep(paths)\nexcept ValueError as e:\n    if 'must match benchmark' in str(e):\n        print('Fix: regenerate all sweep files with the same benchmark/fixture/series')\n    else:\n        raise","preventionTips":["Keep results from different benchmark configurations in separate directories.","Regenerate the full sweep set whenever the fixture, sample budget, or engine list changes.","Never mix sweep files generated before and after a plot-fts schema change."],"tags":["python","benchmarking","input-validation"],"backgroundTag":"invalid-argument-value","analyzedSha":"8d4a589f8d13ac184700d2a8f724f27e1995be3b","analyzedAt":"2026-09-20T13:18:14.658Z","contentChangedAt":"2026-09-20T13:18:14.658Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}