{"record":{"id":"805a917e1ace8d76","repo":"tursodatabase/turso","slug":"profiled-timings-must-not-be-used-for-benchmark-comparisons","errorCode":null,"errorMessage":"profiled timings must not be used for benchmark comparisons","messagePattern":"profiled timings must not be used for benchmark comparisons","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"perf/fts/plot/plot-fts.py","lineNumber":100,"sourceCode":"                raise ValueError(f\"{path}: each run must contain the same series and all six query cases\")\n    if configuration is None:\n        raise ValueError(\"no results found\")\n    configuration = dict(zip(CONFIGURATION, configuration))\n    configuration[\"percentile\"] = percentile\n    return configuration, samples\n\n\ndef read_file(path, percentile, samples):\n    configuration = None\n    runs = {}\n    with path.open(newline=\"\") as stream:\n        reader = csv.DictReader(stream)\n        required = set(CONFIGURATION) | {\"engine\", \"mode\", \"run\", \"query\", \"queries\"}\n        if not required <= set(reader.fieldnames or []):\n            raise ValueError(f\"{path}: missing benchmark columns\")\n        for row in reader:\n            if row.get(\"profiled\", \"false\") != \"false\":\n                raise ValueError(\"profiled timings must not be used for benchmark comparisons\")\n            current = tuple(row[key] for key in CONFIGURATION)\n            if configuration is None:\n                configuration = current\n            if configuration != current:\n                raise ValueError(\"plot only one configuration at a time\")\n            key = (row[\"engine\"], row[\"mode\"])\n            if key not in SERIES:\n                raise ValueError(f\"unsupported engine/mode: {key}\")\n            series = samples.setdefault(key, {query: [] for query in QUERIES})\n            query = row[\"query\"]\n            seen = runs.setdefault(row[\"run\"], {}).setdefault(key, set())\n            if query not in QUERIES or query in seen:\n                raise ValueError(f\"{path}: unknown or duplicate query {query}\")\n            seen.add(query)\n            series[query].append(read_measurement(row, percentile))\n    if not runs:\n        raise ValueError(f\"{path}: no results found\")\n    return configuration, runs","sourceCodeStart":82,"sourceCodeEnd":118,"githubUrl":"https://github.com/tursodatabase/turso/blob/8d4a589f8d13ac184700d2a8f724f27e1995be3b/perf/fts/plot/plot-fts.py#L82-L118","documentation":"read_file() validates every CSV row produced by the FTS benchmark harness before plotting. If a row has profiled=true, its timings were collected under a profiler and are not comparable to clean runs, so the script refuses to plot them. This protects users from publishing skewed benchmark comparisons.","triggerScenarios":"A results CSV whose 'profiled' column is anything other than 'false' is passed to read_runs()/read_file(), e.g. plotting output collected with the profiler enabled.","commonSituations":"Developer forgot to disable profiling when generating benchmark results; a mixed CSV contains both profiled and clean rows; a stale results file from a profiling session was reused for plotting.","solutions":["Regenerate the benchmark CSV with profiling disabled so the 'profiled' column is false for all rows","Remove profiled rows from the CSV, keeping only rows where profiled=false","Keep profiled data separate and plot it with a different (profiling-aware) tool"],"exampleFix":"// before (CSV row)\nengine,turso,...,profiled=true\n// after\nengine,turso,...,profiled=false","handlingStrategy":"validation","validationCode":"def validate_not_profiled(row):\n    if row.get(\"profiled\", \"false\") != \"false\":\n        raise ValueError(\"profiled timings must not be used for benchmark comparisons\")","typeGuard":"def is_clean_timing(row) -> bool:\n    return row.get(\"profiled\", \"false\") == \"false\"","tryCatchPattern":"try:\n    configuration, runs = read_runs(path, percentile)\nexcept ValueError as e:\n    print(f\"skipping {path}: {e}\")","preventionTips":["Always generate benchmark CSVs with profiling disabled","Keep profiled runs in separate files from comparison runs","Check the profiled column before sharing results files"],"tags":["python","benchmark","validation"],"backgroundTag":"invalid-config-value","analyzedSha":"8d4a589f8d13ac184700d2a8f724f27e1995be3b","analyzedAt":"2026-09-20T13:18:14.658Z","contentChangedAt":"2026-09-20T13:18:14.658Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}