tursodatabase/turso · error · ValueError

unknown benchmark

Error message

unknown benchmark

What it means

read_measurement encountered a value in the benchmark column it does not recognize, so it cannot interpret the row's measurement (seconds vs peak_heap_bytes vs search-count rules). This is a generic validation guard: the CSV's benchmark column holds a value other than the supported 'memory' and 'search' kinds, usually a typo or a new benchmark kind the plotter was not updated for.

Solutions

1. Fix the benchmark value in the CSV to one of memory|search|throughput
2. Add handling for the new benchmark kind in read_measurement()
3. Use the correct plotting tool for that benchmark

Example fix

// before (CSV)
benchmark=latency,...
// after
benchmark=search,...

KNOWN_BENCHMARKS = {"memory", "search", "throughput"}
def validate_benchmark(row):
    if row["benchmark"] not in KNOWN_BENCHMARKS:
        raise ValueError(f"unknown benchmark {row['benchmark']}")

def is_known_benchmark(row) -> bool:
    return row.get("benchmark") in {"memory", "search", "throughput"}

try:
    configuration, runs = read_runs(path, percentile)
except ValueError as e:
    print(f"unknown benchmark kind: {e}")

When it happens

Trigger: CSV contains benchmark='latency' or a typo like 'thruput'; a new benchmark kind added by the harness but not the plot script.

Common situations: Renamed benchmark in the collector script; hand-edited CSV; plotting an unrelated benchmark's output with this tool.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• unsupported engine/mode
• heap results require Turso and the requested queries per…
• measurements must be finite and positive
• no query finished on any engine
• no results found

AI-assisted analysis of tursodatabase/turso@8d4a589f8d (2026-09-20). Data as JSON: /api/errors/b57560eea4db8d28. Report an issue: GitHub.

Appendix: source

Thrown at perf/fts/plot/plot-fts.py:140

    queries = int(row["queries"])
    positive_number(queries)
    if row["benchmark"] != "memory":
        seconds = positive_number(row["seconds"])
    if row["benchmark"] == "memory":
        if row["engine"] != "turso" or queries != int(row["requested_queries"]) * int(row["connections"]):
            raise ValueError("heap results require Turso and the requested queries per connection")
        value = float(row["peak_heap_bytes"]) / (1024 * 1024)
    elif row["benchmark"] == "search":
        if not 0 < int(row["connections"]) <= queries or queries != int(row["requested_queries"]):
            raise ValueError("search results require at least one sample per connection and the requested query count")
        value = float(row[f"{percentile}_ms"])
    elif row["benchmark"] == "throughput":
        minimum = positive_number(row["min_seconds"])
        if seconds < minimum:
            raise ValueError("throughput results must meet a positive minimum duration")
        value = queries / seconds
    else:
        raise ValueError("unknown benchmark")
    return positive_number(value)


def positive_number(value):
    value = float(value)
    if not np.isfinite(value) or value <= 0:
        raise ValueError("measurements must be finite and positive")
    return value


def plot_sweep(configuration, points, output, relative=False):
    import matplotlib

    matplotlib.use("Agg")
    import matplotlib.pyplot as plt
    from matplotlib.ticker import FuncFormatter, MaxNLocator, NullLocator

    importlib.import_module("scienceplots")

View on GitHub (pinned to 8d4a589f8d)