locustio/locust · error · ValueError

Ground truth length is less than limit: {len(ground_truth)}

Error message

Ground truth length is less than limit: {len(ground_truth)} < {limit}

What it means

MilvusUser's recall calculation in get_recall raises ValueError when the ground-truth ID list is shorter than the requested limit, because slicing ground_truth[:limit] would produce a misleading (too small) denominator for recall. It is a guard against computing recall with insufficient ground truth data.

Source

Thrown at locust/contrib/milvus.py:218

    def get_recall(search_results, ground_truth, limit=None):
        """Calculate recall for V2 client search results."""
        try:
            # Extract IDs from V2 search results
            retrieved_ids = []
            if isinstance(search_results, list) and len(search_results) > 0:
                # search_results[0] contains the search results for the first query
                for hit in search_results[0] if isinstance(search_results[0], list) else search_results:
                    if isinstance(hit, dict) and "id" in hit:
                        retrieved_ids.append(hit["id"])
                    elif hasattr(hit, "get"):
                        retrieved_ids.append(hit.get("id"))

            # Apply limit if specified
            if limit is None:
                limit = len(retrieved_ids)

            if len(ground_truth) < limit:
                raise ValueError(f"Ground truth length is less than limit: {len(ground_truth)} < {limit}")

            # Calculate recall
            ground_truth_set = set(ground_truth[:limit])
            retrieved_set = set(retrieved_ids)
            intersect = len(ground_truth_set.intersection(retrieved_set))
            return intersect / len(ground_truth_set)

        except Exception:
            return 0.0

    def query(self, filter, output_fields=None):
        if output_fields is None:
            output_fields = ["id"]

        start = time.time()
        try:
            result = self.client.query(
                collection_name=self.collection_name,

View on GitHub (pinned to f391a716e1)

Solutions

Reduce the limit parameter to at most len(ground_truth)
Provide more ground truth IDs so its length >= limit
Pass limit=None so it defaults to len(retrieved_ids) and validate ground truth separately

Example fix

// before
recall = get_recall(ground_truth=["1", "2"], retrieved_ids=ids, limit=10)
// after
limit = min(10, len(ground_truth))
recall = get_recall(ground_truth=ground_truth, retrieved_ids=ids, limit=limit)

Defensive patterns

Strategy: validation

Validate before calling

def validate_recall_inputs(ground_truth, retrieved_ids, limit):
    limit = len(retrieved_ids) if limit is None else limit
    if len(ground_truth) < limit:
        limit = len(ground_truth)  # or raise with a clear message
    return limit

Prevention

Ensure ground truth data covers at least the top-k limit requested
Clamp limit to min(limit, len(ground_truth)) before searching
Generate ground truth from the same corpus size as the searched collection

When it happens

Trigger: Calling search(...) (which calls get_recall) with ground_truth containing fewer IDs than the specified limit parameter.

Common situations: Test datasets with fewer known-relevant results than the top-k limit requested in the search; misconfigured limit values in benchmark locustfiles; ground truth generated from a smaller corpus than the search collection.

Related errors

AI-assisted analysis of locustio/locust@f391a716e1 (2026-08-29). Data as JSON: /api/errors/4987fa4e100e8471. Report an issue: GitHub.