unclecode/crawl4ai · warning · IndexError

Priority queue empty

Error message

Priority queue empty

What it means

Raised byPriorityQueue.extract(top_n) in crawl4ai/deep_crawling/crazy.py when the internal heap holds no items at extraction time. extract() pops up to top_n entries and raises IndexError if it collected none. In best-first crawling this means the crawl frontier is exhausted (no pending URLs) at the moment of extraction.

Source

Thrown at crawl4ai/deep_crawling/crazy.py:89

    def __init__(self):
        self._heap: List[Tuple[PriorityT, float, P]] = []
        self._index: Dict[P, int] = {}

    def insert(self, priority: PriorityT, item: P) -> None:
        tiebreaker = time.time()  # Ensure FIFO for equal priorities
        heappush(self._heap, (priority, tiebreaker, item))
        self._index[item] = len(self._heap) - 1

    def extract(self, top_n = 1) -> P:
        items = []
        for _ in range(top_n):
            if not self._heap:
                break
            priority, _, item = heappop(self._heap)
            del self._index[item]
            items.append(item)
        if not items:
            raise IndexError("Priority queue empty")
        return items
        # while self._heap:
        #     _, _, item = heappop(self._heap)
        #     if item in self._index:
        #         del self._index[item]
        #         return item
        raise IndexError("Priority queue empty")


    def is_empty(self) -> bool:
        return not bool(self._heap)

class BloomFilter:
    """Optimal Bloom filter using murmur3 hash avalanche"""
    __slots__ = ('size', 'hashes', 'bits')

    def __init__(self, capacity: int, error_rate: float):
        self.size = self._optimal_size(capacity, error_rate)

View on GitHub (pinned to 7e80152142)

Solutions

1. Guard every extraction with if queue.is_empty(): break before calling extract()
2. Catch IndexError around extract() and treat it as a normal end-of-crawl condition
3. If you expect items, verify links are actually being added (check your link_filter/score getter) — an over-restrictive filter can leave the queue permanently empty

Example fix

// before
items = queue.extract(top_n=5)  # IndexError when frontier exhausted

// after
if queue.is_empty():
    break
items = queue.extract(top_n=5)

if queue.is_empty():
    break  # frontier exhausted, end crawl normally
items = queue.extract(top_n=5)

def can_extract(queue, top_n: int = 1) -> bool:
    """True when the queue has at least one item to extract."""
    return not queue.is_empty()

try:
    items = queue.extract(top_n=5)
except IndexError:
    items = []  # treat empty frontier as normal termination

When it happens

Trigger: Calling extract() on a queue whose heap is empty, or with top_n larger than the remaining item count after a prior extract drained it. Also reachable when a crawl loop keeps extracting after all links have been visited or after cancel_event stops adding new links.

Common situations: Custom best-first crawl loops that do not check is_empty() between iterations; concurrent shutdown where the cancel event empties pending work while another task extracts; small sites where the frontier runs dry before the expected page budget is met.

Related errors

• CrawlerRunConfig must be provided
• CrawlerRunConfig must be provided

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/a5e6419bf508db84. Report an issue: GitHub.