{"record":{"id":"a5e6419bf508db84","repo":"unclecode/crawl4ai","slug":"priority-queue-empty","errorCode":null,"errorMessage":"Priority queue empty","messagePattern":"Priority queue empty","errorType":"exception","errorClass":"IndexError","httpStatus":null,"severity":"warning","filePath":"crawl4ai/deep_crawling/crazy.py","lineNumber":89,"sourceCode":"    def __init__(self):\n        self._heap: List[Tuple[PriorityT, float, P]] = []\n        self._index: Dict[P, int] = {}\n\n    def insert(self, priority: PriorityT, item: P) -> None:\n        tiebreaker = time.time()  # Ensure FIFO for equal priorities\n        heappush(self._heap, (priority, tiebreaker, item))\n        self._index[item] = len(self._heap) - 1\n\n    def extract(self, top_n = 1) -> P:\n        items = []\n        for _ in range(top_n):\n            if not self._heap:\n                break\n            priority, _, item = heappop(self._heap)\n            del self._index[item]\n            items.append(item)\n        if not items:\n            raise IndexError(\"Priority queue empty\")\n        return items\n        # while self._heap:\n        #     _, _, item = heappop(self._heap)\n        #     if item in self._index:\n        #         del self._index[item]\n        #         return item\n        raise IndexError(\"Priority queue empty\")\n\n\n    def is_empty(self) -> bool:\n        return not bool(self._heap)\n\nclass BloomFilter:\n    \"\"\"Optimal Bloom filter using murmur3 hash avalanche\"\"\"\n    __slots__ = ('size', 'hashes', 'bits')\n\n    def __init__(self, capacity: int, error_rate: float):\n        self.size = self._optimal_size(capacity, error_rate)","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/unclecode/crawl4ai/blob/7e801521428ee12509994d39151006f64055ebe3/crawl4ai/deep_crawling/crazy.py#L71-L107","documentation":"Raised byPriorityQueue.extract(top_n) in crawl4ai/deep_crawling/crazy.py when the internal heap holds no items at extraction time. extract() pops up to top_n entries and raises IndexError if it collected none. In best-first crawling this means the crawl frontier is exhausted (no pending URLs) at the moment of extraction.","triggerScenarios":"Calling extract() on a queue whose heap is empty, or with top_n larger than the remaining item count after a prior extract drained it. Also reachable when a crawl loop keeps extracting after all links have been visited or after cancel_event stops adding new links.","commonSituations":"Custom best-first crawl loops that do not check is_empty() between iterations; concurrent shutdown where the cancel event empties pending work while another task extracts; small sites where the frontier runs dry before the expected page budget is met.","solutions":["Guard every extraction with if queue.is_empty(): break before calling extract()","Catch IndexError around extract() and treat it as a normal end-of-crawl condition","If you expect items, verify links are actually being added (check your link_filter/score getter) — an over-restrictive filter can leave the queue permanently empty"],"exampleFix":"// before\nitems = queue.extract(top_n=5)  # IndexError when frontier exhausted\n\n// after\nif queue.is_empty():\n    break\nitems = queue.extract(top_n=5)","handlingStrategy":"type-guard","validationCode":"if queue.is_empty():\n    break  # frontier exhausted, end crawl normally\nitems = queue.extract(top_n=5)","typeGuard":"def can_extract(queue, top_n: int = 1) -> bool:\n    \"\"\"True when the queue has at least one item to extract.\"\"\"\n    return not queue.is_empty()","tryCatchPattern":"try:\n    items = queue.extract(top_n=5)\nexcept IndexError:\n    items = []  # treat empty frontier as normal termination","preventionTips":["Check is_empty() before every extract() in crawl loops","Treat IndexError from extract as end-of-frontier, not a bug","After cancellation, drain or abandon the queue rather than extracting from it"],"tags":["deep-crawling","priority-queue","frontier-exhausted"],"backgroundTag":null,"analyzedSha":"7e801521428ee12509994d39151006f64055ebe3","analyzedAt":"2026-08-14T20:46:20.673Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}