ZhuLinsen/daily_stock_analysis · warning · TimeoutError

{task_name} ranking fetch in-flight limit reached

Error message

{task_name} ranking fetch in-flight limit reached

What it means

TimeoutError(f'{task_name} ranking fetch in-flight limit reached') is raised by MarketHotspotService._get_or_submit_ranking_fetch when the non-blocking acquire of _ranking_fetch_slots fails — the global cap on concurrent ranking fetch worker threads is exhausted. The semaphore is only released when futures complete or are forgotten, so the cap reflects genuinely busy workers plus any not-yet-reaped finished ones.

Source

Thrown at src/services/market_hotspot_service.py:486

                        f"{task_name} ranking fetch cooling down after previous timeout"
                    )
                cls._ranking_fetch_retry_after.pop(inflight_key, None)
                if cls._ranking_fetch_futures.get(inflight_key) is retry_future:
                    cls._ranking_fetch_futures.pop(inflight_key, None)
                    if retry_future.done() or retry_future.cancelled():
                        cls._ranking_fetch_slots.release()
                    else:
                        cls._ranking_fetch_detached_futures.add(retry_future)

            current = cls._ranking_fetch_futures.get(inflight_key)
            if current is not None:
                if not current.done():
                    return current
                cls._ranking_fetch_futures.pop(inflight_key, None)
                cls._ranking_fetch_slots.release()

            if not cls._ranking_fetch_slots.acquire(blocking=False):
                raise TimeoutError(f"{task_name} ranking fetch in-flight limit reached")

            future: Future = Future()
            cls._ranking_fetch_retry_after.pop(inflight_key, None)
            cls._ranking_fetch_futures[inflight_key] = future
            future.add_done_callback(
                lambda done_future: cls._forget_ranking_fetch(inflight_key, done_future)
            )
            worker = threading.Thread(
                target=cls._run_ranking_fetch,
                args=(future, task),
                daemon=True,
                name=f"market-hotspot-{task_name}",
            )
            submitted = future
        try:
            worker.start()
        except BaseException as exc:
            cls._drop_unstarted_ranking_fetch(inflight_key, submitted)

View on GitHub (pinned to 5159bd72e8)

Solutions

1. Reduce fan-out: stagger or batch ranking fetches so concurrent distinct fetches stay under the slot limit
2. Check for stuck workers (thread named 'market-hotspot-<task>') and hanging sockets; ensure upstream requests have their own socket timeout
3. Retry shortly after — completed futures release slots via _forget_ranking_fetch, so the limit is transient
4. If the workload legitimately needs more parallelism, raise the ranking fetch slot configuration rather than bypassing the semaphore

Example fix

# before: all markets fetched at once exhausts slots
results = {m: fetch_ranking(m) for m in ALL_MARKETS}  # raises for some

# after: bounded concurrency
with ThreadPoolExecutor(max_workers=2) as pool:
    results = dict(zip(ALL_MARKETS, pool.map(fetch_ranking, ALL_MARKETS)))

try:
    data = fetch_ranking(key)
except TimeoutError as exc:
    if "in-flight limit reached" in str(exc):
        time.sleep(0.1)  # slots free as futures complete
        data = fetch_ranking(key)

When it happens

Trigger: Issuing more distinct ranking fetches concurrently than the configured slot count (each unique inflight_key holds a slot until done), e.g. fanning out all markets/boards at once; or slots leaked by detached futures that never complete.

Common situations: A burst scheduler requesting all rankings simultaneously at market open, retry storms after partial timeouts holding slots, or worker threads stuck on a hanging upstream connection never releasing their slot.

Related errors

• {task_name} ranking fetch timeout after {timeout_value:g}s
• {task_name} ranking fetch cooling down after previous timeou
• 题材新闻搜索等待超过调用截止时间
• {task_name} ranking fetch timeout
• request_conflict

AI-assisted analysis of ZhuLinsen/daily_stock_analysis@5159bd72e8 (2026-08-15). Data as JSON: /api/errors/d1cfe3dfd3ceca6f. Report an issue: GitHub.