NanmiCoder/MediaCrawler · warning · Exception

[ProxyIpPool.get_proxy] current ip invalid and again get it

Error message

[ProxyIpPool.get_proxy] current ip invalid and again get it

What it means

Raised by ProxyIpPool.get_proxy() when IP validation is enabled and the randomly chosen proxy fails the _is_valid_proxy() connectivity check. Despite the message saying 'again get it', the code does NOT retry — it raises and leaves the caller responsible for calling get_proxy() again. The failed proxy has already been removed from proxy_list.

Source

Thrown at proxy/proxy_ip_pool.py:112

            utils.logger.info(
                f"[ProxyIpPool._is_valid_proxy] testing {proxy.ip} err: {e}"
            )
            raise e

    @retry(stop=stop_after_attempt(3), wait=wait_fixed(1))
    async def get_proxy(self) -> IpInfoModel:
        """
        Randomly extract a proxy IP from the proxy pool
        :return:
        """
        if len(self.proxy_list) == 0:
            await self._reload_proxies()

        proxy = random.choice(self.proxy_list)
        self.proxy_list.remove(proxy)  # Remove an IP once extracted
        if self.enable_validate_ip:
            if not await self._is_valid_proxy(proxy):
                raise Exception(
                    "[ProxyIpPool.get_proxy] current ip invalid and again get it"
                )
        self.current_proxy = proxy  # Save currently used proxy
        return proxy

    def is_current_proxy_expired(self, buffer_seconds: int = 30) -> bool:
        """
        Check if current proxy has expired
        Args:
            buffer_seconds: Buffer time (seconds), how many seconds ahead to consider expired
        Returns:
            bool: True means expired or no current proxy, False means still valid
        """
        if self.current_proxy is None:
            return True
        return self.current_proxy.is_expired(buffer_seconds)

    async def get_or_refresh_proxy(self, buffer_seconds: int = 30) -> IpInfoModel:

View on GitHub (pinned to d6f7c5bb90)

Solutions

  1. Wrap get_proxy() in a retry loop (3-5 attempts) — each call removes the bad IP so the next attempt gets a fresh one
  2. If the pool keeps failing, force a reload: proxy_list empty triggers _reload_proxies() automatically on the next call
  3. Check local egress: can the host open a TCP connection to the proxy ip:port at all
  4. Set enable_validate_ip=False for static providers (the factory already does this) so known-good static IPs are not re-validated

Example fix

// before
proxy = await pool.get_proxy()

// after
for attempt in range(5):
    try:
        proxy = await pool.get_proxy()
        break
    except Exception as e:
        if attempt == 4:
            raise
        utils.logger.warning(f"proxy invalid, retrying ({attempt+1}/5): {e}")
Defensive patterns

Strategy: retry

Try / catch

async def get_valid_proxy(pool, attempts=5):
    for i in range(attempts):
        try:
            return await pool.get_proxy()
        except Exception as e:
            if "ip invalid" not in str(e) or i == attempts - 1:
                raise
            utils.logger.warning(f"invalid proxy, retry {i+1}/{attempts}")
    raise RuntimeError("no valid proxy after retries")

Prevention

When it happens

Trigger: Calling get_proxy() with enable_validate_ip=True while the pool holds dead/expired proxies; provider gave IPs that expired between fetch and use; the local machine cannot reach the proxy network (firewall blocking proxy ports).

Common situations: Long-lived crawler where cached IPs expire mid-run; proxy provider delivering a batch with some dead nodes; corporate firewall blocking outbound SOCKS/HTTP proxy ports; pool nearly empty so a single failure surfaces immediately.

Related errors

AI-assisted analysis of NanmiCoder/MediaCrawler@d6f7c5bb90 (2026-08-15). Data as JSON: /api/errors/3b34b10b421b8f21. Report an issue: GitHub.