unclecode/crawl4ai · error · ValueError

[NSTProxy] Invalid API response — expected a non-empty list

Error message

[NSTProxy] Invalid API response — expected a non-empty list

What it means

Raised when the NSTProxy API returns a 200 response whose JSON body is not a non-empty list. The code expects data = [{ip, port, username, password}, ...] and takes data[0]; anything else (empty list, dict without err, scalar) is treated as a contract violation.

Source

Thrown at crawl4ai/async_configs.py:1095

        if state:
            params["state"] = state
        if city:
            params["city"] = city

        url = "https://api.nstproxy.com/api/v1/generate/apiproxies"

        try:
            response = requests.get(url, params=params, timeout=10)
            response.raise_for_status()

            data = response.json()

            # --- Handle API error response ---
            if isinstance(data, dict) and data.get("err"):
                raise PermissionError(f"[NSTProxy] API Error: {data.get('msg', 'Unknown error')}")

            if not isinstance(data, list) or not data:
                raise ValueError("[NSTProxy] Invalid API response — expected a non-empty list")

            proxy_info = data[0]

            # --- Apply proxy config ---
            self.proxy_config = ProxyConfig(
                server=f"{protocol}://{proxy_info['ip']}:{proxy_info['port']}",
                username=proxy_info["username"],
                password=proxy_info["password"],
            )

        except Exception as e:
            print(f"[NSTProxy] ❌ Failed to set proxy: {e}")
            raise

class VirtualScrollConfig:
    """Configuration for virtual scroll handling.
    
    This config enables capturing content from pages with virtualized scrolling

View on GitHub (pinned to 7e80152142)

Solutions

1. Loosen the geo filters (drop state/city) and retry — empty availability is the most common cause
2. Manually curl https://api.nstproxy.com/api/v1/generate/apiproxies with your params to inspect the raw body and confirm the schema
3. If the schema changed, pin/upgrade the crawl4ai version that matches the current NSTProxy contract
4. Retry after a short delay — transient zero-inventory responses usually resolve

Example fix

// before
cfg.set_nstproxy(token=t, channel_id=c, country="us", state="ca", city="tiny-town")
// after
cfg.set_nstproxy(token=t, channel_id=c, country="us")  # widen availability

try:
    cfg.set_nstproxy(token=t, channel_id=c)
except ValueError as e:
    if "Invalid API response" in str(e):
        cfg.set_nstproxy(token=t, channel_id=c, country="", state="", city="")  # widen filters
    else:
        raise

When it happens

Trigger: NSTProxy changing its response schema; an intermediary (corporate proxy, captive portal) returning JSON that is not the proxy list; the API returning [] when no proxy is currently available for the requested filters.

Common situations: Upstream API version bump; requesting a very narrow geo filter (country+state+city) with no inventory; TLS-inspecting middleboxes rewriting responses.

Related errors

• [NSTProxy] API Error: {data.get('msg', 'Unknown error')}
• [NSTProxy] token and channel_id are required
• [NSTProxy] Invalid protocol: {protocol}
• Invalid proxy string format: {proxy_str}
• str(e)

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/032b8fe7892b51d6. Report an issue: GitHub.