What causes "fetch gae fail:%d" in XX-net/XX-Net?

Retry — the handler retries on GAE_Exception and will pick another connection/IP

XX-net/XX-Net · error · GAE_Exception

fetch gae fail:%d

Error message

fetch gae fail:%d

What it means

GAE_Exception thrown when the frontend returns a synthetic status >= 600 (XX-Net uses 600+ internally to signal transport errors through the status field). The message includes the pseudo-status number, e.g. 600 for connection issues, distinguishing it from real HTTP statuses.

Source

Thrown at code/default/gae_proxy/local/gae_handler.py:330

        return response
    except Exception as e:
        response.worker.close("unpack protocol error")
        raise GAE_Exception(600, "unpack protocol:%r at:%s" % (e, traceback.format_exc()))


def request_gae_server(headers, body, url, timeout):
    # process on http protocol
    # process status code return by http server
    # raise error, let up layer retry.

    try:
        response = front.request(b"POST", b"", b"/_gh/", headers, body, timeout)
        if not response:
            raise GAE_Exception(600, "fetch gae fail")

        if response.status >= 600:
            raise GAE_Exception(
                response.status, "fetch gae fail:%d" % response.status)

        appid = response.ssl_sock.host.split(".")[0]
        if response.status == 404:
            # xlog.warning('APPID %r not exists, remove it.', response.ssl_sock.appid)
            front.appid_manager.report_not_exist(
                appid, response.ssl_sock.ip_str)
            # google_ip.report_connect_closed(response.ssl_sock.ip_str, "appid not exist")
            response.worker.close("appid not exist:%s" % appid)
            raise GAE_Exception(603, "appid not exist %s" % appid)

        if response.status == 503:
            xlog.warning('APPID %r out of Quota, remove it. %s',
                         appid, response.ssl_sock.ip_str)
            front.appid_manager.report_out_of_quota(appid)
            # google_ip.report_connect_closed(response.ssl_sock.ip_str, "out of quota")
            response.worker.close("appid out of quota:%s" % appid)
            raise GAE_Exception(604, "appid out of quota:%s" % appid)

View on GitHub (pinned to cfa5bc17b6)

Solutions

Retry — the handler retries on GAE_Exception and will pick another connection/IP
Update XX-Net; IP blocking patterns change and newer versions adapt
Check whether other Google IPs work (XX-Net status page) and let the scanner refresh the IP pool

Defensive patterns

Strategy: retry

Try / catch

try:
    resp = request_gae_proxy(...)
except GAE_Exception as e:
    if e.code >= 600:  # synthetic transport status
        resp = request_gae_proxy(...)  # picks new connection
    else:
        raise

Prevention

Treat any status >= 600 as internal/transport, never as a real HTTP status
Let the connection pool recycle failing workers (request already closes them)
Keep the app updated to benefit from improved connection handling

When it happens

Trigger: request_gae_proxy calls where front.request encounters a connection-level error and encodes it as an internal status >= 600 instead of raising directly; e.g. TLS handshake failure or reset during the POST /_gh/ request.

Common situations: Connection reset mid-request by firewall, Google IP flakiness, SNI-based blocking of the selected IP.

Related errors

AI-assisted analysis of XX-net/XX-Net@cfa5bc17b6 (2026-08-27). Data as JSON: /api/errors/3d8b4ce18c9e937b. Report an issue: GitHub.