projectdiscovery/katana · error

hybrid: response is nil

Error message

hybrid: response is nil

What it means

navigateRequest in the hybrid engine returns 'hybrid: response is nil' when, after issuing the HTTP request, both the response object and its underlying Resp are nil while err is nil. It is an internal invariant guard ensuring downstream code never dereferences a nil response.

Source

Thrown at pkg/engine/hybrid/crawl.go:380

	if domErr != nil {
		gologger.Warning().Msgf("could not get dom for %s: %s (continuing with page HTML)", request.URL, domErr)
	}

	// Use basePage with a fresh timeout for HTML retrieval so it succeeds
	// even if the navigation or DOM timeout was exhausted.
	body, err := sessionPage.Timeout(timeout).HTML()
	if err != nil {
		return nil, errkit.Wrap(err, "hybrid: could not get html")
	}

	parsed, err := urlutil.Parse(request.URL)
	if err != nil {
		return nil, errkit.Wrap(err, "hybrid: url could not be parsed")
	}

	if response == nil || response.Resp == nil {
		// err is guaranteed to be nil, due to previous checks.
		return nil, errors.New("hybrid: response is nil")
	}
	response.Resp.Request.URL = parsed.URL

	// Create a copy of interpolated shadow DOM elements and parse them separately
	if domResult != nil && domResult.Root != nil {
		var builder strings.Builder
		traverseDOMNode(domResult.Root, &builder)

		responseCopy := *response
		responseCopy.Body = builder.String()

		responseCopy.Reader, _ = goquery.NewDocumentFromReader(strings.NewReader(responseCopy.Body))
		if responseCopy.Reader != nil {
			navigationRequests := c.Options.Parser.ParseResponse(&responseCopy)
			c.Enqueue(s.Queue, navigationRequests...)
		}
	}

View on GitHub (pinned to e3e742739c)

Solutions

1. Inspect the HTTP client/transport in use — ensure it never returns a nil response with nil error
2. Update katana and dependencies to latest versions where transport behavior is fixed
3. Wrap custom RoundTrippers to always return either a non-nil response or a non-nil error

Example fix

// before: custom client can return (nil, nil)
resp, _ := client.Do(req)
return resp, nil
// after
resp, err := client.Do(req)
if err != nil { return nil, err }
if resp == nil { return nil, errors.New("empty response from client") }
return resp, nil

resp, err := client.Do(req)
if err != nil { return nil, err }
if resp == nil || resp.Resp == nil {
    return nil, errors.New("client returned nil response")
}

func hasValidResponse(r *types.Response) bool { return r != nil && r.Resp != nil }

result, err := navigateRequest(ctx, req)
if err != nil {
    if strings.Contains(err.Error(), "response is nil") { log.Warn("transport bug") }
    return nil, err
}

When it happens

Trigger: navigateRequest (crawl.go:380) after the request round-trip: response == nil || response.Resp == nil — e.g. a transport/client returning a non-error but empty result, or misconfigured response handling in custom clients.

Common situations: Custom HTTP clients/proxies that return (nil, nil) on certain failures; mocking layers in tests; version drift where a dependency changed its return contract.

AI-assisted analysis of projectdiscovery/katana@e3e742739c (2026-09-03). Data as JSON: /api/errors/48dab3054dea9a5a. Report an issue: GitHub.