projectdiscovery/katana · warning

could not create new request

Error message

could not create new request

What it means

netHTTPRequestFromProto converts a CDP proto.NetworkRequest (captured by the Fetch domain interceptor) into a Go *http.Request, and wraps http.NewRequest failures with this message. http.NewRequest only fails on an unparseable URL or an invalid HTTP method, so this error indicates a malformed request URL or method arriving from the browser event stream. The caller (the FetchRequestPaused handler) silently drops the request when this happens.

Source

Thrown at pkg/engine/headless/browser/browser.go:625

	if err != nil {
		return nil, err
	}

	if !r.Base64Encoded {
		return []byte(r.Body), nil
	}

	bs, err := base64.StdEncoding.DecodeString(r.Body)
	if err != nil {
		return nil, err
	}
	return bs, nil
}

func netHTTPRequestFromProto(e *proto.NetworkRequest) (*http.Request, error) {
	req, err := http.NewRequest(e.Method, e.URL, nil)
	if err != nil {
		return nil, errors.Wrap(err, "could not create new request")
	}
	for k, v := range e.Headers {
		req.Header.Set(k, v.Str())
	}
	if e.PostData != "" {
		req.Body = io.NopCloser(strings.NewReader(e.PostData))
		req.ContentLength = int64(len(e.PostData))
	}
	return req, nil
}

func netHTTPResponseFromProto(e *proto.FetchRequestPaused, body []byte) *http.Response {
	httpresp := &http.Response{
		Proto:         "HTTP/1.1",
		ProtoMajor:    1,
		ProtoMinor:    1,
		Header:        make(http.Header),
		StatusCode:    *e.ResponseStatusCode,

View on GitHub (pinned to e3e742739c)

Solutions

1. Filter non-HTTP(S) URL schemes (blob:, data:, chrome-extension:, about:) from e.URL before calling netHTTPRequestFromProto.
2. Validate that e.URL parses with url.Parse and has http/https scheme, and that e.Method is a valid HTTP token; skip conversion otherwise.
3. If URLs legitimately lack a host due to redirects, resolve them against the page's base URL before conversion.
4. This error is swallowed by the handler (returns silently) — if you need those requests, add logging at the call site (browser.go:550).

Example fix

// before
func netHTTPRequestFromProto(e *proto.NetworkRequest) (*http.Request, error) {
    req, err := http.NewRequest(e.Method, e.URL, nil)
    ...
}

// after: skip non-HTTP schemes upstream
u, perr := url.Parse(e.Request.URL)
if perr != nil || (u.Scheme != "http" && u.Scheme != "https") {
    return // not convertible; skip callback emission
}
httpreq, err := netHTTPRequestFromProto(e.Request)

func isConvertibleNetworkRequest(e *proto.NetworkRequest) bool {
    u, err := url.Parse(e.URL)
    return err == nil && u.IsAbs() && (u.Scheme == "http" || u.Scheme == "https") && e.Method != ""
}

// Go-style guard used before conversion
if !isConvertibleNetworkRequest(e.Request) {
    return // skip non-HTTP/unparseable intercepted requests
}

httpreq, err := netHTTPRequestFromProto(e.Request)
if err != nil {
    slog.Debug("skipping non-convertible intercepted request", "url", e.Request.URL, "error", err)
    return
}

When it happens

Trigger: e.URL being empty, relative, or otherwise unparseable (e.g. 'blob:', 'data:', malformed absolute URLs, or missing scheme) or e.Method being an invalid token, when a paused response event is converted for the RequestCallback pipeline.

Common situations: Crawling pages that issue requests to non-HTTP schemes (blob:, data:, chrome-extension:) that CDP reports but net/url cannot parse as absolute HTTP URLs; redirect chains or service workers producing URLs without a host; corrupted/empty NetworkRequest fields from unusual resources like WebSocket or manifest fetches.

Related errors

• could not create new page
• could not initialize stealth
• could not initialize javascript env
• could not enable fetch domain
• failed to inject utils.js

AI-assisted analysis of projectdiscovery/katana@e3e742739c (2026-09-03). Data as JSON: /api/errors/ac47f183087e16a6. Report an issue: GitHub.