projectdiscovery/katana · warning

err.Error()

Error message

err.Error()

What it means

In the crawl debugger's HTTP handler handleActiveURLs, any failure encoding the active-URLs JSON is written back as an HTTP 500 with err.Error(). This is a JSON marshaling/encoding failure at response time — the payload (timestamps, active URL list) could not be serialized. In practice rare, since the payload contains only strings and ints.

Source

Thrown at pkg/engine/headless/debugger.go:108

	now := time.Now()
	for _, au := range cd.activeURLs {
		copy := *au
		copy.Duration = now.Sub(au.StartTime).String()
		urls = append(urls, copy)
	}
	return urls
}

// HTTP handlers
func (cd *CrawlDebugger) handleActiveURLs(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	active := cd.GetActiveURLs()
	if err := json.NewEncoder(w).Encode(map[string]interface{}{
		"timestamp":   time.Now().Format(time.RFC3339),
		"active_urls": active,
		"count":       len(active),
	}); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}
}

func (cd *CrawlDebugger) handleHealth(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	if err := json.NewEncoder(w).Encode(map[string]interface{}{
		"status":    "ok",
		"timestamp": time.Now().Format(time.RFC3339),
	}); err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}
}

func (cd *CrawlDebugger) Close() {
	if cd == nil {
		return
	}

View on GitHub (pinned to e3e742739c)

Solutions

1. Check the client connection; disconnecting browsers cause broken-pipe errors that surface here.
2. Verify GetActiveURLs returns only JSON-serializable types (strings/slices).
3. Log the error server-side as well, since writing it to a broken connection loses the diagnostic.
4. Consider http.StatusClientClosedRequest handling for write errors after headers are sent.

Example fix

// before
if err := json.NewEncoder(w).Encode(payload); err != nil {
    http.Error(w, err.Error(), http.StatusInternalServerError)
}
// after
if err := json.NewEncoder(w).Encode(payload); err != nil {
    log.Printf("debugger: encode active_urls: %v", err) // http.Error may also fail if headers already sent
    return
}

// sanity-check payload marshalability before writing
if _, err := json.Marshal(cd.GetActiveURLs()); err != nil {
    log.Printf("active_urls not marshalable: %v", err)
}

// Go: log encode errors instead of a 500 that also fails
if err := json.NewEncoder(w).Encode(payload); err != nil {
    log.Printf("debugger active_urls encode: %v", err)
}

When it happens

Trigger: json.NewEncoder(w).Encode fails, e.g. because the ResponseWriter breaks mid-write (client disconnected), or the payload contains unsupported values from GetActiveURLs.

Common situations: Client closed the debugger connection while the handler was writing; custom types returned by GetActiveURLs that are not JSON-marshalable after code changes.

Related errors

• could not create new request

AI-assisted analysis of projectdiscovery/katana@e3e742739c (2026-09-03). Data as JSON: /api/errors/94574bc5696e2226. Report an issue: GitHub.