gocolly/colly · error

internal server error

Error message

internal server error

What it means

The WebDebugger's statusHandler marshals the WebDebugger state to JSON for its /status HTTP endpoint; if json.MarshalIndent fails it responds with 500 'internal server error' and logs the underlying marshal error. This means the debugger's internal state could not be serialized (e.g. an unmarshalable field such as a channel, func, or cyclic reference).

Source

Thrown at debug/webdebugger.go:150

    }
    setTimeout(fetchStatus, 1000);
  });
}
$(document).ready(function() {
    fetchStatus();
});
</script>
</body>
</html>
`))
}

func (w *WebDebugger) statusHandler(wr http.ResponseWriter, r *http.Request) {
	w.Lock()
	jsonData, err := json.MarshalIndent(w, "", "  ")
	w.Unlock()
	if err != nil {
		http.Error(wr, "internal server error", http.StatusInternalServerError)
		log.Println("Error marshaling status JSON:", err)
		return
	}
	wr.Write(jsonData)
}

View on GitHub (pinned to 17d1d6ca92)

Solutions

1. Check the server log line 'Error marshaling status JSON:' for the underlying marshal error
2. Avoid creating/cyclic request state that breaks JSON marshaling of the debugger, or upgrade colly if a serialization bug was fixed
3. Serve status from a sanitized snapshot struct that excludes unserializable fields
4. If only scraping colly itself, treat HTTP 500 from the status endpoint as debugger-level failure and retry/backoff

resp, err := http.Get(debuggerStatusURL)
if err != nil { return err }
if resp.StatusCode == http.StatusInternalServerError {
    return errors.New("debugger status endpoint failed to marshal state")
}

func fetchStatus(url string) ([]byte, error) {
    resp, err := http.Get(url)
    if err != nil { return nil, err }
    defer resp.Body.Close()
    if resp.StatusCode != http.StatusOK {
        return nil, fmt.Errorf("status endpoint returned %d", resp.StatusCode)
    }
    return io.ReadAll(resp.Body)
}

When it happens

Trigger: GETting the web debugger's status endpoint while WebDebugger state contains values json.MarshalIndent cannot encode (channels, funcs, cyclic data), returning the literal 500 body 'internal server error'.

Common situations: Scraping the debugger status page programmatically and getting 500; running colly with debug/webdebugger enabled and checking status during or after complex scraping; custom modifications adding non-serializable fields to WebDebugger.

Understand the failure class

• HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

• ErrAbortedBeforeRequest
• Not following redirect to %q: %w

AI-assisted analysis of gocolly/colly@17d1d6ca92 (2026-08-30). Data as JSON: /api/errors/0502f3507f734077. Report an issue: GitHub.