projectdiscovery/katana · info

response filtered by similarity detection

Error message

response filtered by similarity detection

What it means

StandardWriter.Write in pkg/output/output.go returns this error when a Result carries a Response object that is entirely empty (no http.Resp, no body, no error string). The library treats such results as duplicates suppressed by similarity detection and intentionally skips writing them, surfacing a sentinel error instead of silent output.

Source

Thrown at pkg/output/output.go:185

	}
	if options.OutputTemplate != "" {
		writer.outputTemplate, err = fasttemplate.NewTemplate(options.OutputTemplate, "{{", "}}")
		if err != nil {
			return nil, errkit.Wrap(err, "output: could not create output format template")
		}
	}
	return writer, nil
}

// Write writes the result to file and/or screen.
func (w *StandardWriter) Write(result *Result) error {
	if result == nil {
		return errors.New("result is nil")
	}

	// Skip empty responses (e.g., from similarity filtering)
	if result.Response != nil && result.Response.Resp == nil && result.Response.Body == "" && result.Error == "" {
		return errors.New("response filtered by similarity detection")
	}

	if len(w.storeFields) > 0 {
		storeFields(result, w.storeFields)
	}

	if !w.extensionValidator.ValidatePath(result.Request.URL) {
		return errors.New("result does not match extension filter")
	}

	if !w.matchOutput(result) {
		return errors.New("result does not match output")
	}
	if w.filterOutput(result) {
		return errors.New("result is filtered out")
	}
	if len(w.filterPageType) > 0 && result.Response != nil && result.Response.KnowledgeBase != nil {
		if pageType, ok := result.Response.KnowledgeBase["PageType"].(string); ok {

View on GitHub (pinned to e3e742739c)

Solutions

1. Check result.Response.Resp/Body/Error before calling Write and skip results that are empty — the error is an expected skip signal, not a fault.
2. Disable or loosen similarity/duplicate filtering in the crawler options so suppressed responses are not emitted.
3. Treat this specific error as a no-op in your Write loop (log at debug level and continue).
4. If you genuinely need to record filtered responses, populate result.Error or the body before writing, or bypass the writer's empty-response guard.

Example fix

// before
if err := writer.Write(result); err != nil {
    return err
}
// after
if err := writer.Write(result); err != nil {
    if err.Error() == "response filtered by similarity detection" {
        continue // expected skip for duplicate/empty responses
    }
    return err
}

func writable(r *output.Result) bool {
    return r != nil && !(r.Response != nil && r.Response.Resp == nil && r.Response.Body == "" && r.Error == "")
}

func hasContent(r *output.Result) bool {
    return r != nil && r.Response != nil && (r.Response.Resp != nil || r.Response.Body != "" || r.Error != "")
}

if err := writer.Write(result); err != nil {
    if err.Error() == "response filtered by similarity detection" {
        return nil // expected skip
    }
    return err
}

When it happens

Trigger: Calling writer.Write(result) where result.Response != nil but result.Response.Resp == nil, result.Response.Body == "", and result.Error == "" — i.e. a result whose response was dropped by similarity/duplicate filtering upstream but still reached the writer.

Common situations: Running a crawl with similarity/duplicate-response filtering enabled and forwarding every emitted result (including suppressed ones) to a custom output writer; building a custom writer that ignores the filter flag; re-processing previously saved results that had empty bodies.

Related errors

• result does not match extension filter
• result does not match output
• result is filtered out
• result filtered by page type
• result is empty

AI-assisted analysis of projectdiscovery/katana@e3e742739c (2026-09-03). Data as JSON: /api/errors/23281450e799b284. Report an issue: GitHub.