projectdiscovery/subfinder · warning

archive results truncated after

Error message

archive results truncated after %d pages, more exist

What it means

This error is emitted by the scanmalware source's enumerateArchive when pagination stops at the page limit (smqlMaxPages) and the last allowed page was still full, meaning more archive results exist than were read. The library raises it instead of silently presenting a locally truncated result set as complete. It is a completeness warning surfaced as an Error-type Result in the results channel.

Solutions

1. Increase the scanmalware page limit configuration (smqlMaxPages) so all archive pages are fetched.
2. Split enumeration into narrower queries (e.g., per-host or time-bounded) to reduce result volume below the page cap.
3. Treat this Result as a signal that results are partial and combine with other sources for coverage.

Example fix

// before: default page cap causes truncation on large domains
subscraping := scanmalware.New() // smqlMaxPages default
// after: raise the page cap for this domain
agent.SetPagesLimit(100) // or the scanmalware-specific pages option

if pagesLimit >= smqlMaxPages { log.Warn("scanmalware archive may be truncated; raise page limit") }

When it happens

Trigger: Running enumerateArchive on a domain whose historical archive listing spans more than smqlMaxPages full pages, so the loop exits with results still remaining.

Common situations: Enumerating very large or old domains where the archive endpoint holds far more pages than the configured page cap; running with default (low) page limits for speed.

Related errors

• observed host results truncated for source
• %s
• unexpected status code
• request failed with status
• request failed with status

AI-assisted analysis of projectdiscovery/subfinder@7a0b91f0fa (2026-09-06). Data as JSON: /api/errors/c7851186c853504a. Report an issue: GitHub.

Appendix: source

Thrown at pkg/subscraping/sources/scanmalware/scanmalware.go:185

		for _, row := range body.Results {
			hosts = append(hosts, hostOf(row.URL), hostOf(row.FinalURL))
		}
		if !s.emit(ctx, hosts, session, results) {
			return
		}

		// A short page is the last page.
		if len(body.Results) < smqlPageSize {
			return
		}
	}

	// Falling out of the loop means the last allowed page was full, so there are more
	// archive results than were read. Report it rather than presenting a locally
	// truncated set as the complete one.
	results <- subscraping.Result{
		Source: s.Name(), Type: subscraping.Error,
		Error: fmt.Errorf("archive results truncated after %d pages, more exist", smqlMaxPages),
	}
	s.errors++
}

// enumerateObservedHosts reads the hosts a browser resolved and requested while
// rendering pages on the domain.
//
// The default source set is deliberate: it is limited to hosts actually contacted.
// The endpoint can also return hosts named in a CSP or extracted statically from
// script source, but only about a third of those were ever contacted, and subfinder
// has no way to carry that distinction to the user.
func (s *Source) enumerateObservedHosts(ctx context.Context, domain string, session *subscraping.Session, results chan subscraping.Result) {
	requestURL := fmt.Sprintf("https://scanmalware.com/api/v1/hosts/%s?subdomains_only=true",
		url.PathEscape(domain))

	s.requests++
	resp, err := session.Get(ctx, requestURL, "", map[string]string{"User-Agent": userAgent})
	if err != nil {

View on GitHub (pinned to 7a0b91f0fa)