{"record":{"id":"c7851186c853504a","repo":"projectdiscovery/subfinder","slug":"archive-results-truncated-after-d-pages-more-exi","errorCode":null,"errorMessage":"archive results truncated after %d pages, more exist","messagePattern":"archive results truncated after (.+?) pages, more exist","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"pkg/subscraping/sources/scanmalware/scanmalware.go","lineNumber":185,"sourceCode":"\t\tfor _, row := range body.Results {\n\t\t\thosts = append(hosts, hostOf(row.URL), hostOf(row.FinalURL))\n\t\t}\n\t\tif !s.emit(ctx, hosts, session, results) {\n\t\t\treturn\n\t\t}\n\n\t\t// A short page is the last page.\n\t\tif len(body.Results) < smqlPageSize {\n\t\t\treturn\n\t\t}\n\t}\n\n\t// Falling out of the loop means the last allowed page was full, so there are more\n\t// archive results than were read. Report it rather than presenting a locally\n\t// truncated set as the complete one.\n\tresults <- subscraping.Result{\n\t\tSource: s.Name(), Type: subscraping.Error,\n\t\tError: fmt.Errorf(\"archive results truncated after %d pages, more exist\", smqlMaxPages),\n\t}\n\ts.errors++\n}\n\n// enumerateObservedHosts reads the hosts a browser resolved and requested while\n// rendering pages on the domain.\n//\n// The default source set is deliberate: it is limited to hosts actually contacted.\n// The endpoint can also return hosts named in a CSP or extracted statically from\n// script source, but only about a third of those were ever contacted, and subfinder\n// has no way to carry that distinction to the user.\nfunc (s *Source) enumerateObservedHosts(ctx context.Context, domain string, session *subscraping.Session, results chan subscraping.Result) {\n\trequestURL := fmt.Sprintf(\"https://scanmalware.com/api/v1/hosts/%s?subdomains_only=true\",\n\t\turl.PathEscape(domain))\n\n\ts.requests++\n\tresp, err := session.Get(ctx, requestURL, \"\", map[string]string{\"User-Agent\": userAgent})\n\tif err != nil {","sourceCodeStart":167,"sourceCodeEnd":203,"githubUrl":"https://github.com/projectdiscovery/subfinder/blob/7a0b91f0fac01b62c65328bd771a3560ae611d6a/pkg/subscraping/sources/scanmalware/scanmalware.go#L167-L203","documentation":"This error is emitted by the scanmalware source's enumerateArchive when pagination stops at the page limit (smqlMaxPages) and the last allowed page was still full, meaning more archive results exist than were read. The library raises it instead of silently presenting a locally truncated result set as complete. It is a completeness warning surfaced as an Error-type Result in the results channel.","triggerScenarios":"Running enumerateArchive on a domain whose historical archive listing spans more than smqlMaxPages full pages, so the loop exits with results still remaining.","commonSituations":"Enumerating very large or old domains where the archive endpoint holds far more pages than the configured page cap; running with default (low) page limits for speed.","solutions":["Increase the scanmalware page limit configuration (smqlMaxPages) so all archive pages are fetched.","Split enumeration into narrower queries (e.g., per-host or time-bounded) to reduce result volume below the page cap.","Treat this Result as a signal that results are partial and combine with other sources for coverage."],"exampleFix":"// before: default page cap causes truncation on large domains\nsubscraping := scanmalware.New() // smqlMaxPages default\n// after: raise the page cap for this domain\nagent.SetPagesLimit(100) // or the scanmalware-specific pages option\n","handlingStrategy":"fallback","validationCode":"if pagesLimit >= smqlMaxPages { log.Warn(\"scanmalware archive may be truncated; raise page limit\") }","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Configure a page limit sized to the largest domains you enumerate.","Treat any Error-typed result mentioning truncation as 'partial data', not fatal.","Combine multiple sources so truncated data from one is covered by others."],"tags":["network","pagination","truncated-results","subdomain-enumeration"],"backgroundTag":"results-truncated","analyzedSha":"7a0b91f0fac01b62c65328bd771a3560ae611d6a","analyzedAt":"2026-09-06T23:52:02.109Z","contentChangedAt":"2026-09-06T23:52:02.109Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}