{"record":{"id":"300405567efdbf1a","repo":"vxcontrol/pentagi","slug":"internal-engine-no-readable-content-found","errorCode":null,"errorMessage":"internal engine: no readable content found","messagePattern":"internal engine: no readable content found","errorType":"exception","errorClass":"FatalError","httpStatus":null,"severity":"error","filePath":"backend/pkg/tools/searchers/internal.go","lineNumber":172,"sourceCode":"\t\t\tcontinue\n\t\t}\n\t\tif len(md) > maxBytes {\n\t\t\tmd = md[:maxBytes]\n\t\t}\n\t\t// The source id (1-based, in fetch order) is what the summarizer cites as\n\t\t// [Source N]; usedURLs tracks the matching URL so it can be listed at the end.\n\t\tblocks = append(blocks, fmt.Sprintf(\"<source id=\\\"%d\\\" url=\\\"%s\\\">\\n%s\\n</source>\", fetched+1, u, md))\n\t\tusedURLs = append(usedURLs, u)\n\t\tfetched++\n\t}\n\n\tif fetched == 0 {\n\t\t// Nothing could be read. If the pages errored (vs. returned empty), surface a\n\t\t// retryable error so the orchestrator can fall back to another engine.\n\t\tif lastFetch != nil {\n\t\t\treturn \"\", Retryable(fmt.Errorf(\"internal engine: all pages failed to fetch: %w\", lastFetch), 0)\n\t\t}\n\t\treturn \"\", Fatal(fmt.Errorf(\"internal engine: no readable content found\"))\n\t}\n\n\tprompt := e.buildPrompt(req.Query, blocks)\n\tanswer, serr := e.summarizer(ctx, prompt)\n\tif serr != nil {\n\t\treturn \"\", Retryable(fmt.Errorf(\"internal engine: summarization failed: %w\", serr), 0)\n\t}\n\n\treturn appendSources(answer, usedURLs), nil\n}\n\n// appendSources appends the list of source URLs the answer was synthesized from, so the\n// [Source N] citations in the summary can be traced back to the page they came from. The\n// numbering matches the source ids fed to the summarizer (1-based, in fetch order).\nfunc appendSources(answer string, urls []string) string {\n\tif len(urls) == 0 {\n\t\treturn answer\n\t}","sourceCodeStart":154,"sourceCodeEnd":190,"githubUrl":"https://github.com/vxcontrol/pentagi/blob/ea665308baaff015b226f308438a68d929d0f29b/backend/pkg/tools/searchers/internal.go#L154-L190","documentation":"Thrown in `internalEngine.analyze` (backend/pkg/tools/searchers/internal.go:172) when zero pages produced usable content but the individual fetches did NOT error — each `FetchMarkdown` succeeded yet returned empty/whitespace-only markdown. Because nothing technically failed, this is a Fatal (not Retryable) error: the engine concludes the discovered pages simply contain no extractable text.","triggerScenarios":"All discovered URLs fetch successfully but return empty markdown — pages that are pure JavaScript-rendered SPAs (no static text), PDFs or images the fetcher cannot convert, or pages that return 200 with an empty body.","commonSituations":"Fetching modern JS-heavy sites whose content loads client-side; link results pointing to media/PDF files; scraper returning empty bodies for pages requiring cookies/headers.","solutions":["Confirm the discovered URLs are HTML pages with static text (open them in a browser/view-source)","Use a fetcher that executes JavaScript or a rendering proxy for SPA targets","Accept the Fatal result and ensure the orchestrator falls back to a non-fetch-based engine (e.g. Perplexity) in web_search.go fallbackStrategy","Rephrase the query to surface text-based results pages"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// quick sanity check that a discovered URL yields static text\nresp, _ := http.Get(u)\nct := resp.Header.Get(\"Content-Type\")\nusable := strings.Contains(ct, \"text/html\") // skip PDFs/images","typeGuard":null,"tryCatchPattern":"_, err := engine.Handle(ctx, req)\nif err != nil && searchers.IsFatal(err) {\n    // do not retry; switch to a different engine immediately\n}","preventionTips":["Prefer queries that surface text-based HTML result pages","Use a JS-rendering fetcher for SPA-heavy targets","Treat this Fatal as a signal to fall back to an LLM-based engine like Perplexity","Skip non-HTML URLs (PDF/images) before fetching"],"tags":["search","scraping","empty-content"],"backgroundTag":"empty-response-body","analyzedSha":"ea665308baaff015b226f308438a68d929d0f29b","analyzedAt":"2026-09-01T14:16:31.421Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}