{"record":{"id":"2fb8af2c99e7c0c8","repo":"fish2018/pansou","slug":"web-search-documents-contained-no-valid-links","errorCode":null,"errorMessage":"web search documents contained no valid links","messagePattern":"web search documents contained no valid links","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"plugin/sousou/sousou.go","lineNumber":238,"sourceCode":"\t\titem := item\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tdefer wg.Done()\n\t\t\tsem <- struct{}{}\n\t\t\tdefer func() { <-sem }()\n\t\t\tresult, err := p.fetchPansoDocument(client, item)\n\t\t\tif err != nil {\n\t\t\t\tdebugLog(\"document %s failed: %v\", item.DocURL, err)\n\t\t\t\treturn\n\t\t\t}\n\t\t\tmu.Lock()\n\t\t\tresults = append(results, result)\n\t\t\tmu.Unlock()\n\t\t}()\n\t}\n\twg.Wait()\n\tif len(results) == 0 {\n\t\treturn nil, fmt.Errorf(\"web search documents contained no valid links\")\n\t}\n\treturn results, nil\n}\n\nfunc (p *SousouAsyncPlugin) fetchPansoDocument(client *http.Client, item pansoSearchItem) (model.SearchResult, error) {\n\tctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)\n\tdefer cancel()\n\treq, err := http.NewRequestWithContext(ctx, http.MethodGet, item.DocURL, nil)\n\tif err != nil {\n\t\treturn model.SearchResult{}, err\n\t}\n\tsetSousouWebHeaders(req, SousouWebURL+\"?q=\")\n\tresp, err := client.Do(req)\n\tif err != nil {\n\t\treturn model.SearchResult{}, err\n\t}\n\tdefer resp.Body.Close()\n\tif resp.StatusCode != http.StatusOK {","sourceCodeStart":220,"sourceCodeEnd":256,"githubUrl":"https://github.com/fish2018/pansou/blob/beaa56133755a548ebc51b090b3816e2ae044aa6/plugin/sousou/sousou.go#L220-L256","documentation":"searchWeb fetches each candidate item's document concurrently (semaphore of 8) and builds final results. If every fetchPansoDocument call failed to yield a valid link, it raises 'web search documents contained no valid links'. Search results were found, but none could be converted into usable download links.","triggerScenarios":"After wg.Wait() in searchWeb, len(results) == 0 because every fetchPansoDocument call failed or produced a result without links — e.g. all detail pages non-200, or link extraction selectors broken.","commonSituations":"Detail-page endpoints moved or now require login; pan links expired/removed upstream; concurrency of 8 triggered rate limiting on all detail fetches; link parsing regex/selector outdated.","solutions":["Log fetchPansoDocument errors per item to see whether failures are HTTP-level or parsing-level.","Reduce concurrency (semaphore of 8) and add retry/backoff if detail pages are rate-limited.","Update the detail-page link extraction logic if panso.vip changed its markup.","Treat some failures as acceptable only when at least one result succeeds; otherwise surface per-item causes."],"exampleFix":"// before\nwg.Wait()\nif len(results) == 0 {\n    return nil, fmt.Errorf(\"web search documents contained no valid links\")\n}\n// after\nwg.Wait()\nif len(results) == 0 {\n    return nil, fmt.Errorf(\"web search documents contained no valid links (items=%d, lastErr=%v)\", len(items), firstFetchErr)\n}","handlingStrategy":"fallback","validationCode":null,"typeGuard":null,"tryCatchPattern":"results, err := plugin.Search(keyword)\nif err != nil && strings.Contains(err.Error(), \"no valid links\") {\n    // all detail fetches failed: inspect per-item errors, throttle, or switch source\n}","preventionTips":["Collect and log per-item fetchPansoDocument errors, not just the aggregate","Throttle detail-page fetches (lower semaphore, add delays) to avoid rate limits","Retry failed detail fetches individually before declaring total failure","Keep detail-page link extraction updated against site markup changes"],"tags":["scraping","concurrency","link-extraction","go"],"backgroundTag":"empty-result-set","analyzedSha":"beaa56133755a548ebc51b090b3816e2ae044aa6","analyzedAt":"2026-09-07T00:31:18.025Z","contentChangedAt":"2026-09-07T00:31:18.025Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}