{"record":{"id":"b30ea81f73ab3a05","repo":"zylon-ai/private-gpt","slug":"timeout-config-timeout-seconds-s-scraping-r-u","errorCode":null,"errorMessage":"Timeout ({config.timeout_seconds}s) scraping {[r.url for r in config.requests]}","messagePattern":"Timeout \\((.+?)s\\) scraping (.+?)","errorType":"exception","errorClass":"TimeoutError","httpStatus":null,"severity":"error","filePath":"private_gpt/components/web/scraper/runner.py","lineNumber":126,"sourceCode":"    the others): each slot in the returned list is either the page HTML or\n    the exception for that URL. Run-level failures still raise.\n    \"\"\"\n    script_path = posixpath.join(base_dir, SCRIPT_FILENAME)\n    config_path = posixpath.join(base_dir, CONFIG_FILENAME)\n    await session.write_file(script_path, _load_script_text().encode(\"utf-8\"))\n    await session.write_file(config_path, config.model_dump_json().encode(\"utf-8\"))\n\n    total_timeout = config.timeout_seconds * max(1, len(config.requests))\n    result = await session.exec(\n        f\"{session.python_executable} {shlex.quote(SCRIPT_FILENAME)}\"\n        f\" {shlex.quote(CONFIG_FILENAME)}\",\n        SandboxExecOptions(\n            cwd=base_dir, timeout=total_timeout + _EXEC_TIMEOUT_MARGIN_SECONDS\n        ),\n    )\n\n    if result.exit_code == 124:\n        raise TimeoutError(\n            f\"Timeout ({config.timeout_seconds}s) scraping \"\n            f\"{[r.url for r in config.requests]}\"\n        )\n    if result.failed:\n        error = result.stderr or result.stdout\n        if _BROWSERS_MISSING_PATTERN.search(error):\n            raise RuntimeError(\n                \"Playwright browsers are not installed. \"\n                \"Run `playwright install` and try again.\"\n            )\n        raise RuntimeError(\n            f\"Scrape script failed (exit_code={result.exit_code}): {error}\"\n        )\n\n    results_raw = json.loads(\n        (await session.read_file(posixpath.join(base_dir, RESULTS_FILENAME))).decode(\n            \"utf-8\"\n        )","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/zylon-ai/private-gpt/blob/4a030776a31a901ad80b1bf4d7faa2c1a367efbb/private_gpt/components/web/scraper/runner.py#L108-L144","documentation":"Raised by run_scrape_in_session when the sandboxed Playwright scrape script exits with code 124, the convention used by GNU timeout for a killed process. The runner gives the script timeout_seconds per request (total = timeout_seconds * max(1, len(requests)) plus a fixed margin) via SandboxExecOptions, so this error means the whole batch exceeded that budget. It is a run-level failure: no per-URL results are returned even if some pages finished.","triggerScenarios":"Calling run_scrape_in_session with a batch whose pages are slow (JS-heavy SPA, blocked network egress from the sandbox, proxy misconfiguration, a hung navigation without load event); timeout_seconds left at the 15s default while batch_size coalesces up to 5 heavy pages into one browser run; a page with an infinite redirect or long-poll resource.","commonSituations":"Corporate proxy/SSL inspection blocking the sandbox's outbound traffic; scraping modern React/Vue sites that never fire 'load' within the default 15s; large batches where one URL hangs and starves the shared timeout budget of the coalesced run.","solutions":["Increase web_fetch.timeout_seconds in settings (e.g. 30-60) so per-request budget covers slow pages.","Reduce web_fetch.batch_size (or set 1) so one slow URL cannot consume the whole run's total timeout.","Verify network egress/proxy from the sandbox session (proxy url/username/password settings, SSL cert config) with a manual session.exec curl.","Test the offending URL(s) manually in a browser or with the scrape script to confirm they render within the budget; drop or special-case pages that hang."],"exampleFix":"# before (settings.yaml)\nweb_fetch:\n  timeout_seconds: 15\n  batch_size: 5\n\n# after\nweb_fetch:\n  timeout_seconds: 45\n  batch_size: 2","handlingStrategy":"retry","validationCode":"# Pre-flight: estimate the run budget before dispatching\nbudget = config.timeout_seconds * max(1, len(config.requests))\nif budget > MAX_ACCEPTABLE_RUN_SECONDS:\n    raise ValueError(f'batch too large/slow: budget={budget}s')","typeGuard":"def is_scrape_timeout_error(exc: BaseException) -> bool:\n    return isinstance(exc, TimeoutError) and 'scraping [' in str(exc)","tryCatchPattern":"try:\n    outputs = await run_scrape_in_session(session, base_dir, config)\nexcept TimeoutError:\n    # one slow URL poisons the batch: retry offending URLs individually with a larger budget\n    config.timeout_seconds *= 2\n    outputs = await run_scrape_in_session(session, base_dir, config)","preventionTips":["Size web_fetch.timeout_seconds to your slowest realistic page, not the average.","Keep web_fetch.batch_size small so the shared budget is not consumed by one hang.","Log per-URL scrape durations and alert on pages approaching the budget."],"tags":["timeout","scraper","playwright","sandbox","batching"],"backgroundTag":null,"analyzedSha":"4a030776a31a901ad80b1bf4d7faa2c1a367efbb","analyzedAt":"2026-08-15T03:51:26.951Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}