zylon-ai/private-gpt · error · TimeoutError

Timeout ({config.timeout_seconds}s) scraping {[r.url for r i

Error message

Timeout ({config.timeout_seconds}s) scraping {[r.url for r in config.requests]}

What it means

Raised by run_scrape_in_session when the sandboxed Playwright scrape script exits with code 124, the convention used by GNU timeout for a killed process. The runner gives the script timeout_seconds per request (total = timeout_seconds * max(1, len(requests)) plus a fixed margin) via SandboxExecOptions, so this error means the whole batch exceeded that budget. It is a run-level failure: no per-URL results are returned even if some pages finished.

Source

Thrown at private_gpt/components/web/scraper/runner.py:126

    the others): each slot in the returned list is either the page HTML or
    the exception for that URL. Run-level failures still raise.
    """
    script_path = posixpath.join(base_dir, SCRIPT_FILENAME)
    config_path = posixpath.join(base_dir, CONFIG_FILENAME)
    await session.write_file(script_path, _load_script_text().encode("utf-8"))
    await session.write_file(config_path, config.model_dump_json().encode("utf-8"))

    total_timeout = config.timeout_seconds * max(1, len(config.requests))
    result = await session.exec(
        f"{session.python_executable} {shlex.quote(SCRIPT_FILENAME)}"
        f" {shlex.quote(CONFIG_FILENAME)}",
        SandboxExecOptions(
            cwd=base_dir, timeout=total_timeout + _EXEC_TIMEOUT_MARGIN_SECONDS
        ),
    )

    if result.exit_code == 124:
        raise TimeoutError(
            f"Timeout ({config.timeout_seconds}s) scraping "
            f"{[r.url for r in config.requests]}"
        )
    if result.failed:
        error = result.stderr or result.stdout
        if _BROWSERS_MISSING_PATTERN.search(error):
            raise RuntimeError(
                "Playwright browsers are not installed. "
                "Run `playwright install` and try again."
            )
        raise RuntimeError(
            f"Scrape script failed (exit_code={result.exit_code}): {error}"
        )

    results_raw = json.loads(
        (await session.read_file(posixpath.join(base_dir, RESULTS_FILENAME))).decode(
            "utf-8"
        )

View on GitHub (pinned to 4a030776a3)

Solutions

1. Increase web_fetch.timeout_seconds in settings (e.g. 30-60) so per-request budget covers slow pages.
2. Reduce web_fetch.batch_size (or set 1) so one slow URL cannot consume the whole run's total timeout.
3. Verify network egress/proxy from the sandbox session (proxy url/username/password settings, SSL cert config) with a manual session.exec curl.
4. Test the offending URL(s) manually in a browser or with the scrape script to confirm they render within the budget; drop or special-case pages that hang.

Example fix

# before (settings.yaml)
web_fetch:
  timeout_seconds: 15
  batch_size: 5

# after
web_fetch:
  timeout_seconds: 45
  batch_size: 2

# Pre-flight: estimate the run budget before dispatching
budget = config.timeout_seconds * max(1, len(config.requests))
if budget > MAX_ACCEPTABLE_RUN_SECONDS:
    raise ValueError(f'batch too large/slow: budget={budget}s')

def is_scrape_timeout_error(exc: BaseException) -> bool:
    return isinstance(exc, TimeoutError) and 'scraping [' in str(exc)

try:
    outputs = await run_scrape_in_session(session, base_dir, config)
except TimeoutError:
    # one slow URL poisons the batch: retry offending URLs individually with a larger budget
    config.timeout_seconds *= 2
    outputs = await run_scrape_in_session(session, base_dir, config)

When it happens

Trigger: Calling run_scrape_in_session with a batch whose pages are slow (JS-heavy SPA, blocked network egress from the sandbox, proxy misconfiguration, a hung navigation without load event); timeout_seconds left at the 15s default while batch_size coalesces up to 5 heavy pages into one browser run; a page with an infinite redirect or long-poll resource.

Common situations: Corporate proxy/SSL inspection blocking the sandbox's outbound traffic; scraping modern React/Vue sites that never fire 'load' within the default 15s; large batches where one URL hangs and starves the shared timeout budget of the coalesced run.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Scrape script failed (exit_code={result.exit_code}): {error}
• Code execution failed: {message}
• Playwright browsers are not installed. Run `playwright insta
• Server did not become ready within {_HEALTH_TIMEOUT}s
• OVERLOADED_CONDENSATION_ERROR

AI-assisted analysis of zylon-ai/private-gpt@4a030776a3 (2026-08-15). Data as JSON: /api/errors/b30ea81f73ab3a05. Report an issue: GitHub.