unclecode/crawl4ai · error · HTTPException

Artifact too large

Error message

Artifact too large

What it means

A 413 from _store_artifact(): write_artifact() raised ArtifactTooLarge, meaning the generated artifact (PNG screenshot, PDF, etc.) exceeds the sandboxed store's per-artifact size cap. The store enforces limits so single payloads can't fill the disk.

Source

Thrown at deploy/docker/server.py:639

        raw_html = results[0].html
        from crawl4ai.utils import preprocess_html_for_schema
        processed_html = preprocess_html_for_schema(raw_html)
        return JSONResponse({"html": processed_html, "url": body.url, "success": True})
    except Exception as e:
        raise HTTPException(500, detail=str(e))
    finally:
        if crawler:
            await release_crawler(crawler)

# ── artifact store helpers ───────────────────────────────────
def _store_artifact(kind: str, data: bytes) -> dict:
    """Write to the sandboxed store; map quota/size errors to HTTP codes."""
    from artifacts import write_artifact, ArtifactTooLarge, QuotaExceeded
    try:
        meta = write_artifact(kind, data)
    except ArtifactTooLarge:
        raise HTTPException(413, "Artifact too large")
    except QuotaExceeded:
        raise HTTPException(507, "Artifact storage quota exceeded")
    return {
        "artifact_id": meta["artifact_id"],
        "url": f"/artifacts/{meta['artifact_id']}",
        "mime": meta["mime"],
        "size": meta["size"],
    }


@app.get("/artifacts/{artifact_id}")
async def get_artifact(artifact_id: str, _td: Dict = Depends(token_dep)):
    """Fetch a previously generated artifact by its opaque id (authed)."""
    from artifacts import resolve_artifact, ArtifactNotFound
    try:
        path, mime = resolve_artifact(artifact_id)
    except ArtifactNotFound:
        raise HTTPException(404, "Artifact not found")

View on GitHub (pinned to 7e80152142)

Solutions

1. Reduce output size: for screenshots set wait_for_images only when needed and target shorter pages; for PDFs, print sub-pages instead of one giant page.
2. Raise the per-artifact size cap in the artifacts store configuration if the deployment genuinely needs large artifacts.
3. Split the crawl: capture per-section URLs and store several smaller artifacts.
4. Check artifact store config (ARTIFACT_MAX_SIZE-style env/settings) in the container.

# Client-side heuristic: avoid full-page captures of extremely long pages
import requests

def page_too_long(url: str, approx_max_px: int = 30000) -> bool:
    r = requests.get(url, timeout=10)
    # rough heuristic on content length; tune to your store's cap
    return len(r.content) > approx_max_px * 40

resp = requests.post(f'{BASE}/screenshot', json=body, headers=hdrs)
if resp.status_code == 413:
    # artifact too large: shrink the capture (split page, lower resolution) and retry

When it happens

Trigger: POST /screenshot or /pdf on a page whose rendered PNG/PDF is larger than the configured per-artifact maximum; e.g. an extremely long page captured full-page at high resolution.

Common situations: Full-page screenshots of infinite-scroll or very long pages; PDFs of huge documents; a lowered store size cap in the deployment config versus earlier versions.

Related errors

• Timeout after {timeout}ms waiting for selector '{css_selecto
• Artifact storage quota exceeded
• Artifact not found
• Invalid CSS selector: '{css_selector}'
• Timeout after {timeout}ms waiting for selector '{wait_for}'

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/ee5c9081bbe07fdf. Report an issue: GitHub.