unclecode/crawl4ai · error · HTTPException

Artifact storage quota exceeded

Error message

Artifact storage quota exceeded

What it means

A 507 from _store_artifact(): write_artifact() raised QuotaExceeded — the sandboxed artifact store's total storage quota is exhausted. Unlike the 413 (single artifact too large), this means cumulative stored data has hit the global cap.

Source

Thrown at deploy/docker/server.py:641

        from crawl4ai.utils import preprocess_html_for_schema
        processed_html = preprocess_html_for_schema(raw_html)
        return JSONResponse({"html": processed_html, "url": body.url, "success": True})
    except Exception as e:
        raise HTTPException(500, detail=str(e))
    finally:
        if crawler:
            await release_crawler(crawler)

# ── artifact store helpers ───────────────────────────────────
def _store_artifact(kind: str, data: bytes) -> dict:
    """Write to the sandboxed store; map quota/size errors to HTTP codes."""
    from artifacts import write_artifact, ArtifactTooLarge, QuotaExceeded
    try:
        meta = write_artifact(kind, data)
    except ArtifactTooLarge:
        raise HTTPException(413, "Artifact too large")
    except QuotaExceeded:
        raise HTTPException(507, "Artifact storage quota exceeded")
    return {
        "artifact_id": meta["artifact_id"],
        "url": f"/artifacts/{meta['artifact_id']}",
        "mime": meta["mime"],
        "size": meta["size"],
    }


@app.get("/artifacts/{artifact_id}")
async def get_artifact(artifact_id: str, _td: Dict = Depends(token_dep)):
    """Fetch a previously generated artifact by its opaque id (authed)."""
    from artifacts import resolve_artifact, ArtifactNotFound
    try:
        path, mime = resolve_artifact(artifact_id)
    except ArtifactNotFound:
        raise HTTPException(404, "Artifact not found")
    return FileResponse(path, media_type=mime, headers={"X-Content-Type-Options": "nosniff"})

View on GitHub (pinned to 7e80152142)

Solutions

1. Prune old artifacts (the store's cleanup/retention mechanism) or wipe the store volume if artifacts are disposable.
2. Raise the store quota / volume size for the deployment.
3. Add client-side retention: only request artifact storage when needed, or fetch-and-delete after consumption.
4. Monitor store usage and alert before the quota is reached.

resp = requests.post(f'{BASE}/pdf', json={'url': url}, headers=hdrs)
if resp.status_code == 507:
    # quota exhausted: fetch-and-consume inline bytes, skip storage-dependent flows
    raise RuntimeError('artifact store full — prune artifacts or raise quota')

When it happens

Trigger: Many /screenshot or /pdf calls accumulating artifacts until the store's total quota is full; the next write of any size fails with 507.

Common situations: Long-running service without artifact retention/cleanup; batch screenshot jobs; container volume sized too small for the workload.

Related errors

• Invalid CSS selector: '{css_selector}'
• Artifact too large
• Artifact not found
• Timeout after {timeout}ms waiting for selector '{css_selecto
• Timeout after {timeout}ms waiting for selector '{wait_for}'

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/265d8f769644b2b8. Report an issue: GitHub.