unclecode/crawl4ai · error · HTTPException

Artifact storage quota exceeded

Error message

Artifact storage quota exceeded

What it means

A 507 from _store_artifact(): write_artifact() raised QuotaExceeded — the sandboxed artifact store's total storage quota is exhausted. Unlike the 413 (single artifact too large), this means cumulative stored data has hit the global cap.

Solutions

  1. Prune old artifacts (the store's cleanup/retention mechanism) or wipe the store volume if artifacts are disposable.
  2. Raise the store quota / volume size for the deployment.
  3. Add client-side retention: only request artifact storage when needed, or fetch-and-delete after consumption.
  4. Monitor store usage and alert before the quota is reached.
Defensive patterns

Strategy: fallback

Try / catch

resp = requests.post(f'{BASE}/pdf', json={'url': url}, headers=hdrs)
if resp.status_code == 507:
    # quota exhausted: fetch-and-consume inline bytes, skip storage-dependent flows
    raise RuntimeError('artifact store full — prune artifacts or raise quota')

Prevention

When it happens

Trigger: Many /screenshot or /pdf calls accumulating artifacts until the store's total quota is full; the next write of any size fails with 507.

Common situations: Long-running service without artifact retention/cleanup; batch screenshot jobs; container volume sized too small for the workload.

Related errors

AI-assisted analysis of unclecode/crawl4ai@7e80152142 (2026-08-14). Data as JSON: /api/errors/265d8f769644b2b8. Report an issue: GitHub.

Appendix: source

Thrown at deploy/docker/server.py:641

        from crawl4ai.utils import preprocess_html_for_schema
        processed_html = preprocess_html_for_schema(raw_html)
        return JSONResponse({"html": processed_html, "url": body.url, "success": True})
    except Exception as e:
        raise HTTPException(500, detail=str(e))
    finally:
        if crawler:
            await release_crawler(crawler)

# ── artifact store helpers ───────────────────────────────────
def _store_artifact(kind: str, data: bytes) -> dict:
    """Write to the sandboxed store; map quota/size errors to HTTP codes."""
    from artifacts import write_artifact, ArtifactTooLarge, QuotaExceeded
    try:
        meta = write_artifact(kind, data)
    except ArtifactTooLarge:
        raise HTTPException(413, "Artifact too large")
    except QuotaExceeded:
        raise HTTPException(507, "Artifact storage quota exceeded")
    return {
        "artifact_id": meta["artifact_id"],
        "url": f"/artifacts/{meta['artifact_id']}",
        "mime": meta["mime"],
        "size": meta["size"],
    }


@app.get("/artifacts/{artifact_id}")
async def get_artifact(artifact_id: str, _td: Dict = Depends(token_dep)):
    """Fetch a previously generated artifact by its opaque id (authed)."""
    from artifacts import resolve_artifact, ArtifactNotFound
    try:
        path, mime = resolve_artifact(artifact_id)
    except ArtifactNotFound:
        raise HTTPException(404, "Artifact not found")
    return FileResponse(path, media_type=mime, headers={"X-Content-Type-Options": "nosniff"})

View on GitHub (pinned to 7e80152142)