{"record":{"id":"186dfdb3ca62f8ac","repo":"JuliusBrussee/caveman","slug":"file-too-large-to-compress-safely-max-500kb-fi-186dfd","errorCode":null,"errorMessage":"File too large to compress safely (max 500KB): {filepath}","messagePattern":"File too large to compress safely \\(max 500KB\\): (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"warning","filePath":"skills/caveman-compress/scripts/compress.py","lineNumber":284,"sourceCode":"\nCOMPRESSED (fix this):\n{compressed}\n\nReturn ONLY the fixed compressed file. No explanation.\n\"\"\"\n\n\n# ---------- Core Logic ----------\n\n\ndef compress_file(filepath: Path) -> bool:\n    # Resolve and validate path\n    filepath = filepath.resolve()\n    MAX_FILE_SIZE = 500_000  # 500KB\n    if not filepath.exists():\n        raise FileNotFoundError(f\"File not found: {filepath}\")\n    if filepath.stat().st_size > MAX_FILE_SIZE:\n        raise ValueError(f\"File too large to compress safely (max 500KB): {filepath}\")\n\n    # Refuse files that look like they contain secrets or PII. Compressing ships\n    # the raw bytes to the Anthropic API — a third-party boundary — so we fail\n    # loudly rather than silently exfiltrate credentials or keys. Override is\n    # intentional: the user must rename the file if the heuristic is wrong.\n    if is_sensitive_path(filepath):\n        raise ValueError(\n            f\"Refusing to compress {filepath}: filename looks sensitive \"\n            \"(credentials, keys, secrets, or known private paths). \"\n            \"Compression sends file contents to the Anthropic API. \"\n            \"Rename the file if this is a false positive.\"\n        )\n\n    print(f\"Processing: {filepath}\")\n\n    if not should_compress(filepath):\n        print(\"Skipping (not natural language)\")\n        return False","sourceCodeStart":266,"sourceCodeEnd":302,"githubUrl":"https://github.com/JuliusBrussee/caveman/blob/27d5a3981a347890211bb1bf2439e5c821a63bc9/skills/caveman-compress/scripts/compress.py#L266-L302","documentation":"Raised by compress_file() when the target file's stat().st_size exceeds MAX_FILE_SIZE (500,000 bytes ≈ 500KB). The cap exists because compression ships the raw file contents to the Anthropic API — a third-party boundary — so large files are rejected before any bytes leave the machine, both to bound cost/latency and to limit exposure.","triggerScenarios":"Compressing a large log, dataset export, generated markdown, or minified bundle over 500KB; a file that grew (e.g. appended logs) since it was last compressed successfully.","commonSituations":"Running the condense-log or condense-file flow on session logs or build outputs; pointing the compressor at node_modules-sized or data-dump files by mistake.","solutions":["Split the file into chunks under 500KB (e.g. split or head/tail by lines) and compress each chunk separately.","Trim the noise first: strip binary/base64 blobs, redundant blocks, or older sections, then compress the remainder.","If the content is genuinely needed whole, compress a summary/index locally and send only that — do not raise the cap casually, it is an exfiltration/cost bound."],"exampleFix":"# before\ncompress_file(Path('session-full.log'))  # 2.1MB -> ValueError\n\n# after\nsubprocess.run(['split', '-b', '480k', 'session-full.log', 'part-'])\nfor part in sorted(glob('part-*')):\n    compress_file(Path(part))","handlingStrategy":"validation","validationCode":"from pathlib import Path\nMAX = 500_000\n\ndef under_size_cap(p: Path) -> bool:\n    return p.exists() and p.stat().st_size <= MAX","typeGuard":null,"tryCatchPattern":"try:\n    compress_file(path)\nexcept ValueError as e:\n    if \"too large\" in str(e):\n        for chunk in split_file(path, 480_000):\n            compress_file(chunk)  # compress each chunk separately\n    raise","preventionTips":["Check file size before invoking the compress skill; split or trim anything near 500KB.","Strip binary/base64/log-noise from inputs first — big inputs are usually mostly non-language bytes that should_compress would skip anyway.","Do not raise MAX_FILE_SIZE to work around it; it bounds what gets shipped to a third-party API."],"tags":["filesystem","size-limit","llm","validation"],"backgroundTag":null,"analyzedSha":"27d5a3981a347890211bb1bf2439e5c821a63bc9","analyzedAt":"2026-08-15T09:26:11.751Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}