JuliusBrussee/caveman · warning · ValueError

File too large to compress safely (max 500KB): {filepath}

Error message

File too large to compress safely (max 500KB): {filepath}

What it means

Raised by compress_file() when the target file's stat().st_size exceeds MAX_FILE_SIZE (500,000 bytes ≈ 500KB). The cap exists because compression ships the raw file contents to the Anthropic API — a third-party boundary — so large files are rejected before any bytes leave the machine, both to bound cost/latency and to limit exposure.

Source

Thrown at skills/caveman-compress/scripts/compress.py:284

COMPRESSED (fix this):
{compressed}

Return ONLY the fixed compressed file. No explanation.
"""


# ---------- Core Logic ----------


def compress_file(filepath: Path) -> bool:
    # Resolve and validate path
    filepath = filepath.resolve()
    MAX_FILE_SIZE = 500_000  # 500KB
    if not filepath.exists():
        raise FileNotFoundError(f"File not found: {filepath}")
    if filepath.stat().st_size > MAX_FILE_SIZE:
        raise ValueError(f"File too large to compress safely (max 500KB): {filepath}")

    # Refuse files that look like they contain secrets or PII. Compressing ships
    # the raw bytes to the Anthropic API — a third-party boundary — so we fail
    # loudly rather than silently exfiltrate credentials or keys. Override is
    # intentional: the user must rename the file if the heuristic is wrong.
    if is_sensitive_path(filepath):
        raise ValueError(
            f"Refusing to compress {filepath}: filename looks sensitive "
            "(credentials, keys, secrets, or known private paths). "
            "Compression sends file contents to the Anthropic API. "
            "Rename the file if this is a false positive."
        )

    print(f"Processing: {filepath}")

    if not should_compress(filepath):
        print("Skipping (not natural language)")
        return False

View on GitHub (pinned to 27d5a3981a)

Solutions

1. Split the file into chunks under 500KB (e.g. split or head/tail by lines) and compress each chunk separately.
2. Trim the noise first: strip binary/base64 blobs, redundant blocks, or older sections, then compress the remainder.
3. If the content is genuinely needed whole, compress a summary/index locally and send only that — do not raise the cap casually, it is an exfiltration/cost bound.

Example fix

# before
compress_file(Path('session-full.log'))  # 2.1MB -> ValueError

# after
subprocess.run(['split', '-b', '480k', 'session-full.log', 'part-'])
for part in sorted(glob('part-*')):
    compress_file(Path(part))

from pathlib import Path
MAX = 500_000

def under_size_cap(p: Path) -> bool:
    return p.exists() and p.stat().st_size <= MAX

try:
    compress_file(path)
except ValueError as e:
    if "too large" in str(e):
        for chunk in split_file(path, 480_000):
            compress_file(chunk)  # compress each chunk separately
    raise

When it happens

Trigger: Compressing a large log, dataset export, generated markdown, or minified bundle over 500KB; a file that grew (e.g. appended logs) since it was last compressed successfully.

Common situations: Running the condense-log or condense-file flow on session logs or build outputs; pointing the compressor at node_modules-sized or data-dump files by mistake.

Related errors

• File not found: {filepath}
• Refusing to compress {filepath}: filename looks sensitive (c
• cave_memory_too_large
• repository intelligence: cwd is not a directory
• ErrMemoryTooLarge

AI-assisted analysis of JuliusBrussee/caveman@27d5a3981a (2026-08-15). Data as JSON: /api/errors/186dfdb3ca62f8ac. Report an issue: GitHub.