MemPalace/mempalace · error · IOError

Could not read {filepath}: file too large ({file_stat.st_siz

Error message

Could not read {filepath}: file too large ({file_stat.st_size // (1024 * 1024)} MB)

What it means

Error "Could not read {filepath}: file too large ({file_stat.st_size // (1024 * 1024)} MB)" thrown in MemPalace/mempalace.

Source

Thrown at mempalace/normalize.py:146

    flags = os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0)
    if os.path.islink(filepath):
        raise IOError(f"Could not read {filepath}: symlinked files are skipped")
    fd = -1
    try:
        try:
            fd = os.open(filepath, flags)
        except OSError as exc:
            if exc.errno != errno.EAGAIN or not stat.S_ISREG(os.lstat(filepath).st_mode):
                raise
            fd = os.open(filepath, flags & ~getattr(os, "O_NONBLOCK", 0))
        file_stat = os.fstat(fd)
        if not stat.S_ISREG(file_stat.st_mode):
            # Text stays prefix-free: this raise is inside the ``try``, so the
            # ``except OSError`` below composes "Could not read <path>: ...".
            raise IOError("not a regular file")
        if file_stat.st_size > 500 * 1024 * 1024:  # 500 MB safety limit
            # Prefix-free for the same reason as the branch above.
            raise IOError(f"file too large ({file_stat.st_size // (1024 * 1024)} MB)")
        with os.fdopen(fd, "r", encoding="utf-8-sig", errors="replace") as f:
            fd = -1
            return f.read()
    except OSError as e:
        raise IOError(f"Could not read {filepath}: {e}") from e
    finally:
        if fd != -1:
            try:
                os.close(fd)
            except OSError:
                pass


def normalize(filepath: str) -> str:
    """
    Load a file and normalize to transcript format if it's a chat export.
    Plain text files pass through unchanged.
    """

View on GitHub (pinned to 06cb6987f0)

Solutions

1. Split the oversized transcript (e.g. split_mega_files) before mining

When it happens

Trigger: Thrown at mempalace/normalize.py:146 when the library encounters an invalid state.

Common situations: Transcript file exceeded the size cap.

AI-assisted analysis of MemPalace/mempalace@06cb6987f0 (2026-08-15). Data as JSON: /api/errors/d6acf6ba75eefe10. Report an issue: GitHub.