RustPython/RustPython · error · ValueError

readinto returned {n} outside buffer size {len(b)}

Error message

readinto returned {n} outside buffer size {len(b)}

What it means

ValueError raised inside RawIOBase.read(size) (the default read implementation shown at the SOURCE) after it allocates a buffer of exactly size bytes and delegates to the subclass hook readinto(b): the hook returned n that is negative or larger than len(b). This breaks the readinto contract - return the number of bytes actually written into the buffer, 0 <= n <= len(b). It never happens with stdlib streams; it always indicates a buggy custom read()/readinto() in a RawIOBase/IOBase subclass, and the ValueError fires before the corrupted count can poison higher layers.

Source

Thrown at Lib/_pyio.py:625

    # primitive operation, but that would lead to nasty recursion in case
    # a subclass doesn't implement either.)

    def read(self, size=-1):
        """Read and return up to size bytes, where size is an int.

        Returns an empty bytes object on EOF, or None if the object is
        set not to block and has no data to read.
        """
        if size is None:
            size = -1
        if size < 0:
            return self.readall()
        b = bytearray(size.__index__())
        n = self.readinto(b)
        if n is None:
            return None
        if n < 0 or n > len(b):
            raise ValueError(f"readinto returned {n} outside buffer size {len(b)}")
        del b[n:]
        return bytes(b)

    def readall(self):
        """Read until EOF, using multiple read() call."""
        res = bytearray()
        while data := self.read(DEFAULT_BUFFER_SIZE):
            res += data
        if res:
            return bytes(res)
        else:
            # b'' or None
            return data

    def readinto(self, b):
        """Read bytes into a pre-allocated bytes-like object b.

        Returns an int representing the number of bytes read (0 for EOF), or

View on GitHub (pinned to aaeab4f754)

Solutions

1. Fix readinto to clamp and report the real write count: n = min(len(data), len(b)); b[:n] = data[:n]; return n
2. Return 0 (never a negative) at EOF
3. Add a unit test asserting 0 <= f.readinto(bytearray(k)) <= k for several buffer sizes k

Example fix

# before
class MemSource(io.RawIOBase):
    def __init__(self, data):
        self.data, self.pos = data, 0
    def readinto(self, b):
        chunk = self.data[self.pos:self.pos + 4096]   # ignores len(b)
        b[:] = chunk                                  # fails when len(b) < 4096
        return len(chunk)

# after
class MemSource(io.RawIOBase):
    def __init__(self, data):
        self.data, self.pos = data, 0
    def readinto(self, b):
        chunk = self.data[self.pos:self.pos + len(b)]
        b[:len(chunk)] = chunk
        self.pos += len(chunk)
        return len(chunk)                             # 0 <= n <= len(b)

def check_readinto_contract(source, sizes=(1, 8, 64, 4096)):
    'Probe a custom readinto before trusting read() on it.'
    for k in sizes:
        buf = bytearray(k)
        n = source.readinto(buf)
        assert n is None or 0 <= n <= k, f'readinto returned {n} for buffer {k}'

def is_contract_ok(source) -> bool:
    try:
        buf = bytearray(8)
        n = source.readinto(buf)
        return n is None or 0 <= n <= 8
    except Exception:
        return False

try:
    data = raw.read(4096)
except ValueError as e:
    if 'readinto returned' in str(e):
        raise RuntimeError(f'custom stream broke the readinto contract: {e}') from e
    raise

When it happens

Trigger: A custom stream whose readinto returns len(data) it wanted to write rather than the bytes actually placed into b; readinto returning -1 for EOF instead of 0; readinto delegating to another object's readinto on a smaller buffer and forwarding the outer count; test fakes that ignore the buffer length.

Common situations: Wrapping sockets, serial ports, or memoryviews in the io interfaces; porting C-style APIs that use negative return codes for errors/EOF; in-memory fake streams in unit tests that copy more than requested.

Related errors

• can't have text and binary mode at once
• can't have read/write/append mode at once
• must have exactly one of read/write/append mode
• binary mode doesn't take an encoding argument
• binary mode doesn't take an errors argument

AI-assisted analysis of RustPython/RustPython@aaeab4f754 (2026-08-17). Data as JSON: /api/errors/13c12ca24bcda229. Report an issue: GitHub.