vllm-project/vllm · error · ValueError

Data for address:id '{address}:{monotonic_id}' has been modi

Error message

Data for address:id '{address}:{monotonic_id}' has been modified or is invalid.

What it means

ShmObjectStorage.get(address, monotonic_id) validates the monotonic id stored in the buffer's metadata before deserializing. A mismatch means the slot at that address has been reallocated (the old object was freed and a newer object with a higher monotonic id now lives there), or the address/id pair is stale/corrupt. Reading further would return another object's data, so it raises.

Source

Thrown at vllm/distributed/device_communicators/shm_object_storage.py:618

        # Write data to buffer
        with self.ring_buffer.access_buf(address) as (data_view, metadata):
            data_view[: self.flag_bytes] = self.ring_buffer.int2byte(0)
            self.copy_to_buffer(
                object_data, data_bytes, object_metadata, md_bytes, data_view
            )
        self.increment_writer_flag(monotonic_id)

        # Update key index
        self.key_index[key] = (address, monotonic_id)
        self.id_index[monotonic_id] = key
        return address, monotonic_id

    def get(self, address: int, monotonic_id: int) -> Any:
        # Read data from buffer
        with self.ring_buffer.access_buf(address) as (data_view, buf_metadata):
            # check id from metadata
            if buf_metadata[0] != monotonic_id:
                raise ValueError(
                    f"Data for address:id '{address}:{monotonic_id}'"
                    " has been modified or is invalid."
                )

            obj = self.ser_de.deserialize(data_view[self.flag_bytes :])

            # decrease the in-use flag for reader reads
            if self._reader_lock is not None:
                with self._reader_lock:
                    self.increment_reader_flag(data_view[: self.flag_bytes])
            else:
                # if self._reader_lock is None, it means we are the writer
                # in this case, we do not need to decrease the reader count
                assert self.is_writer

        return obj

    def touch(

View on GitHub (pinned to c794754062)

Solutions

Treat the ValueError as a cache miss: re-fetch the object by key from a current key_index lookup (or the producer) instead of reusing a stale address:id pair
Keep readers from lagging: consume objects promptly so in-use flags keep buffers from being freed
Re-lookup the key before every get(): address, mid = storage.key_index[key]

Example fix

# before
obj = storage.get(address, monotonic_id)  # stale after wraparound

# after
address, monotonic_id = storage.key_index[key]  # refresh pair
try:
    obj = storage.get(address, monotonic_id)
except ValueError:
    obj = recompute_or_refetch(key)  # slot was recycled

Defensive patterns

Strategy: try-catch

Validate before calling

address, mid = storage.key_index[key]  # always re-resolve fresh pairs
# verify liveness before the expensive read if API exposes it

Try / catch

try:
    obj = storage.get(address, mid)
except ValueError as e:
    if "modified or is invalid" in str(e):
        obj = refetch_from_producer(key)  # treat as eviction
    else:
        raise

Prevention

Never cache (address, monotonic_id) pairs across consumer pauses; re-resolve from key_index
Keep readers fast enough that buffers are not reclaimed under them
After writer restarts/clears, discard all cached key handles

When it happens

Trigger: Holding an (address, id) pair across a free_unused()/ring-buffer wraparound and then calling get(); using a key_index entry from before the storage was cleared; readers lagging behind while the writer aggressively recycles buffer space.

Common situations: Readers falling behind under memory pressure so buffers get reclaimed (MemoryError-driven free_unused() on the writer); a crash+restart of the writer clearing storage while readers keep old handles; races where consumption outlives retention.

Related errors

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/d491dd689ab6411c. Report an issue: GitHub.