sgl-project/sglang · critical · OSError

Failed to create shm file: {e}

Error message

Failed to create shm file: {e}

What it means

os.open('/dev/shm/...') failed while creating the shared-memory backing file for the host KV cache pool. The original exception is wrapped into OSError with 'Failed to create shm file'. Typical underlying causes are ENOSPC (tmpfs full) or EACCES/ENOENT on /dev/shm.

Source

Thrown at python/sglang/srt/mem_cache/storage/mmap/mmap_allocator.py:178

    # Create an anonymous shared memory file descriptor via memfd_create
    fd = None
    try:
        # MFD_CLOEXEC is standard on Linux 3.17+
        fd = os.memfd_create(
            f"sglang_host_pool_{uuid.uuid4().hex}",
            flags=getattr(os, "MFD_CLOEXEC", 1),
        )
    except (AttributeError, OSError):
        # Fallback to creating a file in /dev/shm if memfd_create is not supported
        shm_path = f"/dev/shm/sglang_host_pool_{uuid.uuid4().hex}.mmap"
        try:
            fd = os.open(shm_path, os.O_CREAT | os.O_RDWR | os.O_TRUNC, 0o600)
            try:
                os.unlink(shm_path)
            except OSError:
                pass
        except Exception as e:
            raise OSError(f"Failed to create shm file: {e}")

    try:
        os.ftruncate(fd, alloc_bytes)
        mm = mmap.mmap(
            fd,
            alloc_bytes,
            flags=mmap.MAP_SHARED | _MAP_POPULATE,
            prot=mmap.PROT_READ | mmap.PROT_WRITE,
        )
        try:
            # MADV_POPULATE_WRITE guarantees pages are populated and writable,
            # throwing an error on failure (e.g. out of memory).
            mm.madvise(_MADV_POPULATE_WRITE)
        except OSError:
            # Fall back to MAP_POPULATE if MADV_POPULATE_WRITE is not supported (<5.14 kernel).
            pass
    except Exception as e:
        if fd is not None:

View on GitHub (pinned to 0132848349)

Solutions

1. Enlarge tmpfs: mount -o remount,size=... /dev/shm or docker --shm-size=
2. Lower --hicache-size below the /dev/shm capacity
3. Clean stale files in /dev/shm
4. Verify /dev/shm is mounted rw (cat /proc/mounts | grep shm)

Example fix

# before
docker run ... # default 64MB shm; hicache-size 10gb -> OSError
# after
docker run --shm-size=32gb ... sglang.launch_server --enable-hicache --hicache-size 20gb

import shutil, os
def shm_can_hold(n_bytes: int) -> bool:
    total, used, free = shutil.disk_usage('/dev/shm')
    return free > n_bytes
assert shm_can_hold(alloc_bytes), '/dev/shm too small; raise --shm-size or lower hicache-size'

try:
    t = alloc_shm(name, size)
except OSError as e:
    if 'Failed to create shm file' in str(e):
        # inspect root cause via __cause__; enlarge /dev/shm or shrink pool
        raise

When it happens

Trigger: alloc_shm() opening /dev/shm/<name> with O_CREAT|O_RDWR|O_TRUNC when /dev/shm has no free space, is not mounted, or the process lacks write permission; too many stale files with the same name.

Common situations: hicache-size larger than the container's /dev/shm limit (common in Docker, default 64MB); read-only or missing /dev/shm in minimal containers; leftover shm files from crashed runs filling tmpfs.

Related errors

• /v1/models ${response.status}
• Unsupported msgpack byte ${b}
• This browser does not support gzip stream decoding
• MiniMax H3 requires ffmpeg and ffprobe for media processing
• Image path not found: {batch.image_path}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/cdd27e7dda0bc139. Report an issue: GitHub.