cocoindex-io/cocoindex · warning · UserWarning

COCOINDEX_RUN_GPU_IN_SUBPROCESS=1 with num_gpus={pool.num_gp

Error message

COCOINDEX_RUN_GPU_IN_SUBPROCESS=1 with num_gpus={pool.num_gpus}: subprocess mode does not yet support per-GPU CUDA_VISIBLE_DEVICES. All subprocess calls run on the same GPU regardless of pool assignment. Use in-process mode for multi-GPU support.

What it means

When COCOINDEX_RUN_GPU_IN_SUBPROCESS=1, GPU work runs in spawned subprocesses, but that mode cannot set per-GPU CUDA_VISIBLE_DEVICES, so every subprocess lands on the same GPU no matter which GPU the pool assigned. The library emits this UserWarning once (guarded by a module-level flag) whenever subprocess mode is on and the GPU pool has more than one GPU, so results and performance degrade silently otherwise.

Source

Thrown at python/cocoindex/_internal/runner.py:497

                ),
            )
        finally:
            await self._release_gpu(gpu_id)


GPU = GPURunner(fraction=1.0)

_subprocess_multi_gpu_warned = False


def _warn_subprocess_multi_gpu() -> None:
    global _subprocess_multi_gpu_warned
    if _subprocess_multi_gpu_warned:
        return
    _subprocess_multi_gpu_warned = True
    pool = _get_default_gpu_pool()
    if pool.num_gpus > 1:
        warnings.warn(
            f"COCOINDEX_RUN_GPU_IN_SUBPROCESS=1 with num_gpus={pool.num_gpus}: "
            "subprocess mode does not yet support per-GPU CUDA_VISIBLE_DEVICES. "
            "All subprocess calls run on the same GPU regardless of pool "
            "assignment. Use in-process mode for multi-GPU support.",
            UserWarning,
            stacklevel=4,
        )

View on GitHub (pinned to e84aa99b32)

Solutions

1. Unset COCOINDEX_RUN_GPU_IN_SUBPROCESS (or set it to 0) so GPU work runs in-process with proper per-GPU CUDA_VISIBLE_DEVICES assignment.
2. If subprocess isolation is required, restrict the pool to one GPU so the warning is moot and behavior matches expectations.
3. Ignore the warning only if you intentionally want all subprocess work on a single GPU.

Example fix

// before
COCOINDEX_RUN_GPU_IN_SUBPROCESS=1 python app.py   # multi-GPU machine: all subprocesses hit GPU 0
// after
unset COCOINDEX_RUN_GPU_IN_SUBPROCESS             # in-process mode honors per-GPU assignment

import os
if os.environ.get("COCOINDEX_RUN_GPU_IN_SUBPROCESS") == "1":
    n = _get_default_gpu_pool().num_gpus  # or torch.cuda.device_count()
    if n > 1:
        del os.environ["COCOINDEX_RUN_GPU_IN_SUBPROCESS"]  # force in-process mode

When it happens

Trigger: Setting the environment variable COCOINDEX_RUN_GPU_IN_SUBPROCESS=1 while the default GPU pool (from _get_default_gpu_pool()) reports num_gpus > 1; emitted from _warn_subprocess_multi_gpu, invoked via run/run_sync_fn on the first GPU-pool-using call.

Common situations: Enabling subprocess GPU execution on a multi-GPU machine (e.g. CUDA_VISIBLE_DEVICES lists 2+ GPUs) to isolate CUDA state or work around driver issues, without realizing subprocess mode is single-GPU only.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• Settings.db_path must be provided
• num_gpus must be >= 1, got {num_gpus}
• fraction must be in (0, 1.0], got {fraction}
• Cannot specify both row_factory and row_type
• Columns {invalid_cols} not found in row_type fields: {field_

AI-assisted analysis of cocoindex-io/cocoindex@e84aa99b32 (2026-09-08). Data as JSON: /api/errors/c48476eb940abf6c. Report an issue: GitHub.