apache/beam · error · RuntimeError

Cuda Error

Error message

Cuda Error: {}

What it means

_assign_or_fail performs CUDA error checking on results returned by cuda-python bindings. If the returned CUresult code is not CUDA_SUCCESS, it raises RuntimeError('Cuda Error: {code}') with the numeric/enum error value.

Solutions

1. Decode the reported CUresult code to identify the specific failure (e.g. via cuda.CUresult(code).name)
2. Verify workers are GPU instances with a matching CUDA driver and that the image's CUDA/tensorrt versions are compatible
3. Check the model/engine size fits the GPU memory budget and adjust builder config (max workspace, precision) accordingly
4. Confirm the runner/scheduler exposes GPUs and CUDA_VISIBLE_DEVICES is not hiding devices

Example fix

// before
handler = TensorRTEngineHandlerNumpy(..., precision_mode=bf16)  # GPU not visible on worker
// after
# run on GPU worker pool, or pick precision supported by the device
handler = TensorRTEngineHandlerNumpy(..., precision_mode=fp16)

from cuda import cuda
def check_cuda_available():
    err, = cuda.cuInit(0)
    if err != cuda.CUresult.CUDA_SUCCESS:
        raise RuntimeError(f'CUDA init failed: {cuda.CUresult(err).name}')

from cuda import cuda
def is_cuda_success(result) -> bool:
    err = result[0]
    return isinstance(err, cuda.CUresult) and err == cuda.CUresult.CUDA_SUCCESS

try:
    predictions = pcoll | RunInference(trt_handler)
except RuntimeError as e:
    if str(e).startswith('Cuda Error'):
        raise RuntimeError('Check GPU availability, driver/CUDA version match, and memory limits on workers') from e
    raise

When it happens

Trigger: Any CUDA API call routed through _assign_or_fail (during TensorRT engine setup in __init__ or inference in _default_tensorRT_inference_fn) returning a non-success status, e.g. CUDA_ERROR_INVALID_VALUE, CUDA_ERROR_OUT_OF_MEMORY, or CUDA_ERROR_NO_DEVICE.

Common situations: Worker machines without a visible GPU or with the wrong CUDA driver; requesting an engine/device config exceeding GPU memory; running the pipeline on a CPU-only runner while the handler requires a GPU; CUDA driver/runtime version mismatch in the container.

Related errors

• Unknown error type
• Failed to load ONNX file
• inference_args were provided, but should be None because…
• Callable create_model_fn must be passedwith…
• Cannot make make an unkeyed model handler with pre or…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/63f8518552cccfc9. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/inference/tensorrt_inference.py:92

def _build_engine(network, builder):
  import tensorrt as trt
  config = builder.create_builder_config()
  runtime = trt.Runtime(TRT_LOGGER)
  plan = builder.build_serialized_network(network, config)
  engine = runtime.deserialize_cuda_engine(plan)
  builder.reset()
  return engine


def _assign_or_fail(args):
  """CUDA error checking."""
  from cuda import cuda
  err, ret = args[0], args[1:]
  if isinstance(err, cuda.CUresult):
    if err != cuda.CUresult.CUDA_SUCCESS:
      raise RuntimeError("Cuda Error: {}".format(err))
  else:
    raise RuntimeError("Unknown error type: {}".format(err))
  # Special case so that no unpacking is needed at call-site.
  if len(ret) == 1:
    return ret[0]
  return ret


class TensorRTEngine:
  def __init__(self, engine: trt.ICudaEngine):
    """Implementation of the TensorRTEngine class which handles
    allocations associated with TensorRT engine.

    Example Usage::

      TensorRTEngine(engine)

    Args:

View on GitHub (pinned to 12126d8942)