ultralytics/yolov5 · error · RuntimeError

TensorRT engine deserialization failed. Re-export the engine

Error message

TensorRT engine deserialization failed. Re-export the engine with the same TensorRT version, CUDA version, and GPU device used for inference.

What it means

DetectMultiBackend raises RuntimeError when TensorRT's deserialize_cuda_engine returns None while loading a .engine file. TensorRT engines are not portable: they are tied to the exact TensorRT version, CUDA version, and GPU architecture they were built with, and deserialization silently returns None on mismatch rather than raising inside TensorRT.

Source

Thrown at models/common.py:545

                ov_model.get_parameters()[0].set_layout(Layout("NCHW"))
            batch_dim = get_batch(ov_model)
            if batch_dim.is_static:
                batch_size = batch_dim.get_length()
            ov_compiled_model = core.compile_model(ov_model, device_name="AUTO")  # AUTO selects best available device
            stride, names = self._load_metadata(Path(w).with_suffix(".yaml"))  # load metadata
        elif engine:  # TensorRT
            LOGGER.info(f"Loading {w} for TensorRT inference...")
            import tensorrt as trt  # https://developer.nvidia.com/nvidia-tensorrt-download

            check_version(trt.__version__, "7.0.0", hard=True)  # require tensorrt>=7.0.0
            if device.type == "cpu":
                device = torch.device("cuda:0")
            Binding = namedtuple("Binding", ("name", "dtype", "shape", "data", "ptr"))
            logger = trt.Logger(trt.Logger.INFO)
            with open(w, "rb") as f, trt.Runtime(logger) as runtime:
                model = runtime.deserialize_cuda_engine(f.read())
            if model is None:
                raise RuntimeError(
                    "TensorRT engine deserialization failed. Re-export the engine with the same TensorRT version, "
                    "CUDA version, and GPU device used for inference."
                )
            context = model.create_execution_context()
            bindings = OrderedDict()
            output_names = []
            fp16 = False  # default updated below
            dynamic = False
            is_trt10 = not hasattr(model, "num_bindings")
            num = range(model.num_io_tensors) if is_trt10 else range(model.num_bindings)
            for i in num:
                if is_trt10:
                    name = model.get_tensor_name(i)
                    dtype = trt.nptype(model.get_tensor_dtype(name))
                    is_input = model.get_tensor_mode(name) == trt.TensorIOMode.INPUT
                    if is_input:
                        if -1 in tuple(model.get_tensor_shape(name)):  # dynamic
                            dynamic = True

View on GitHub (pinned to 20d1d78a08)

Solutions

Re-export the engine on the exact machine/GPU and TensorRT version used for inference: python export.py --weights yolov5s.pt --include engine --device 0.
Align versions end-to-end: same tensorrt pip package, same CUDA, same GPU model between export and inference.
If you must ship one artifact, ship the .onnx or .pt and build the engine as a deployment step on the target host.
Verify integrity of the engine file (size, sha256) if it was copied between hosts.

Example fix

# before (engine exported elsewhere)
model = DetectMultiBackend('yolov5s.engine', device=torch.device('cuda:0'))

# after (build on the inference host, then load)
# python export.py --weights yolov5s.pt --include engine --device 0 --half
model = DetectMultiBackend('yolov5s.engine', device=torch.device('cuda:0'))

Defensive patterns

Strategy: validation

Validate before calling

import tensorrt as trt

def engine_parses(path: str) -> bool:
    logger = trt.Logger(trt.Logger.ERROR)
    with open(path, 'rb') as f, trt.Runtime(logger) as rt:
        return rt.deserialize_cuda_engine(f.read()) is not None

Try / catch

try:
    model = DetectMultiBackend('yolov5s.engine', device=device)
except RuntimeError as e:
    if 'deserialization failed' in str(e):
        subprocess.run(['python', 'export.py', '--weights', 'yolov5s.pt', '--include', 'engine'], check=True)
        model = DetectMultiBackend('yolov5s.engine', device=device)

Prevention

Build engines on the target host as a deployment step, never ship them across GPU types.
Pin the TensorRT/CUDA versions of export and inference containers to identical tags.
Log trt.__version__ and GPU name at both export and inference time for auditability.

When it happens

Trigger: Passing an .engine file exported on another machine to DetectMultiBackend; running an engine built with TensorRT 8.x under TensorRT 10.x; an engine built for a different GPU compute capability (e.g. exported on A100, run on T4); a truncated engine file from a partial copy.

Common situations: Shipping .engine files in Docker images without pinning the TensorRT base image; upgrading the tensorrt pip package (or the NGC container tag) without re-exporting; deploying the same artifact fleet-wide across heterogeneous GPUs.

Related errors

AI-assisted analysis of ultralytics/yolov5@20d1d78a08 (2026-08-15). Data as JSON: /api/errors/f3a39ade6fd38ebd. Report an issue: GitHub.