ultralytics/yolov5 · error · RuntimeError

TensorRT engine deserialization failed. Re-export the…

Error message

TensorRT engine deserialization failed. Re-export the engine with the same TensorRT version, CUDA version, and GPU device used for inference.

What it means

DetectMultiBackend raises RuntimeError when TensorRT's deserialize_cuda_engine returns None while loading a .engine file. TensorRT engines are not portable: they are tied to the exact TensorRT version, CUDA version, and GPU architecture they were built with, and deserialization silently returns None on mismatch rather than raising inside TensorRT.

Solutions

1. Re-export the engine on the exact machine/GPU and TensorRT version used for inference: python export.py --weights yolov5s.pt --include engine --device 0.
2. Align versions end-to-end: same tensorrt pip package, same CUDA, same GPU model between export and inference.
3. If you must ship one artifact, ship the .onnx or .pt and build the engine as a deployment step on the target host.
4. Verify integrity of the engine file (size, sha256) if it was copied between hosts.

Example fix

# before (engine exported elsewhere)
model = DetectMultiBackend('yolov5s.engine', device=torch.device('cuda:0'))

# after (build on the inference host, then load)
# python export.py --weights yolov5s.pt --include engine --device 0 --half
model = DetectMultiBackend('yolov5s.engine', device=torch.device('cuda:0'))

import tensorrt as trt

def engine_parses(path: str) -> bool:
    logger = trt.Logger(trt.Logger.ERROR)
    with open(path, 'rb') as f, trt.Runtime(logger) as rt:
        return rt.deserialize_cuda_engine(f.read()) is not None

try:
    model = DetectMultiBackend('yolov5s.engine', device=device)
except RuntimeError as e:
    if 'deserialization failed' in str(e):
        subprocess.run(['python', 'export.py', '--weights', 'yolov5s.pt', '--include', 'engine'], check=True)
        model = DetectMultiBackend('yolov5s.engine', device=device)

When it happens

Trigger: Passing an .engine file exported on another machine to DetectMultiBackend; running an engine built with TensorRT 8.x under TensorRT 10.x; an engine built for a different GPU compute capability (e.g. exported on A100, run on T4); a truncated engine file from a partial copy.

Common situations: Shipping .engine files in Docker images without pinning the TensorRT base image; upgrading the tensorrt pip package (or the NGC container tag) without re-exporting; deploying the same artifact fleet-wide across heterogeneous GPUs.

Related errors

• failed to load ONNX file
• Cannot specify args and kwargs at the same time
• ERROR: is not a supported format
• ERROR: YOLOv5 TF.js inference is not supported
• Expected inputs, got .

AI-assisted analysis of ultralytics/yolov5@20d1d78a08 (2026-08-15). Data as JSON: /api/errors/f3a39ade6fd38ebd. Report an issue: GitHub.

Appendix: source

Thrown at models/common.py:545

                ov_model.get_parameters()[0].set_layout(Layout("NCHW"))
            batch_dim = get_batch(ov_model)
            if batch_dim.is_static:
                batch_size = batch_dim.get_length()
            ov_compiled_model = core.compile_model(ov_model, device_name="AUTO")  # AUTO selects best available device
            stride, names = self._load_metadata(Path(w).with_suffix(".yaml"))  # load metadata
        elif engine:  # TensorRT
            LOGGER.info(f"Loading {w} for TensorRT inference...")
            import tensorrt as trt  # https://developer.nvidia.com/nvidia-tensorrt-download

            check_version(trt.__version__, "7.0.0", hard=True)  # require tensorrt>=7.0.0
            if device.type == "cpu":
                device = torch.device("cuda:0")
            Binding = namedtuple("Binding", ("name", "dtype", "shape", "data", "ptr"))
            logger = trt.Logger(trt.Logger.INFO)
            with open(w, "rb") as f, trt.Runtime(logger) as runtime:
                model = runtime.deserialize_cuda_engine(f.read())
            if model is None:
                raise RuntimeError(
                    "TensorRT engine deserialization failed. Re-export the engine with the same TensorRT version, "
                    "CUDA version, and GPU device used for inference."
                )
            context = model.create_execution_context()
            bindings = OrderedDict()
            output_names = []
            fp16 = False  # default updated below
            dynamic = False
            is_trt10 = not hasattr(model, "num_bindings")
            num = range(model.num_io_tensors) if is_trt10 else range(model.num_bindings)
            for i in num:
                if is_trt10:
                    name = model.get_tensor_name(i)
                    dtype = trt.nptype(model.get_tensor_dtype(name))
                    is_input = model.get_tensor_mode(name) == trt.TensorIOMode.INPUT
                    if is_input:
                        if -1 in tuple(model.get_tensor_shape(name)):  # dynamic
                            dynamic = True

View on GitHub (pinned to 20d1d78a08)