{"record":{"id":"1981ef127fa545b6","repo":"xai-org/x-algorithm","slug":"nccl-version-query-failed-ncclgeterrorstring-re","errorCode":null,"errorMessage":"NCCL version query failed: ${ncclGetErrorString(result)}","messagePattern":"NCCL version query failed: (.+?)","errorType":"exception","errorClass":"std::runtime_error","httpStatus":null,"severity":"error","filePath":"phoenix/xrex/cuda/async_emb/src/async_emb_api.cc","lineNumber":833,"sourceCode":"      nb::call_guard<nb::gil_scoped_release>()\n  );\n  m.def(\n      \"reset_table_binding\",\n      [](int64_t context_id) {\n        auto ctx = readyContext(context_id);\n        if (ctx == nullptr) {\n          throw std::invalid_argument(\"async_emb context not initialized\");\n        }\n        ctx->resetTableBinding();\n      },\n      nb::call_guard<nb::gil_scoped_release>()\n  );\n  m.def(\"_test_snapshot\", &testSnapshot);\n  m.def(\"nccl_version\", [] {\n    int version = 0;\n    ncclResult_t result = ncclGetVersion(&version);\n    if (result != ncclSuccess) {\n      throw std::runtime_error(ncclGetErrorString(result));\n    }\n    return version;\n  });\n  m.def(\"nccl_library_path\", [] {\n    Dl_info info{};\n    if (dladdr(reinterpret_cast<void*>(&ncclGetVersion), &info) == 0 || info.dli_fname == nullptr) {\n      throw std::runtime_error(\"could not resolve loaded NCCL library path\");\n    }\n    return std::string(info.dli_fname);\n  });\n}\n\n}\n","sourceCodeStart":815,"sourceCodeEnd":847,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/phoenix/xrex/cuda/async_emb/src/async_emb_api.cc#L815-L847","documentation":"The nanobind module exposes nccl_version(), which calls ncclGetVersion. If the NCCL call returns anything other than ncclSuccess, a std::runtime_error carrying ncclGetErrorString(result) is thrown — indicating the linked NCCL library is broken or failed the trivial version query.","triggerScenarios":"Calling async_emb.nccl_version() when ncclGetVersion fails — rare, but possible with a corrupted/ABI-mismatched NCCL shared library or a dlopen'd stub that returns an error.","commonSituations":"Mixing NCCL versions (e.g. a conda nccl shadowing the system one the extension was built against); broken LD_LIBRARY_PATH resolution; container images with mismatched NCCL builds.","solutions":["Verify which NCCL is loaded: use async_emb.nccl_library_path() or ldd on the extension .so","Rebuild/reinstall the extension against the NCCL version present at runtime (or fix LD_LIBRARY_PATH so the built-against version loads)","If using containers, align the NCCL version with the one used to build phoenix/xrex cuda extensions"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"# cheap preflight: confirm a NCCL library is resolvable\nimport ctypes; ctypes.CDLL(\"libnccl.so\")","typeGuard":null,"tryCatchPattern":"try:\n    v = async_emb.nccl_version()\nexcept RuntimeError as e:\n    log.error(\"NCCL broken: %s; path=%s\", e, None)\n    raise SystemExit(\"fix NCCL install/library path\")","preventionTips":["Pin NCCL version in containers to the build-time version","Verify with ldd that the extension resolves the intended libnccl"],"tags":["async-emb","cpp","nccl","version-mismatch","environment"],"backgroundTag":"nccl-library-mismatch","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}