xai-org/x-algorithm · error · std::runtime_error

NCCL version query failed: ${ncclGetErrorString(result)}

Error message

NCCL version query failed: ${ncclGetErrorString(result)}

What it means

The nanobind module exposes nccl_version(), which calls ncclGetVersion. If the NCCL call returns anything other than ncclSuccess, a std::runtime_error carrying ncclGetErrorString(result) is thrown — indicating the linked NCCL library is broken or failed the trivial version query.

Source

Thrown at phoenix/xrex/cuda/async_emb/src/async_emb_api.cc:833

      nb::call_guard<nb::gil_scoped_release>()
  );
  m.def(
      "reset_table_binding",
      [](int64_t context_id) {
        auto ctx = readyContext(context_id);
        if (ctx == nullptr) {
          throw std::invalid_argument("async_emb context not initialized");
        }
        ctx->resetTableBinding();
      },
      nb::call_guard<nb::gil_scoped_release>()
  );
  m.def("_test_snapshot", &testSnapshot);
  m.def("nccl_version", [] {
    int version = 0;
    ncclResult_t result = ncclGetVersion(&version);
    if (result != ncclSuccess) {
      throw std::runtime_error(ncclGetErrorString(result));
    }
    return version;
  });
  m.def("nccl_library_path", [] {
    Dl_info info{};
    if (dladdr(reinterpret_cast<void*>(&ncclGetVersion), &info) == 0 || info.dli_fname == nullptr) {
      throw std::runtime_error("could not resolve loaded NCCL library path");
    }
    return std::string(info.dli_fname);
  });
}

}

View on GitHub (pinned to 24c60942c5)

Solutions

1. Verify which NCCL is loaded: use async_emb.nccl_library_path() or ldd on the extension .so
2. Rebuild/reinstall the extension against the NCCL version present at runtime (or fix LD_LIBRARY_PATH so the built-against version loads)
3. If using containers, align the NCCL version with the one used to build phoenix/xrex cuda extensions

# cheap preflight: confirm a NCCL library is resolvable
import ctypes; ctypes.CDLL("libnccl.so")

try:
    v = async_emb.nccl_version()
except RuntimeError as e:
    log.error("NCCL broken: %s; path=%s", e, None)
    raise SystemExit("fix NCCL install/library path")

When it happens

Trigger: Calling async_emb.nccl_version() when ncclGetVersion fails — rare, but possible with a corrupted/ABI-mismatched NCCL shared library or a dlopen'd stub that returns an error.

Common situations: Mixing NCCL versions (e.g. a conda nccl shadowing the system one the extension was built against); broken LD_LIBRARY_PATH resolution; container images with mismatched NCCL builds.

Related errors

• could not resolve loaded NCCL library path
• async_emb: ${label} timed out waiting for step ${step}
• NCCL ${operation} failed: ${ncclGetErrorString(result)}
• async_emb context not initialized
• unknown async_emb snapshot region: ${region}

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/1981ef127fa545b6. Report an issue: GitHub.