xai-org/x-algorithm · critical · ImportError

the compiled async_emb extension at {async_emb_api.file}

Error message

the compiled async_emb extension at {async_emb_api.__file__} does not match these bindings (stale build?): {e}

What it means

The bindings register FFI targets against the compiled async_emb_api extension by calling attribute accessors like rowwise_adagrad_update_done(); an AttributeError means the installed .so at async_emb_api.__file__ was built from an older/newer source than these Python bindings expect. It is a stale or version-mismatched build of the extension relative to the bindings module.

Source

Thrown at phoenix/xrex/cuda/async_emb/async_emb.py:59

            "execute": async_emb_api.rowwise_adagrad_update_start(),
        },
        platform="CUDA",
    )
    jax.ffi.register_ffi_target(
        "xrex_async_emb_rowwise_adagrad_lazy_update_start",
        fn={
            "initialize": async_emb_api.rowwise_adagrad_lazy_update_start_init(),
            "execute": async_emb_api.rowwise_adagrad_lazy_update_start(),
        },
        platform="CUDA",
    )
    jax.ffi.register_ffi_target(
        "xrex_async_emb_rowwise_adagrad_update_done",
        fn=async_emb_api.rowwise_adagrad_update_done(),
        platform="CUDA",
    )
except AttributeError as e:
    raise ImportError(
        f"the compiled async_emb extension at {async_emb_api.__file__} does not "
        f"match these bindings (stale build?): {e}"
    ) from e


class AsyncEmbContextHandle(NamedTuple):
    context_id: int
    group_size: int
    flatten_replicas: tuple[int, ...]
    tokens_per_rank: int
    shard_width: int
    emb_width: int
    num_unique: int
    num_devices_per_node: int
    mesh: jax.sharding.Mesh
    table_axis: tuple[str, ...]
    data_axis: tuple[str, ...]

View on GitHub (pinned to 24c60942c5)

Solutions

Rebuild the async_emb extension from the current checkout (see build notes in xrex/cuda/__init__.py), making sure the freshly built module is the one being imported.
Print async_emb_api.__file__ (e.g. python -c "import xrex.cuda.async_emb.src.async_emb_api as m; print(m.__file__)") to find the stale .so and delete/replace it.
Uninstall any wheel version of the extension that shadows your source build, then rebuild.
Verify the required symbols exist: dir(async_emb_api) should include rowwise_adagrad_update_done and the other registered entry points.

Example fix

# before: stale .so missing new symbols
python -c "from xrex.cuda.async_emb.src import async_emb_api; print(async_emb_api.__file__)"
# -> /site-packages/async_emb_api.cpython-310.so   (old wheel)

# after: remove stale artifact and rebuild from current source
pip uninstall async_emb  # if a wheel shadows the build
cd phoenix/xrex/cuda/async_emb/src && python setup.py build_ext --inplace
python -c "from xrex.cuda.async_emb.src import async_emb_api; print(async_emb_api.__file__)"
# -> .../phoenix/xrex/cuda/async_emb/src/async_emb_api...so

Defensive patterns

Strategy: validation

Validate before calling

from xrex.cuda.async_emb.src import async_emb_api
REQUIRED = ("rowwise_adagrad_update_done",)  # extend with the other FFI entry points
missing = [s for s in REQUIRED if not hasattr(async_emb_api, s)]
if missing:
    raise RuntimeError(
        f"stale async_emb build at {async_emb_api.__file__}; rebuild src/. missing: {missing}"
    )

Try / catch

try:
    from xrex.cuda.async_emb.async_emb import AsyncEmbContextHandle
except ImportError as e:
    if "stale build" in str(e):
        raise RuntimeError("rebuild xrex/cuda/async_emb/src after pulling new code") from e
    raise

Prevention

Rebuild the extension as part of your pull/CI script (make cuda or equivalent), not manually.
Check async_emb_api.__file__ points into your checkout, not an old wheel.

When it happens

Trigger: Compiling the async_emb src/ from one repo checkout (or an older commit) and then running Python bindings from another; pulling new code that registers new FFI symbols while the previously built .so lacks them; partial rebuild that copied an old artifact.

Common situations: git pull / branch switch after the kernels gained new entry points; a wheel-installed extension alongside a source checkout; Docker layer caching that reuses an old build; mixing pip install -e with a manually built .so.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/ac811879231ae8ab. Report an issue: GitHub.