What causes "element has no resolved accessibility identity" in Hmbown/CodeWhale?

Obtain the target from observe()/screenshot annotation output instead of constructing it manually, so path/windowIndex/role are populated.

Hmbown/CodeWhale · error · ExecError

element has no resolved accessibility identity

Error message

element has no resolved accessibility identity

What it means

assertBoundElement also requires that a target element carry a resolved accessibility identity: an array `path` of child indices, an integer `windowIndex`, and a `role`. A target lacking any of these cannot be located in the AX tree, so the backend refuses to act on it rather than guessing.

Solutions

Obtain the target from observe()/screenshot annotation output instead of constructing it manually, so path/windowIndex/role are populated.
Validate the element object has Array path, integer windowIndex, and non-empty role before calling the tool.
Check that your serialization of elements preserves all fields (avoid hand-written JSON for targets).
If the element genuinely cannot be resolved, use a coordinate-based pointer action on a bound app window instead of an element target.

Example fix

// before
await click({ element: { role: "AXButton" } }); // no path/windowIndex

// after (validate first)
if (!Array.isArray(el.path) || !Number.isInteger(el.windowIndex) || !el.role)
  throw new Error("unresolved element — re-observe");
await click({ element: el });

Defensive patterns

Strategy: validation

Validate before calling

function isResolvedElement(el) {
  return !!el && Array.isArray(el.path) && Number.isInteger(el.windowIndex) && typeof el.role === 'string' && el.role.length > 0;
}

Type guard

function isResolvedElement(el) {
  return !!el && Array.isArray(el.path) && Number.isInteger(el.windowIndex) && typeof el.role === 'string' && el.role.length > 0;
}

Try / catch

if (!isResolvedElement(el)) {
  const fresh = await observe(); // re-resolve
  el = fresh[el?.label] ?? null;
}
try {
  await click({ element: el });
} catch (e) {
  if (String(e.message).includes('no resolved accessibility identity')) throw new Error('target unresolved — re-observe required');
  throw e;
}

Prevention

Only pass element objects returned by observe() — never hand-build them.
JSON round-trip elements with full field preservation; avoid lossy serialization.
Validate path/windowIndex/role before every element-targeted tool call.
When the model composes tool arguments, enforce a schema requiring path, windowIndex, and role.

When it happens

Trigger: Passing a hand-crafted or deserialized element object to an input tool where `path` is not an array, `windowIndex` is missing/non-integer, or `role` is empty/absent — e.g. constructing a target from a screenshot coordinate instead of from observe() output, or JSON round-tripping that dropped fields.

Common situations: Manually building element descriptors in scripts; persisting observe() results to disk and re-reading with lossy serialization; LLM-generated tool arguments that omit required element fields; sending a coordinate-style target where an element target is required.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@73e0f67d83 (2026-09-22). Data as JSON: /api/errors/ba6857c11dfa0fe6. Report an issue: GitHub.

Appendix: source

Thrown at crates/tui/plugins/computer-use/src/backends/darwin.mjs:311

    }
    const result = tryJson(r.stdout, null);
    const interference = leaseVerdict(result);
    if (interference !== null) result.user_input_during_lease = interference;
    if (state.previewEnabled && state.inputApp && ["type", "key_event", "pointer_sequence", "bg_pointer", "bg_key", "set_value", "select_text", "perform_action", "hit_test", "click_element", "scroll_element", "focus_element"].includes(tool)) {
      try { await updatePreview(); } catch (error) { result.preview_error = error.message; }
    }
    return result;
  }

  async function requireBackgroundActions() {
    if ((await native("input_capabilities"))?.background_actions !== 1) {
      throw Object.assign(new ExecError("Update the Computer Use helper to use background focus, selection, context menus and scrolling."), { code: "app_upgrade_required" });
    }
  }

  function assertBoundElement(target) {
    if (!state.inputApp || target.app_ref?.pid !== state.inputApp.pid) throw new ExecError("element does not belong to the bound application — open_application and observe again");
    if (!Array.isArray(target.path) || !Number.isInteger(target.windowIndex) || !target.role) throw new ExecError("element has no resolved accessibility identity");
  }

  async function nativeLease(tool, args) {
    if (tool === "pointer_sequence") requireSharedPointer();
    if (!exec.runInputLease) throw new ExecError("This executor cannot safely own held input; update Computer Use");
    if ((await native("input_capabilities"))?.input_lease !== 1) throw new ExecError("The native helper needs an update for disconnect-safe held input");
    const helper = await nativeHelper();
    try {
      return await exec.runInputLease(helper, [JSON.stringify({ tool, args: { ...args, ...yieldArgs, input_app_ref: state.inputApp, foreground_input: state.foregroundInput, owner_pipe: true, input_lease: true } })]);
    } catch (error) {
      error.code = nativeErrorCode(error.message) ?? error.code;
      throw error;
    }
  }

  async function updatePreview(show = false) {
    const win = await native("window_info", { app_ref: state.inputApp });
    const dir = path.join(stateDir(), "preview");

View on GitHub (pinned to 73e0f67d83)