Hmbown/CodeWhale · error · ExecError

element press was not acknowledged

Error message

element press was not acknowledged

What it means

The darwin computer-use backend pressed an accessibility element (via native click_element or perform_action with AXPress) and the native helper's receipt did not confirm action_sent. The library throws this rather than silently retrying or falling back, because an AX frame can cover other controls — a refused press must never be converted into another element's press or a raw coordinate click. The error message is then augmented with guidance to take a fresh screenshot/OCR observation.

Solutions

  1. Take a fresh screenshot or OCR observation and re-fetch element indices via get_app_state before pressing — the element reference is likely stale.
  2. Verify the target element's role and choose an advertised action it actually supports instead of assuming AXPress.
  3. Re-grant Screen Recording / Accessibility permissions to the native helper, then retry.
  4. If the element is genuinely covered by an AX frame, target a different element or use a separate computer/agent scope as the error suggests.

Example fix

// before: pressing a stale element from an old snapshot
await press({ type: 'element', index: 42 });

// after: re-observe, then press with the fresh index and verified receipt
const state = await get_app_state();
const el = state.elements.find(e => e.role === 'AXButton' && e.label === 'Submit');
const receipt = await press({ type: 'element', index: el.index });
if (!receipt.action_sent) throw new Error('press refused — re-observe');
Defensive patterns

Strategy: try-catch

Validate before calling

const state = await get_app_state();
const el = state.elements.find(e => e.index === target.index);
if (!el || !['AXButton','AXMenuItem','AXRow','AXCell'].includes(el.role)) {
  throw new Error('element not pressable in current snapshot — re-observe');
}

Type guard

function isElementTarget(t) {
  return t != null && t.type === 'element' && Number.isInteger(t.index);
}

Try / catch

try {
  const receipt = await press({ type: 'element', index: i });
  if (!receipt.action_sent) throw new Error('unacknowledged press');
} catch (e) {
  if (String(e.message).includes('not acknowledged')) {
    // take fresh screenshot / OCR observation and re-select the element
    await recoverViaObservation();
  } else throw e;
}

Prevention

When it happens

Trigger: Calling press on an element target whose native AXPress receipt lacks action_sent: the OS accessibility API refused the press (element not pressable, stale index after UI changed), the helper lacked permission, or the target element was swallowed by an overlapping AX frame.

Common situations: Stale element index after the app re-rendered between get_app_state and the press; the element's role was advertised as pressable but the app rejected AXPress (web content, non-standard controls); accessibility permissions for the native helper were revoked; a modal sheet or overlay intercepted the accessibility action.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@73e0f67d83 (2026-09-22). Data as JSON: /api/errors/2c381d553fcc5c25. Report an issue: GitHub.

Appendix: source

Thrown at crates/tui/plugins/computer-use/src/backends/darwin.mjs:1009

    },
    preview: async ({ enabled = true } = {}) => {
      state.previewEnabled = enabled;
      if (!enabled) { stopPreviewLoop(); await quiescePreview(); await native("preview_notify", { enabled: false }); return { enabled: false }; }
      if (!state.inputApp) throw new ExecError("open_application first to choose the preview app");
      return updatePreview(true);
    },
    screenshot,
    zoom,
    left_click: async ({ target, strategy = "auto" } = {}) => {
      if (target?.type !== "element" || strategy === "event" || strategy === "app") return pointerClick("left", target?.x, target?.y, 1, strategy);
      if (!["auto", "a11y"].includes(strategy)) throw new ExecError(`strategy must be auto, a11y, app or event (got ${JSON.stringify(strategy)})`);
      try {
        assertBoundElement(target);
        if ((await native("input_capabilities"))?.element_identity !== 1) throw new ExecError("native helper needs an update for element identity validation");
        const semantic = ["AXTextField", "AXTextArea", "AXComboBox", "AXRow", "AXCell", "AXMenuItem"].includes(target.role);
        if (semantic) await requireBackgroundActions();
        const receipt = await native(semantic ? "click_element" : "perform_action", { target, action: "AXPress" });
        if (!receipt?.action_sent) throw new ExecError("element press was not acknowledged");
        return { ...receipt, action: receipt.action ?? "AXPress", strategy: "a11y", pointer_moved: false,
          element: { role: target.role, label: target.label ?? null }, verified: receipt.verified ?? false, verification_required: "observation" };
      } catch (error) {
        // An AX frame can cover other controls. Never turn a refused or
        // ambiguous element press into another element's press or a raw click.
        error.message += ' — no coordinate fallback was sent; take a fresh screenshot or OCR observation and choose an advertised action or a separate computer';
        throw error;
      }
    },
    double_click: ({ target } = {}) => pointerClick("left", target?.x, target?.y, 2),
    triple_click: ({ target } = {}) => pointerClick("left", target?.x, target?.y, 3),
    right_click: async ({ target } = {}) => {
      if (target?.type !== "element") return pointerClick("right", target?.x, target?.y, 1);
      assertBoundElement(target);
      await requireBackgroundActions();
      return native("click_element", { target, context: true });
    },
    middle_click: ({ target } = {}) => pointerClick("middle", target?.x, target?.y, 1),

View on GitHub (pinned to 73e0f67d83)