{"record":{"id":"2c381d553fcc5c25","repo":"Hmbown/CodeWhale","slug":"element-press-was-not-acknowledged","errorCode":null,"errorMessage":"element press was not acknowledged","messagePattern":"element press was not acknowledged","errorType":"exception","errorClass":"ExecError","httpStatus":null,"severity":"error","filePath":"crates/tui/plugins/computer-use/src/backends/darwin.mjs","lineNumber":1009,"sourceCode":"    },\n    preview: async ({ enabled = true } = {}) => {\n      state.previewEnabled = enabled;\n      if (!enabled) { stopPreviewLoop(); await quiescePreview(); await native(\"preview_notify\", { enabled: false }); return { enabled: false }; }\n      if (!state.inputApp) throw new ExecError(\"open_application first to choose the preview app\");\n      return updatePreview(true);\n    },\n    screenshot,\n    zoom,\n    left_click: async ({ target, strategy = \"auto\" } = {}) => {\n      if (target?.type !== \"element\" || strategy === \"event\" || strategy === \"app\") return pointerClick(\"left\", target?.x, target?.y, 1, strategy);\n      if (![\"auto\", \"a11y\"].includes(strategy)) throw new ExecError(`strategy must be auto, a11y, app or event (got ${JSON.stringify(strategy)})`);\n      try {\n        assertBoundElement(target);\n        if ((await native(\"input_capabilities\"))?.element_identity !== 1) throw new ExecError(\"native helper needs an update for element identity validation\");\n        const semantic = [\"AXTextField\", \"AXTextArea\", \"AXComboBox\", \"AXRow\", \"AXCell\", \"AXMenuItem\"].includes(target.role);\n        if (semantic) await requireBackgroundActions();\n        const receipt = await native(semantic ? \"click_element\" : \"perform_action\", { target, action: \"AXPress\" });\n        if (!receipt?.action_sent) throw new ExecError(\"element press was not acknowledged\");\n        return { ...receipt, action: receipt.action ?? \"AXPress\", strategy: \"a11y\", pointer_moved: false,\n          element: { role: target.role, label: target.label ?? null }, verified: receipt.verified ?? false, verification_required: \"observation\" };\n      } catch (error) {\n        // An AX frame can cover other controls. Never turn a refused or\n        // ambiguous element press into another element's press or a raw click.\n        error.message += ' — no coordinate fallback was sent; take a fresh screenshot or OCR observation and choose an advertised action or a separate computer';\n        throw error;\n      }\n    },\n    double_click: ({ target } = {}) => pointerClick(\"left\", target?.x, target?.y, 2),\n    triple_click: ({ target } = {}) => pointerClick(\"left\", target?.x, target?.y, 3),\n    right_click: async ({ target } = {}) => {\n      if (target?.type !== \"element\") return pointerClick(\"right\", target?.x, target?.y, 1);\n      assertBoundElement(target);\n      await requireBackgroundActions();\n      return native(\"click_element\", { target, context: true });\n    },\n    middle_click: ({ target } = {}) => pointerClick(\"middle\", target?.x, target?.y, 1),","sourceCodeStart":991,"sourceCodeEnd":1027,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/73e0f67d83c59909b571efdfc88c4bc28c309cb1/crates/tui/plugins/computer-use/src/backends/darwin.mjs#L991-L1027","documentation":"The darwin computer-use backend pressed an accessibility element (via native click_element or perform_action with AXPress) and the native helper's receipt did not confirm action_sent. The library throws this rather than silently retrying or falling back, because an AX frame can cover other controls — a refused press must never be converted into another element's press or a raw coordinate click. The error message is then augmented with guidance to take a fresh screenshot/OCR observation.","triggerScenarios":"Calling press on an element target whose native AXPress receipt lacks action_sent: the OS accessibility API refused the press (element not pressable, stale index after UI changed), the helper lacked permission, or the target element was swallowed by an overlapping AX frame.","commonSituations":"Stale element index after the app re-rendered between get_app_state and the press; the element's role was advertised as pressable but the app rejected AXPress (web content, non-standard controls); accessibility permissions for the native helper were revoked; a modal sheet or overlay intercepted the accessibility action.","solutions":["Take a fresh screenshot or OCR observation and re-fetch element indices via get_app_state before pressing — the element reference is likely stale.","Verify the target element's role and choose an advertised action it actually supports instead of assuming AXPress.","Re-grant Screen Recording / Accessibility permissions to the native helper, then retry.","If the element is genuinely covered by an AX frame, target a different element or use a separate computer/agent scope as the error suggests."],"exampleFix":"// before: pressing a stale element from an old snapshot\nawait press({ type: 'element', index: 42 });\n\n// after: re-observe, then press with the fresh index and verified receipt\nconst state = await get_app_state();\nconst el = state.elements.find(e => e.role === 'AXButton' && e.label === 'Submit');\nconst receipt = await press({ type: 'element', index: el.index });\nif (!receipt.action_sent) throw new Error('press refused — re-observe');","handlingStrategy":"try-catch","validationCode":"const state = await get_app_state();\nconst el = state.elements.find(e => e.index === target.index);\nif (!el || !['AXButton','AXMenuItem','AXRow','AXCell'].includes(el.role)) {\n  throw new Error('element not pressable in current snapshot — re-observe');\n}","typeGuard":"function isElementTarget(t) {\n  return t != null && t.type === 'element' && Number.isInteger(t.index);\n}","tryCatchPattern":"try {\n  const receipt = await press({ type: 'element', index: i });\n  if (!receipt.action_sent) throw new Error('unacknowledged press');\n} catch (e) {\n  if (String(e.message).includes('not acknowledged')) {\n    // take fresh screenshot / OCR observation and re-select the element\n    await recoverViaObservation();\n  } else throw e;\n}","preventionTips":["Always re-run get_app_state immediately before pressing; never reuse element indices across observations.","Check the receipt.verified / action_sent fields after every press and treat false as a failure requiring re-observation.","Keep the native helper updated so element identity validation is supported (input_capabilities.element_identity).","Prefer advertised actions matching the element's role rather than assuming AXPress."],"tags":["accessibility","macos","element-press","stale-reference"],"backgroundTag":"unexpected-response-shape","analyzedSha":"73e0f67d83c59909b571efdfc88c4bc28c309cb1","analyzedAt":"2026-09-22T01:30:00.501Z","contentChangedAt":"2026-09-22T01:30:00.501Z","schemaVersion":2},"datasetVersion":"2026-09-22T16:17:23.217Z"}