{"record":{"id":"8285e64ec41bbde8","repo":"Hmbown/CodeWhale","slug":"uitest-uiinput-has-no-right-click-use-longclick-semantics","errorCode":null,"errorMessage":"uitest uiInput has no right-click; use longClick semantics via hold or left_click","messagePattern":"uitest uiInput has no right-click; use longClick semantics via hold or left_click","errorType":"exception","errorClass":"ExecError","httpStatus":null,"severity":"error","filePath":"crates/tui/plugins/computer-use/src/backends/harmonyos.mjs","lineNumber":224,"sourceCode":"      const { path: outPath } = args;\n      const dir = process.env.CODEWHALE_CU_RECORDINGS_DIR || path.join(os.homedir(), \".codewhale-cu\", \"recordings\");\n      fs.mkdirSync(dir, { recursive: true });\n      const file = outPath || path.join(dir, `shot-${new Date().toISOString().replace(/[:.]/g, \"-\")}-${crypto.randomBytes(3).toString(\"hex\")}.jpeg`);\n      await snapshot(file);\n      const buf = fs.readFileSync(file);\n      displayPixels = jpegSize(buf) ?? displayPixels;\n      return { file, bytes: buf.length, pixels: jpegSize(buf), scale: 1, points: jpegSize(buf) };\n    },\n    zoom: async ({ region, path: outPath }) => {\n      throw new ExecError(\"zoom is not supported on the harmony backend yet — screenshot + region on the host is the workaround\");\n    },\n    left_click: ({ target, strategy }) => { assertEventStrategy(strategy); return uiInput([\"click\", String(Math.round(target.x)), String(Math.round(target.y))]); },\n    double_click: ({ target }) => uiInput([\"doubleClick\", String(Math.round(target.x)), String(Math.round(target.y))]),\n    triple_click: async ({ target }) => {\n      await uiInput([\"doubleClick\", String(Math.round(target.x)), String(Math.round(target.y))]);\n      return uiInput([\"click\", String(Math.round(target.x)), String(Math.round(target.y))]);\n    },\n    right_click: async () => { throw new ExecError(\"uitest uiInput has no right-click; use longClick semantics via hold or left_click\"); },\n    middle_click: async () => { throw new ExecError(\"middle click is not exposed by uitest uiInput\"); },\n    mouse_move: async () => ({ action_sent: false, note: \"hover without press is not exposed by uitest uiInput\" }),\n    left_click_drag: ({ from_target: from, to }) =>\n      uiInput([\"swipe\", String(Math.round(from.x)), String(Math.round(from.y)), String(Math.round(to.x)), String(Math.round(to.y)), \"200\"], { timeoutMs: 30_000 }),\n    left_mouse_down: async () => { throw new ExecError(\"low-level press/release is not exposed by uitest uiInput; use left_click_drag\"); },\n    left_mouse_up: async () => { throw new ExecError(\"low-level press/release is not exposed by uitest uiInput; use left_click_drag\"); },\n    scroll: ({ target, direction = \"down\", amount = 300 }) => {\n      const dist = Math.max(60, Math.min(1200, amount * 24));\n      const dx = direction === \"left\" ? dist : direction === \"right\" ? -dist : 0;\n      const dy = direction === \"up\" ? dist : direction === \"down\" ? -dist : 0;\n      return uiInput([\"swipe\", String(Math.round(target.x)), String(Math.round(target.y)), String(Math.round(target.x + dx)), String(Math.round(target.y + dy)), \"400\"]);\n    },\n    type: async ({ text }) => {\n      if (!text) return { action_sent: false, note: \"empty text\" };\n      await uiInput([\"inputText\", \"300\", \"300\", escDeviceText(text)]).catch(async (e) => {\n        // Some builds require coordinates of the focused field; retry with a click-first pattern.\n        throw e;\n      });","sourceCodeStart":206,"sourceCodeEnd":242,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/73e0f67d83c59909b571efdfc88c4bc28c309cb1/crates/tui/plugins/computer-use/src/backends/harmonyos.mjs#L206-L242","documentation":"The harmony backend maps input injection to uitest's `uiInput` shell command, which has no right-click sub-command (touch devices have no secondary button). `right_click` therefore always throws this ExecError. The message points at the available substitute: longClick semantics via hold_key({text:\"click\"}) or a plain left_click.","triggerScenarios":"Calling the right_click action ({ action: \"right_click\", target }) on a backend created in harmonyos.mjs; an agent plan written for desktop that includes context-menu right-clicks run against a HarmonyOS device. Unconditional — target coordinates are never examined.","commonSituations":"Desktop-oriented automation scripts that open context menus; a shared action vocabulary executed across multiple backends where right_click is valid on some but not harmonyos; ported test suites that assume mouse semantics.","solutions":["Replace the right-click with longClick semantics: perform_action({ target, action: \"longClick\" }) or hold_key({ text: \"click\" }) — this is the touch-device idiom for a context menu.","If a long press does not open the desired menu, use left_click instead and drive the UI through whatever on-screen affordance exists.","Guard in the caller: branch on backend type (or a capabilities flag) and rewrite right_click to longClick before dispatching.","If the device build ever gains right-click input, implement it in harmonyos.mjs's right_click slot; until then treat it as unsupported."],"exampleFix":"// before\nawait backend.right_click({ target });\n\n// after\n// long-press is the touch equivalent of right-click\nawait backend.perform_action({ target, action: \"longClick\" });","handlingStrategy":"try-catch","validationCode":"const isTouchBackend = (backend) => backend.kind === \"harmonyos\";\nif (isTouchBackend(backend)) {\n  await backend.perform_action({ target, action: \"longClick\" });\n} else {\n  await backend.right_click({ target });\n}","typeGuard":"const supportsRightClick = (backend) => backend.kind !== \"harmonyos\";","tryCatchPattern":"try {\n  await backend.right_click({ target });\n} catch (e) {\n  if (/uitest uiInput has no right-click/.test(e.message)) {\n    await backend.perform_action({ target, action: \"longClick\" });\n  } else {\n    throw e;\n  }\n}","preventionTips":["Treat right_click as a desktop-only action; map it to longClick for touch backends in the action-dispatch layer.","Author agent prompts/plans with backend-aware action vocabularies.","Add a smoke test per backend that enumerates every action so unsupported slots surface in CI, not in the field.","Prefer composite gestures (longClick, swipe) when writing portable plans."],"tags":["harmonyos","right-click","uitest","touch-input","computer-use"],"backgroundTag":"unsupported-operation","analyzedSha":"73e0f67d83c59909b571efdfc88c4bc28c309cb1","analyzedAt":"2026-09-22T01:30:00.501Z","contentChangedAt":"2026-09-22T01:30:00.501Z","schemaVersion":2},"datasetVersion":"2026-09-22T21:17:16.096Z"}