{"record":{"id":"961e01b797eba804","repo":"Hmbown/CodeWhale","slug":"middle-click-is-not-exposed-by-uitest-uiinput","errorCode":null,"errorMessage":"middle click is not exposed by uitest uiInput","messagePattern":"middle click is not exposed by uitest uiInput","errorType":"exception","errorClass":"ExecError","httpStatus":null,"severity":"error","filePath":"crates/tui/plugins/computer-use/src/backends/harmonyos.mjs","lineNumber":225,"sourceCode":"      const dir = process.env.CODEWHALE_CU_RECORDINGS_DIR || path.join(os.homedir(), \".codewhale-cu\", \"recordings\");\n      fs.mkdirSync(dir, { recursive: true });\n      const file = outPath || path.join(dir, `shot-${new Date().toISOString().replace(/[:.]/g, \"-\")}-${crypto.randomBytes(3).toString(\"hex\")}.jpeg`);\n      await snapshot(file);\n      const buf = fs.readFileSync(file);\n      displayPixels = jpegSize(buf) ?? displayPixels;\n      return { file, bytes: buf.length, pixels: jpegSize(buf), scale: 1, points: jpegSize(buf) };\n    },\n    zoom: async ({ region, path: outPath }) => {\n      throw new ExecError(\"zoom is not supported on the harmony backend yet — screenshot + region on the host is the workaround\");\n    },\n    left_click: ({ target, strategy }) => { assertEventStrategy(strategy); return uiInput([\"click\", String(Math.round(target.x)), String(Math.round(target.y))]); },\n    double_click: ({ target }) => uiInput([\"doubleClick\", String(Math.round(target.x)), String(Math.round(target.y))]),\n    triple_click: async ({ target }) => {\n      await uiInput([\"doubleClick\", String(Math.round(target.x)), String(Math.round(target.y))]);\n      return uiInput([\"click\", String(Math.round(target.x)), String(Math.round(target.y))]);\n    },\n    right_click: async () => { throw new ExecError(\"uitest uiInput has no right-click; use longClick semantics via hold or left_click\"); },\n    middle_click: async () => { throw new ExecError(\"middle click is not exposed by uitest uiInput\"); },\n    mouse_move: async () => ({ action_sent: false, note: \"hover without press is not exposed by uitest uiInput\" }),\n    left_click_drag: ({ from_target: from, to }) =>\n      uiInput([\"swipe\", String(Math.round(from.x)), String(Math.round(from.y)), String(Math.round(to.x)), String(Math.round(to.y)), \"200\"], { timeoutMs: 30_000 }),\n    left_mouse_down: async () => { throw new ExecError(\"low-level press/release is not exposed by uitest uiInput; use left_click_drag\"); },\n    left_mouse_up: async () => { throw new ExecError(\"low-level press/release is not exposed by uitest uiInput; use left_click_drag\"); },\n    scroll: ({ target, direction = \"down\", amount = 300 }) => {\n      const dist = Math.max(60, Math.min(1200, amount * 24));\n      const dx = direction === \"left\" ? dist : direction === \"right\" ? -dist : 0;\n      const dy = direction === \"up\" ? dist : direction === \"down\" ? -dist : 0;\n      return uiInput([\"swipe\", String(Math.round(target.x)), String(Math.round(target.y)), String(Math.round(target.x + dx)), String(Math.round(target.y + dy)), \"400\"]);\n    },\n    type: async ({ text }) => {\n      if (!text) return { action_sent: false, note: \"empty text\" };\n      await uiInput([\"inputText\", \"300\", \"300\", escDeviceText(text)]).catch(async (e) => {\n        // Some builds require coordinates of the focused field; retry with a click-first pattern.\n        throw e;\n      });\n      return { action_sent: true, chars: text.length, note: \"inputText at 300,300 — click the field first for focused input\" };","sourceCodeStart":207,"sourceCodeEnd":243,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/73e0f67d83c59909b571efdfc88c4bc28c309cb1/crates/tui/plugins/computer-use/src/backends/harmonyos.mjs#L207-L243","documentation":"uitest's `uiInput` command exposes only click/doubleClick/longClick/swipe/keyEvent/inputText — there is no middle-click primitive, so the harmony backend's `middle_click` slot throws unconditionally. Touch screens have no middle button, and hdc offers no substitute, so the backend fails closed rather than emulating something with different semantics.","triggerScenarios":"Calling the middle_click action ({ action: \"middle_click\", target }) on the harmony backend from harmonyos.mjs; generic tool-loop replaying a recorded desktop action stream containing middle_click against a HarmonyOS device.","commonSituations":"Replaying desktop browser-automation gestures (middle-click = open in new tab) on a phone; a cross-platform agent that emits middle_click for 'auxiliary click'; shared action logs between backend types.","solutions":["Replace middle_click with left_click on touch devices; find the touch equivalent of the gesture (e.g. long-press menu or a dedicated UI control for 'open in new tab').","If the intent was scrolling, use the scroll action instead — it is implemented via swipe on this backend.","Filter the action stream: skip or remap middle_click when the backend is harmonyos, logging the substitution.","File/implement backend support only if a hdc/uitest primitive for middle click appears; today none exists."],"exampleFix":"// before\nawait backend.middle_click({ target });\n\n// after\nif (backendKind === \"harmonyos\") {\n  await backend.left_click({ target }); // middle click has no touch equivalent\n} else {\n  await backend.middle_click({ target });\n}","handlingStrategy":"validation","validationCode":"const MIDDLE_CLICK_UNSUPPORTED = new Set([\"harmonyos\"]);\nfunction canMiddleClick(backend) {\n  return !MIDDLE_CLICK_UNSUPPORTED.has(backend.kind);\n}\nif (!canMiddleClick(backend)) throw new Error(\"remap middle_click before dispatch\");","typeGuard":"const supportsMiddleClick = (backend) => backend.kind !== \"harmonyos\";","tryCatchPattern":"try {\n  await backend.middle_click({ target });\n} catch (e) {\n  if (/middle click is not exposed by uitest/.test(e.message)) {\n    await backend.left_click({ target });\n  } else {\n    throw e;\n  }\n}","preventionTips":["Filter middle_click out of recorded desktop action streams before replaying on touch devices.","Document the touch action vocabulary (click/doubleClick/longClick/swipe/keyEvent/inputText) next to the dispatcher.","Route scrolling needs through the scroll action, never middle-click drag.","Validate plans against a per-backend allowed-action list at plan load time."],"tags":["harmonyos","middle-click","uitest","touch-input","computer-use"],"backgroundTag":"unsupported-operation","analyzedSha":"73e0f67d83c59909b571efdfc88c4bc28c309cb1","analyzedAt":"2026-09-22T01:30:00.501Z","contentChangedAt":"2026-09-22T01:30:00.501Z","schemaVersion":2},"datasetVersion":"2026-09-22T21:17:16.096Z"}