Hmbown/CodeWhale · error · ExecError

middle click is not exposed by uitest uiInput

Error message

middle click is not exposed by uitest uiInput

What it means

uitest's `uiInput` command exposes only click/doubleClick/longClick/swipe/keyEvent/inputText — there is no middle-click primitive, so the harmony backend's `middle_click` slot throws unconditionally. Touch screens have no middle button, and hdc offers no substitute, so the backend fails closed rather than emulating something with different semantics.

Solutions

  1. Replace middle_click with left_click on touch devices; find the touch equivalent of the gesture (e.g. long-press menu or a dedicated UI control for 'open in new tab').
  2. If the intent was scrolling, use the scroll action instead — it is implemented via swipe on this backend.
  3. Filter the action stream: skip or remap middle_click when the backend is harmonyos, logging the substitution.
  4. File/implement backend support only if a hdc/uitest primitive for middle click appears; today none exists.

Example fix

// before
await backend.middle_click({ target });

// after
if (backendKind === "harmonyos") {
  await backend.left_click({ target }); // middle click has no touch equivalent
} else {
  await backend.middle_click({ target });
}
Defensive patterns

Strategy: validation

Validate before calling

const MIDDLE_CLICK_UNSUPPORTED = new Set(["harmonyos"]);
function canMiddleClick(backend) {
  return !MIDDLE_CLICK_UNSUPPORTED.has(backend.kind);
}
if (!canMiddleClick(backend)) throw new Error("remap middle_click before dispatch");

Type guard

const supportsMiddleClick = (backend) => backend.kind !== "harmonyos";

Try / catch

try {
  await backend.middle_click({ target });
} catch (e) {
  if (/middle click is not exposed by uitest/.test(e.message)) {
    await backend.left_click({ target });
  } else {
    throw e;
  }
}

Prevention

When it happens

Trigger: Calling the middle_click action ({ action: "middle_click", target }) on the harmony backend from harmonyos.mjs; generic tool-loop replaying a recorded desktop action stream containing middle_click against a HarmonyOS device.

Common situations: Replaying desktop browser-automation gestures (middle-click = open in new tab) on a phone; a cross-platform agent that emits middle_click for 'auxiliary click'; shared action logs between backend types.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of Hmbown/CodeWhale@73e0f67d83 (2026-09-22). Data as JSON: /api/errors/961e01b797eba804. Report an issue: GitHub.

Appendix: source

Thrown at crates/tui/plugins/computer-use/src/backends/harmonyos.mjs:225

      const dir = process.env.CODEWHALE_CU_RECORDINGS_DIR || path.join(os.homedir(), ".codewhale-cu", "recordings");
      fs.mkdirSync(dir, { recursive: true });
      const file = outPath || path.join(dir, `shot-${new Date().toISOString().replace(/[:.]/g, "-")}-${crypto.randomBytes(3).toString("hex")}.jpeg`);
      await snapshot(file);
      const buf = fs.readFileSync(file);
      displayPixels = jpegSize(buf) ?? displayPixels;
      return { file, bytes: buf.length, pixels: jpegSize(buf), scale: 1, points: jpegSize(buf) };
    },
    zoom: async ({ region, path: outPath }) => {
      throw new ExecError("zoom is not supported on the harmony backend yet — screenshot + region on the host is the workaround");
    },
    left_click: ({ target, strategy }) => { assertEventStrategy(strategy); return uiInput(["click", String(Math.round(target.x)), String(Math.round(target.y))]); },
    double_click: ({ target }) => uiInput(["doubleClick", String(Math.round(target.x)), String(Math.round(target.y))]),
    triple_click: async ({ target }) => {
      await uiInput(["doubleClick", String(Math.round(target.x)), String(Math.round(target.y))]);
      return uiInput(["click", String(Math.round(target.x)), String(Math.round(target.y))]);
    },
    right_click: async () => { throw new ExecError("uitest uiInput has no right-click; use longClick semantics via hold or left_click"); },
    middle_click: async () => { throw new ExecError("middle click is not exposed by uitest uiInput"); },
    mouse_move: async () => ({ action_sent: false, note: "hover without press is not exposed by uitest uiInput" }),
    left_click_drag: ({ from_target: from, to }) =>
      uiInput(["swipe", String(Math.round(from.x)), String(Math.round(from.y)), String(Math.round(to.x)), String(Math.round(to.y)), "200"], { timeoutMs: 30_000 }),
    left_mouse_down: async () => { throw new ExecError("low-level press/release is not exposed by uitest uiInput; use left_click_drag"); },
    left_mouse_up: async () => { throw new ExecError("low-level press/release is not exposed by uitest uiInput; use left_click_drag"); },
    scroll: ({ target, direction = "down", amount = 300 }) => {
      const dist = Math.max(60, Math.min(1200, amount * 24));
      const dx = direction === "left" ? dist : direction === "right" ? -dist : 0;
      const dy = direction === "up" ? dist : direction === "down" ? -dist : 0;
      return uiInput(["swipe", String(Math.round(target.x)), String(Math.round(target.y)), String(Math.round(target.x + dx)), String(Math.round(target.y + dy)), "400"]);
    },
    type: async ({ text }) => {
      if (!text) return { action_sent: false, note: "empty text" };
      await uiInput(["inputText", "300", "300", escDeviceText(text)]).catch(async (e) => {
        // Some builds require coordinates of the focused field; retry with a click-first pattern.
        throw e;
      });
      return { action_sent: true, chars: text.length, note: "inputText at 300,300 — click the field first for focused input" };

View on GitHub (pinned to 73e0f67d83)