{"record":{"id":"c36b2086d91d7cbe","repo":"Hmbown/CodeWhale","slug":"zoom-is-not-supported-on-the-harmony-backend-yet-screenshot","errorCode":null,"errorMessage":"zoom is not supported on the harmony backend yet — screenshot + region on the host is the workaround","messagePattern":"zoom is not supported on the harmony backend yet — screenshot \\+ region on the host is the workaround","errorType":"exception","errorClass":"ExecError","httpStatus":null,"severity":"error","filePath":"crates/tui/plugins/computer-use/src/backends/harmonyos.mjs","lineNumber":216,"sourceCode":"    get_app_state: async (args = {}) => {\n      rejectAppSelectors(args);\n      const tree = await dumpLayout();\n      const els = flatten(tree);\n      return { bundle_id: tree.attributes?.bundleName ?? null, elements: els, truncated: els.length >= 600 };\n    },\n    screenshot: async (args = {}) => {\n      rejectAppSelectors(args);\n      const { path: outPath } = args;\n      const dir = process.env.CODEWHALE_CU_RECORDINGS_DIR || path.join(os.homedir(), \".codewhale-cu\", \"recordings\");\n      fs.mkdirSync(dir, { recursive: true });\n      const file = outPath || path.join(dir, `shot-${new Date().toISOString().replace(/[:.]/g, \"-\")}-${crypto.randomBytes(3).toString(\"hex\")}.jpeg`);\n      await snapshot(file);\n      const buf = fs.readFileSync(file);\n      displayPixels = jpegSize(buf) ?? displayPixels;\n      return { file, bytes: buf.length, pixels: jpegSize(buf), scale: 1, points: jpegSize(buf) };\n    },\n    zoom: async ({ region, path: outPath }) => {\n      throw new ExecError(\"zoom is not supported on the harmony backend yet — screenshot + region on the host is the workaround\");\n    },\n    left_click: ({ target, strategy }) => { assertEventStrategy(strategy); return uiInput([\"click\", String(Math.round(target.x)), String(Math.round(target.y))]); },\n    double_click: ({ target }) => uiInput([\"doubleClick\", String(Math.round(target.x)), String(Math.round(target.y))]),\n    triple_click: async ({ target }) => {\n      await uiInput([\"doubleClick\", String(Math.round(target.x)), String(Math.round(target.y))]);\n      return uiInput([\"click\", String(Math.round(target.x)), String(Math.round(target.y))]);\n    },\n    right_click: async () => { throw new ExecError(\"uitest uiInput has no right-click; use longClick semantics via hold or left_click\"); },\n    middle_click: async () => { throw new ExecError(\"middle click is not exposed by uitest uiInput\"); },\n    mouse_move: async () => ({ action_sent: false, note: \"hover without press is not exposed by uitest uiInput\" }),\n    left_click_drag: ({ from_target: from, to }) =>\n      uiInput([\"swipe\", String(Math.round(from.x)), String(Math.round(from.y)), String(Math.round(to.x)), String(Math.round(to.y)), \"200\"], { timeoutMs: 30_000 }),\n    left_mouse_down: async () => { throw new ExecError(\"low-level press/release is not exposed by uitest uiInput; use left_click_drag\"); },\n    left_mouse_up: async () => { throw new ExecError(\"low-level press/release is not exposed by uitest uiInput; use left_click_drag\"); },\n    scroll: ({ target, direction = \"down\", amount = 300 }) => {\n      const dist = Math.max(60, Math.min(1200, amount * 24));\n      const dx = direction === \"left\" ? dist : direction === \"right\" ? -dist : 0;\n      const dy = direction === \"up\" ? dist : direction === \"down\" ? -dist : 0;","sourceCodeStart":198,"sourceCodeEnd":234,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/73e0f67d83c59909b571efdfc88c4bc28c309cb1/crates/tui/plugins/computer-use/src/backends/harmonyos.mjs#L198-L234","documentation":"The harmony (HarmonyOS) backend of the computer-use plugin does not implement the zoom action. `create()` returns a capabilities object whose `zoom` method unconditionally throws this ExecError. It is a deliberate fail-fast stub: the backend drives the device over hdc/uitest and offers no zoomed-screenshot primitive, so the library throws instead of silently returning a scaled image.","triggerScenarios":"Calling the zoom action (e.g. { action: \"zoom\", region, path }) against a backend created by create() in crates/tui/plugins/computer-use/src/backends/harmonyos.mjs when the connected device is HarmonyOS. The throw is unconditional — no region or path value avoids it.","commonSituations":"An automation flow that works against the desktop backend is pointed at a HarmonyOS device and issues zoom to read small text or fine UI detail; a generic agent tool-loop enumerates all actions including zoom on a touch device; porting a computer-use script from macOS/Windows hosts to a HarmonyOS phone.","solutions":["Take a normal screenshot with the screenshot action, then crop the saved JPEG to the region on the host (e.g. with sharp, jimp, or ffmpeg) — this is exactly the workaround the error message names.","If higher effective resolution is needed, compute the crop coordinates in the screenshot's pixel space (the returned `pixels`/`scale` fields) rather than zooming.","Guard the call: check backend capabilities (or skip zoom for harmonyos) before dispatching the action, and fall back to the screenshot+crop path.","If zoom support is genuinely needed, implement it in the harmony backend as screenshot + host-side crop following the same stub signature."],"exampleFix":"// before\nawait backend.zoom({ region: { x: 100, y: 100, width: 200, height: 200 }, path: \"zoom.jpeg\" });\n\n// after\nconst shot = await backend.screenshot({ path: \"full.jpeg\" });\n// crop `region` out of full.jpeg on the host (sharp example)\nawait sharp(\"full.jpeg\")\n  .extract({ left: 100, top: 100, width: 200, height: 200 })\n  .toFile(\"zoom.jpeg\");","handlingStrategy":"fallback","validationCode":"const ZOOM_SUPPORTED = (backend) => backend.kind !== \"harmonyos\";\nif (ZOOM_SUPPORTED(backend)) {\n  await backend.zoom({ region, path });\n} else {\n  const shot = await backend.screenshot({});\n  await cropOnHost(shot.file, region, path);\n}","typeGuard":"const supportsZoom = (backend) => typeof backend.zoom === \"function\" && backend.capabilities?.zoom !== false;","tryCatchPattern":"try {\n  await backend.zoom({ region, path });\n} catch (e) {\n  if (/zoom is not supported on the harmony backend/.test(e.message)) {\n    const shot = await backend.screenshot({});\n    await cropOnHost(shot.file, region, path);\n  } else {\n    throw e;\n  }\n}","preventionTips":["Maintain a per-backend capability table and consult it before dispatching any action.","Write computer-use plans against a touch-safe action subset (screenshot/click/longClick/swipe) when targeting HarmonyOS.","Test cross-backend scripts with each backend before release; unsupported actions throw, not no-op.","Centralize host-side image cropping as a shared helper so screenshot+crop is a one-line fallback."],"tags":["harmonyos","zoom","unsupported-operation","screenshot","computer-use"],"backgroundTag":"unsupported-operation","analyzedSha":"73e0f67d83c59909b571efdfc88c4bc28c309cb1","analyzedAt":"2026-09-22T01:30:00.501Z","contentChangedAt":"2026-09-22T01:30:00.501Z","schemaVersion":2},"datasetVersion":"2026-09-22T21:17:16.096Z"}