{"record":{"id":"c5b1ded48ff02e57","repo":"Hmbown/CodeWhale","slug":"windows-semantic-actions-require-a-fresh-observed-element","errorCode":null,"errorMessage":"Windows semantic actions require a fresh observed element with window and element runtime identities","messagePattern":"Windows semantic actions require a fresh observed element with window and element runtime identities","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/tui/plugins/computer-use/src/backends/win32.mjs","lineNumber":131,"sourceCode":"/** Coordinate clicks on this backend are always raw pointer events; strategy=\"a11y\" must fail closed rather than silently degrade. */\nfunction assertEventStrategy(strategy) {\n  if (strategy != null && strategy !== \"auto\" && strategy !== \"event\") {\n    throw new ExecError(`strategy \"${strategy}\" is macOS-only; this backend dispatches coordinate clicks as raw pointer events — use an element target for a semantic action`);\n  }\n}\n\nfunction unsupportedSelector(message) {\n  return Object.assign(new ExecError(message), { code: \"unsupported_selector\" });\n}\n\n// A cached child path alone is unsafe: sibling/window order can change.\n// Runtime IDs bind both the root and leaf to the observation that supplied them.\nfunction elementScript(target) {\n  const ids = [target?.runtime_id, target?.window_runtime_id];\n  if (ids.some(id => !Array.isArray(id) || !id.length || !id.every(Number.isInteger))\n      || !Array.isArray(target?.path) || target.path[0] !== 0 || !target.path.every(i => Number.isInteger(i) && i >= 0)\n      || (target.windowIndex != null && target.windowIndex !== 0) || Object.hasOwn(target, \"window_id\")) {\n    throw unsupportedSelector(\"Windows semantic actions require a fresh observed element with window and element runtime identities\");\n  }\n  const encoded = Buffer.from(JSON.stringify(target), \"utf16le\").toString(\"base64\");\n  return `${UIA_PRELUDE}\n$target = [Text.Encoding]::Unicode.GetString([Convert]::FromBase64String('${encoded}')) | ConvertFrom-Json;\n$root = [System.Windows.Automation.AutomationElement]::RootElement;\n$windows = @($root.FindAll([System.Windows.Automation.TreeScope]::Children, [System.Windows.Automation.Condition]::TrueCondition));\n$matches = @($windows | Where-Object { ($_.GetRuntimeId() -join ',') -ceq ($target.window_runtime_id -join ',') });\nif ($matches.Count -ne 1) { throw 'element_stale: observed window no longer exists' }\n$cur = $matches[0];\nif ($target.app_ref.name -and $cur.Current.Name -cne $target.app_ref.name) { throw 'element_stale: window identity changed' }\nif ($target.app_ref.pid -and $cur.Current.ProcessId -ne $target.app_ref.pid) { throw 'element_stale: window process changed' }\nfor ($step = 1; $step -lt $target.path.Count; $step++) {\n  $kids = $cur.FindAll([System.Windows.Automation.TreeScope]::Children, [System.Windows.Automation.Condition]::TrueCondition);\n  $i = $target.path[$step];\n  if ($i -ge $kids.Count) { throw 'element_stale: observed path changed' }\n  $cur = $kids[$i];\n}\nif (($cur.GetRuntimeId() -join ',') -cne ($target.runtime_id -join ',')) { throw 'element_stale: observed element was replaced' }","sourceCodeStart":113,"sourceCodeEnd":149,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/73e0f67d83c59909b571efdfc88c4bc28c309cb1/crates/tui/plugins/computer-use/src/backends/win32.mjs#L113-L149","documentation":"On Windows, win32.mjs's UI Automation element actions can only re-resolve an element that was freshly observed: the selector must carry both the element's runtime_id and the owning window's window_runtime_id (non-empty integer arrays), a `path` array starting at 0 of non-negative integers, and must not carry a stale window handle (`window_id`) or a non-zero windowIndex. elementScript validates these invariants and throws unsupportedSelector with this message when the selector lacks or contradicts the observation identities, because the generated PowerShell would otherwise target the wrong or a nonexistent element.","triggerScenarios":"Calling a Windows semantic action (click/type/etc.) with an element whose selector came from a stale snapshot (window or element rebuilt since observation, so runtime ids changed), a hand-built selector missing runtime_id/window_runtime_id, a selector carrying a legacy `window_id` field or a nonzero windowIndex, or a `path` that is empty, doesn't start with 0, or contains non-integer/negative indices.","commonSituations":"A UI automation script caches element references across a window repaint or app restart, invalidating runtime ids; a developer reuses a macOS/Linux-style selector shape on Windows where the platform requires runtime identities; serialization round-trips drop or rename the runtime_id arrays; an older selector format with window_id is fed to the new backend.","solutions":["Re-observe the target element immediately before the action so the selector carries fresh runtime_id and window_runtime_id arrays.","Strip any `window_id` field and ensure windowIndex is 0 or absent from the selector passed to Windows semantic actions.","Validate the `path` array: it must start with 0 and contain only non-negative integers; fix the observation code producing it if not.","If selectors cross a process/serialization boundary, verify runtime_id and window_runtime_id survive as integer arrays (not strings or dropped fields)."],"exampleFix":"// before\nawait click({ path: [0, 3], window_id: 0x1a0b2c }); // stale handle-based selector\n// throws: Windows semantic actions require a fresh observed element...\n// after\nconst observed = await observe(); // fresh observation\nawait click({\n  runtime_id: observed.runtime_id,           // e.g. [42, 123456]\n  window_runtime_id: observed.window_runtime_id,\n  path: observed.path,                        // starts at 0, integers\n  windowIndex: 0\n});","handlingStrategy":"type-guard","validationCode":"// Before invoking any Windows semantic action:\nfunction validWinSelector(t) {\n  const idsOk = (a) => Array.isArray(a) && a.length > 0 && a.every(Number.isInteger);\n  return !!t\n    && idsOk(t.runtime_id)\n    && idsOk(t.window_runtime_id)\n    && Array.isArray(t.path) && t.path[0] === 0\n    && t.path.every((i) => Number.isInteger(i) && i >= 0)\n    && (t.windowIndex == null || t.windowIndex === 0)\n    && !Object.hasOwn(t, \"window_id\");\n}","typeGuard":"function isFreshObservedElement(t) {\n  return validWinSelector(t); // same predicate as above\n}","tryCatchPattern":"try {\n  await click(target);\n} catch (err) {\n  if (String(err.message).includes('fresh observed element')) {\n    const fresh = await observe(targetHints); // re-observe to get new runtime ids\n    await click(fresh);\n  } else throw err;\n}","preventionTips":["Never cache element selectors across UI changes on Windows; re-observe before each action.","Do not hand-construct selectors — always take them from the observation API.","Strip legacy window_id fields when porting selectors from other platforms.","When serializing selectors between processes, assert runtime_id/window_runtime_id survive as integer arrays.","Validate selectors with a shared predicate before any semantic action call."],"tags":["windows","ui-automation","selector","validation"],"backgroundTag":"invalid-argument-value","analyzedSha":"73e0f67d83c59909b571efdfc88c4bc28c309cb1","analyzedAt":"2026-09-22T01:30:00.501Z","contentChangedAt":"2026-09-22T01:30:00.501Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}