Hmbown/CodeWhale · error · ServerError

unknown_element

unknown_element

Error message

element index ${target.index} is outside state ${stateId} (0..${st.elements.length - 1})

What it means

After resolving the state, resolveElement indexes st.elements[target.index] and throws code "unknown_element" when that slot is empty — the index is outside the element list of the referenced state (valid range 0..length-1). The message includes both the state_id and the valid range so the caller can correct the index against the observation it holds.

Solutions

1. Clamp/validate the index against the observation you hold: 0 <= index < elements.length before sending.
2. Re-run get_app_state and re-resolve the element by its role/label in the fresh observation; indexes shift when UI changes.
3. If your list is 1-based, subtract 1 before sending.
4. Pair the index with the state_id it actually came from — never combine an index and state_id from different observations.

Example fix

// before
const idx = Number(uiRow); // 1-based from a rendered list
await act({ type: "element", state_id: st.state_id, index: idx });

// after
const idx = Number(uiRow) - 1;
if (idx < 0 || idx >= st.elements.length) throw new Error(`row ${uiRow} not in observation`);
await act({ type: "element", state_id: st.state_id, index: idx });

if (!Number.isInteger(index) || index < 0 || index >= observation.elements.length) throw new Error(`element index ${index} out of range 0..${observation.elements.length - 1}`);

const isValidIndex = (obs, i) => obs && Array.isArray(obs.elements) && Number.isInteger(i) && i >= 0 && i < obs.elements.length;

try {
  return await act({ type: "element", state_id, index });
} catch (e) {
  if (e.code === "unknown_element") {
    const fresh = await getAppState(computerId);
    return await act({ type: "element", state_id: fresh.state_id, index: relocate(fresh, index) });
  }
  throw e;
}

When it happens

Trigger: target.type === "element" with an index >= st.elements.length or < 0 (negative indexes hit holes, not the last element). Typical when an index from an older observation is reused against a different/smaller state, or the index is 1-based from a human-facing list pasted into a 0-based API.

Common situations: Mixing index from state A with state_id from state B; off-by-one after the UI shrank between turns; copy-pasting element numbers from a rendered accessibility tree that numbers from 1; loops that run past the end of the elements array.

Related errors

• app_denied
• computer_observation_required
• consent_required
• control_stopped
• foreground_denied

AI-assisted analysis of Hmbown/CodeWhale@73e0f67d83 (2026-09-22). Data as JSON: /api/errors/787c6ce4826ad529. Report an issue: GitHub.

Appendix: source

Thrown at crates/tui/plugins/computer-use/mcp/server.mjs:165

async function getBackend(computer, binding) {
  if (!binding.backend) binding.backend = (await backendFor(computer)).backend;
  return binding.backend;
}

/**
 * Element target -> enriched target with cached app identity and AX path.
 * An explicit state_id pins a specific observation; a bare index addresses
 * the latest observation on this computer — the flat addressing a caller
 * uses when it acts on what it just saw.
 */
function resolveElement(target, computer) {
  const stateId = target.state_id ?? latestStateByComputer.get(computer.id);
  const st = stateId ? appStates.get(stateId) : null;
  if (!st) throw new ServerError("unknown_state", target.state_id
    ? `state_id "${target.state_id}" is unknown or expired — call get_app_state again`
    : "no observation on this computer yet — call get_app_state first");
  const el = st.elements[target.index];
  if (!el) throw new ServerError("unknown_element", `element index ${target.index} is outside state ${stateId} (0..${st.elements.length - 1})`);
  return { state: st, element: el, stateId };
}

class ServerError extends Error {
  constructor(code, message, extra = null) { super(message); this.code = code; if (extra) this.extra = extra; }
}

/** Map raster-pixel coordinates to screen points using the bound raster. */
function rasterToPoints(computerId, x, y) {
  const r = lastRasters.get(computerId);
  if (!r) throw new ServerError("no_raster", "no screenshot bound on this computer yet — call screenshot first so pixel targets have a frame");
  if (r.pixels?.w != null && r.pixels?.h != null && (x < 0 || y < 0 || x >= r.pixels.w || y >= r.pixels.h)) {
    throw new ServerError("target_outside_raster", `target (${x},${y}) is outside the bound raster (${r.pixels.w}x${r.pixels.h} pixels) — take a fresh screenshot`);
  }
  const scale = r.scale && r.scale > 0 ? r.scale : 1;
  return { x: (r.origin?.x ?? 0) + x / scale, y: (r.origin?.y ?? 0) + y / scale };
}

View on GitHub (pinned to 73e0f67d83)