{"record":{"id":"d1b577bf5f5ce7ae","repo":"Hmbown/CodeWhale","slug":"too-many-active-engine-pet-spans","errorCode":null,"errorMessage":"Too many active Engine pet spans.","messagePattern":"Too many active Engine pet spans\\.","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"crates/tui/src/tui/pet_watch/pet-native.js","lineNumber":1749,"sourceCode":"            throw new Error('Invalid Engine pet metadata.');\n        const e = value;\n        const allowed = ['event', 'index', 'channel', 'tool_call_id', 'tool_name', 'id', 'worker_status', 'failed'];\n        if (Object.keys(e).some(k => !allowed.includes(k)) || typeof e.event !== 'string'\n            || Object.values(e).some(v => typeof v === 'string' && v.length > 4096)\n            || e.channel !== undefined && !['text', 'reasoning'].includes(e.channel)\n            || ['tool_call_id', 'tool_name', 'id', 'worker_status'].some(k => e[k] !== undefined && typeof e[k] !== 'string')\n            || e.failed !== undefined && typeof e.failed !== 'boolean'\n            || e.index !== undefined && (!Number.isSafeInteger(e.index) || e.index < 0))\n            throw new Error('Invalid Engine pet metadata fields.');\n        this.lastTime = at;\n        this.events = this.events.filter(span => span.endTime >= at - 12_800);\n        const id = (field) => { const s = e[field]; if (typeof s !== 'string' || !s)\n            throw new Error(`Missing Engine ${field}.`); return s; };\n        const index = () => { if (!Number.isSafeInteger(e.index))\n            throw new Error('Missing Engine index.'); return String(e.index); };\n        const start = (key, name, category, agentId) => {\n            if (this.active.size >= 256 && !this.active.has(key))\n                throw new Error('Too many active Engine pet spans.');\n            this.active.set(key, this.add(name, category, at, agentId));\n        };\n        const finish = (key) => { this.pulse(key, at); this.active.delete(key); };\n        switch (e.event) {\n            case 'turn_started':\n                this.active.clear();\n                this.waiting = undefined;\n                break;\n            case 'message_started':\n                start(`message:${index()}`, 'assistant_message', 'communication');\n                this.waiting = undefined;\n                break;\n            case 'thinking_started':\n                start(`thinking:${index()}`, 'thinking', 'reasoning');\n                this.waiting = undefined;\n                break;\n            case 'response_delta': {\n                const reasoning = e.channel === 'reasoning';","sourceCodeStart":1731,"sourceCodeEnd":1767,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/73e0f67d83c59909b571efdfc88c4bc28c309cb1/crates/tui/src/tui/pet_watch/pet-native.js#L1731-L1767","documentation":"The start() helper in observe enforces a concurrency bound of 256 simultaneously active spans (tool:, thinking:, message:, agent: keys). It throws 'Too many active Engine pet spans.' when a new span key would exceed that cap and the key is not already active. This bounds memory and keeps the activity view meaningful.","triggerScenarios":"More than 256 tool calls or agent spawns without matching completion events; tool_call_started storms without tool_call_complete; agent_spawned events for a fleet larger than the cap; leaked keys from missing *_complete events accumulating across a turn.","commonSituations":"Parallel subagent fan-out beyond 256 live workers; a dropped/lost completion event (crash, cancel path) leaving keys in active forever; a replay that spawns many keys but whose complete events arrive at timestamps already pruned/rejected.","solutions":["Ensure every started span has a matching completion event (tool_call_complete, agent_complete, *_complete)","Clear stale spans on turn boundaries — turn_started/turn_complete already call active.clear(); route through turn events","Cap actual parallelism below 256 or coalesce per-agent spans","On replay, feed turn_started first so prior leaked keys are cleared"],"exampleFix":"// before\nfor (const c of manyCalls) telemetry.observe({ event: 'tool_call_started', ... }, at); // 300 starts, no completes\n// after\ntelemetry.observe({ event: 'turn_started' }, at);\nfor (const c of manyCalls.slice(0, 256)) {\n  telemetry.observe({ event: 'tool_call_started', tool_call_id: c.id, tool_name: c.name }, at);\n  telemetry.observe({ event: 'tool_call_complete', tool_call_id: c.id }, at);\n}","handlingStrategy":"validation","validationCode":"if (telemetry.active.size >= 256 && !telemetry.active.has(key)) {\n  throw new Error('Too many active Engine pet spans.');\n}\ntelemetry.observe({ event: 'tool_call_started', tool_call_id: id, tool_name: name }, at);","typeGuard":null,"tryCatchPattern":"try {\n  telemetry.observe(meta, at);\n} catch (err) {\n  if (err.message === 'Too many active Engine pet spans.') {\n    telemetry.active.clear(); // or emit synthetic completes for leaked keys, then retry\n    telemetry.observe(meta, at);\n  } else throw err;\n}","preventionTips":["Guarantee every *_started event has a matching *_complete","Feed turn_started/turn_complete so active spans are cleared per turn","Bound actual parallel tool/agent fan-out below 256","Detect leaked keys (spans active across turn boundaries) in tests"],"tags":["telemetry","resource-limit","concurrency"],"backgroundTag":"resource-limit-exceeded","analyzedSha":"73e0f67d83c59909b571efdfc88c4bc28c309cb1","analyzedAt":"2026-09-22T01:30:00.501Z","contentChangedAt":"2026-09-22T01:30:00.501Z","schemaVersion":2},"datasetVersion":"2026-09-22T16:17:23.217Z"}