{"record":{"id":"e87e10bf1d34659c","repo":"Hmbown/CodeWhale","slug":"too-many-active-engine-pet-spans-e87e10","errorCode":null,"errorMessage":"Too many active Engine pet spans.","messagePattern":"Too many active Engine pet spans\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"pet/src/core/pet-engine.ts","lineNumber":89,"sourceCode":"\n  observe(value: unknown, at: number): void {\n    if (!Number.isFinite(at) || at < this.lastTime || at > PET_MAX_SECONDS * 1000) throw new Error('Invalid Engine pet clock.');\n    if (!value || typeof value !== 'object' || Array.isArray(value)) throw new Error('Invalid Engine pet metadata.');\n    const e = value as Record<string, unknown>;\n    const allowed = ['event', 'index', 'channel', 'tool_call_id', 'tool_name', 'id', 'worker_status', 'failed'];\n    if (Object.keys(e).some(k => !allowed.includes(k)) || typeof e.event !== 'string'\n      || Object.values(e).some(v => typeof v === 'string' && v.length > 4096)\n      || e.channel !== undefined && !['text', 'reasoning'].includes(e.channel as string)\n      || ['tool_call_id', 'tool_name', 'id', 'worker_status'].some(k => e[k] !== undefined && typeof e[k] !== 'string')\n      || e.failed !== undefined && typeof e.failed !== 'boolean'\n      || e.index !== undefined && (!Number.isSafeInteger(e.index) || (e.index as number) < 0))\n      throw new Error('Invalid Engine pet metadata fields.');\n    this.lastTime = at;\n    this.events = this.events.filter(span => span.endTime >= at - 12_800);\n    const id = (field: string) => { const s = e[field]; if (typeof s !== 'string' || !s) throw new Error(`Missing Engine ${field}.`); return s; };\n    const index = () => { if (!Number.isSafeInteger(e.index)) throw new Error('Missing Engine index.'); return String(e.index); };\n    const start = (key: string, name: string, category: Category, agentId?: string) => {\n      if (this.active.size >= 256 && !this.active.has(key)) throw new Error('Too many active Engine pet spans.');\n      this.active.set(key, this.add(name, category, at, agentId));\n    };\n    const finish = (key: string) => { this.pulse(key, at); this.active.delete(key); };\n    switch (e.event) {\n      case 'turn_started': this.active.clear(); this.waiting = undefined; break;\n      case 'message_started': start(`message:${index()}`, 'assistant_message', 'communication'); this.waiting = undefined; break;\n      case 'thinking_started': start(`thinking:${index()}`, 'thinking', 'reasoning'); this.waiting = undefined; break;\n      case 'response_delta': {\n        const reasoning = e.channel === 'reasoning';\n        const key = `${reasoning ? 'thinking' : 'message'}:${index()}`;\n        if (!this.active.has(key)) start(key, reasoning ? 'thinking' : 'assistant_message', reasoning ? 'reasoning' : 'communication');\n        else this.pulse(key, at);\n        this.waiting = undefined; break;\n      }\n      case 'message_complete': finish(`message:${index()}`); break;\n      case 'thinking_complete': finish(`thinking:${index()}`); break;\n      case 'tool_call_started': start(`tool:${id('tool_call_id')}`, id('tool_name'), toolCategory(id('tool_name'))); this.waiting = undefined; break;\n      case 'tool_call_heartbeat': for (const key of this.active.keys()) if (key.startsWith('tool:')) this.pulse(key, at); break;","sourceCodeStart":71,"sourceCodeEnd":107,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/433685b2024e7bc4c99e1e2e326bcad39b4d9d65/pet/src/core/pet-engine.ts#L71-L107","documentation":"The internal `start()` helper opens a new pet span unless 256 spans are already active. This error means the engine has hit its concurrency cap of 256 live spans and the incoming event would start span number 257 with a key that is not already active. It protects memory from runaway/unpaired span starts.","triggerScenarios":"Emitting many `*_started` events (messages, tools, turns) whose matching finish events never arrive, so spans accumulate in `this.active`; a high-volume loop that opens more than 256 distinct keyed spans before closing any.","commonSituations":"Unpaired start/finish pairs after an error path skipped the finish call; a `turn_started` clear() never being observed between turns; parallel workers each opening tool/message spans beyond 256 concurrently; a long-running session where spans leak one per turn.","solutions":["Ensure every started span gets a matching finish event so it is removed from the active map.","Emit a `turn_started` event between turns — the engine clears `this.active` on it, releasing leaked spans.","Audit for duplicated span keys (same id reused) that keep old spans alive; keys must be unique per open span.","Back-pressure: batch or defer events when the number of concurrently open spans approaches 256.","On this error, treat it as a leak signal — log `this.active.size` and the keys still open to find the unpaired starts."],"exampleFix":"// before\nfor (const t of manyToolCalls) start(t); // never finished, > 256 open\n\n// after\nfor (const t of manyToolCalls) {\n  start(t);\n  finish(t); // pair every start with a finish\n}","handlingStrategy":"try-catch","validationCode":"// no pre-call validation possible: cap depends on engine's internal active-span map\n// guard at the application level by tracking how many spans you have open:\nif (openSpanCount >= 250) await drainPendingFinishes(); // close spans before opening more","typeGuard":null,"tryCatchPattern":"try {\n  engine.observe(event, at);\n} catch (err) {\n  if (err instanceof Error && err.message === 'Too many active Engine pet spans.') {\n    console.error('span leak: >256 open spans; emitting turn_started to reset');\n    engine.observe({ event: 'turn_started' }, at); // clears active spans\n    engine.observe(event, at); // retry once\n    return;\n  }\n  throw err;\n}","preventionTips":["Always pair every start event with its finish event, including on error paths (use try/finally).","Emit turn_started between turns — it clears the active span map.","Track open-span count in your own instrumentation and alert before reaching 256.","Audit for duplicate span keys that keep stale spans alive in the active map."],"tags":["resource-limit","leak","spans","telemetry"],"backgroundTag":"value-out-of-range","analyzedSha":"433685b2024e7bc4c99e1e2e326bcad39b4d9d65","analyzedAt":"2026-09-15T12:24:24.634Z","contentChangedAt":"2026-09-15T12:24:24.634Z","schemaVersion":2},"datasetVersion":"2026-09-22T16:17:23.217Z"}