{"record":{"id":"73ac4bc89f90e7d3","repo":"aaif-goose/goose","slug":"failed-to-create-token-counter-error","errorCode":null,"errorMessage":"Failed to create token counter: {error}","messagePattern":"Failed to create token counter: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/goose/src/context_mgmt/mod.rs","lineNumber":214,"sourceCode":"        Err(error) => {\n            warn!(\"Failed to count retained context tokens, using billable output tokens: {error}\");\n            summarization_usage.usage.output_tokens.unwrap_or(0)\n        }\n    };\n\n    Ok(CompactionResult {\n        conversation,\n        usage: summarization_usage,\n        retained_context_tokens,\n    })\n}\n\n/// Estimate the tokens of the agent-visible conversation, counted the same way\n/// as the fallback estimation in `check_if_compaction_needed`.\npub(crate) async fn count_context_tokens(conversation: &Conversation) -> Result<i32> {\n    let counter = create_token_counter()\n        .await\n        .map_err(|error| anyhow::anyhow!(\"Failed to create token counter: {error}\"))?;\n    let total: usize = conversation\n        .messages()\n        .iter()\n        .filter(|message| message.is_agent_visible())\n        .map(|message| counter.count_chat_tokens(\"\", std::slice::from_ref(message), &[]))\n        .sum();\n    Ok(total.try_into()?)\n}\n\n/// Check if messages exceed the auto-compaction threshold\npub async fn check_if_compaction_needed(\n    provider: &dyn Provider,\n    conversation: &Conversation,\n    threshold_override: Option<f64>,\n    session: &crate::session::Session,\n) -> Result<bool> {\n    if provider.manages_own_context() {\n        return Ok(false);","sourceCodeStart":196,"sourceCodeEnd":232,"githubUrl":"https://github.com/aaif-goose/goose/blob/3810898a7447ec3299be72e223d3570a7aabf0ab/crates/goose/src/context_mgmt/mod.rs#L196-L232","documentation":"count_context_tokens estimates the token count of agent-visible messages (used to report retained context around compaction) by first building a token counter with create_token_counter. Any failure in counter construction is re-wrapped with this message, so the {error} suffix carries the real cause (e.g. tokenizer/model initialization failure).","triggerScenarios":"Calling count_context_tokens when create_token_counter() fails — typically tokenizer assets cannot be loaded (offline first run, missing cache) or the underlying model/tokenizer init errors out.","commonSituations":"Air-gapped or proxied environments where tokenizer files cannot be fetched; corrupted token-counter cache; calling the context-token helper before any provider has warmed up the tokenizer.","solutions":["Read the wrapped {error} detail — it identifies the exact initialization failure","Fix the underlying cause: ensure network access to tokenizer assets on first use, or clear/repair the token-counter cache","Where possible rely on session.usage.total_tokens (session metadata) instead of estimating, as check_if_compaction_needed already does when usage is present"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// Warm the token counter once at startup and surface its error early:\ncreate_token_counter().await?;\n// after this, count_context_tokens cannot fail on counter creation","typeGuard":null,"tryCatchPattern":"match count_context_tokens(&conversation).await {\n    Err(e) if e.to_string().contains(\"Failed to create token counter\") => {\n        // fall back to a rough estimate (chars/4) or skip reporting; do not abort the session\n    }\n    other => other?,\n}","preventionTips":["Initialize the token counter during provider warm-up, not lazily mid-session","Ensure first-run network access for tokenizer assets or pre-seed the cache","Prefer session.usage.total_tokens when the provider reports usage"],"tags":["context-management","tokens","compaction"],"backgroundTag":null,"analyzedSha":"3810898a7447ec3299be72e223d3570a7aabf0ab","analyzedAt":"2026-08-16T10:14:26.282Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}