{"record":{"id":"46f95f7ba2f6e73a","repo":"aaif-goose/goose","slug":"failed-to-create-token-counter","errorCode":null,"errorMessage":"Failed to create token counter: {}","messagePattern":"Failed to create token counter: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/goose/src/context_mgmt/mod.rs","lineNumber":257,"sourceCode":"            .get_param::<f64>(\"GOOSE_AUTO_COMPACT_THRESHOLD\")\n            .unwrap_or(DEFAULT_COMPACTION_THRESHOLD)\n    });\n\n    let model_config = session\n        .model_config\n        .clone()\n        .unwrap_or_else(|| ModelConfig::new(\"unknown\"));\n    let context_limit = provider\n        .get_context_limit(&model_config)\n        .await\n        .unwrap_or_else(|_| model_config.context_limit());\n\n    let (current_tokens, _token_source) = match session.usage.total_tokens {\n        Some(tokens) => (tokens as usize, \"session metadata\"),\n        None => {\n            let token_counter = create_token_counter()\n                .await\n                .map_err(|e| anyhow::anyhow!(\"Failed to create token counter: {}\", e))?;\n\n            let token_counts: Vec<_> = messages\n                .iter()\n                .filter(|m| m.is_agent_visible())\n                .map(|msg| token_counter.count_chat_tokens(\"\", std::slice::from_ref(msg), &[]))\n                .collect();\n\n            (token_counts.iter().sum(), \"estimated\")\n        }\n    };\n\n    let usage_ratio = current_tokens as f64 / context_limit as f64;\n\n    let needs_compaction = if threshold <= 0.0 || threshold >= 1.0 {\n        false // Auto-compact is disabled.\n    } else {\n        usage_ratio > threshold\n    };","sourceCodeStart":239,"sourceCodeEnd":275,"githubUrl":"https://github.com/aaif-goose/goose/blob/3810898a7447ec3299be72e223d3570a7aabf0ab/crates/goose/src/context_mgmt/mod.rs#L239-L275","documentation":"check_if_compaction_needed prefers session.usage.total_tokens (source 'session metadata') and only falls back to local estimation when usage is None. In that fallback it must build a token counter via create_token_counter; failure of that construction is re-wrapped with this message, preserving the underlying error text.","triggerScenarios":"A session whose metadata lacks total_tokens (older sessions, providers that do not report usage) on a machine where create_token_counter fails — e.g. tokenizer download/init blocked by network or a corrupted cache.","commonSituations":"Offline environments; sessions created by older goose versions without usage tracking; provider outages that also prevent tokenizer fetches.","solutions":["Inspect the wrapped error text for the concrete initialization failure","Restore tokenizer availability (network/cache) so estimation can run","Use sessions that carry usage metadata (total_tokens), which skips counter creation entirely"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"// Skip estimation when trustworthy usage metadata exists:\nif session.usage.total_tokens.is_none() {\n    create_token_counter().await?; // fail fast here, not inside the compaction check\n}\ncheck_if_compaction_needed(provider, &model_config, &session, &messages).await?;","typeGuard":null,"tryCatchPattern":"match check_if_compaction_needed(...).await {\n    Err(e) if e.to_string().contains(\"Failed to create token counter\") => {\n        // treat compaction as not-yet-needed and retry after the counter issue is fixed,\n        // or compute a conservative estimate from message lengths\n    }\n    other => other?,\n}","preventionTips":["Keep sessions carrying usage metadata (total_tokens) so estimation is never needed","Verify tokenizer availability (network/cache) before long sessions","Handle this error non-fatally so auto-compaction failure does not kill the agent loop"],"tags":["context-management","tokens","compaction"],"backgroundTag":null,"analyzedSha":"3810898a7447ec3299be72e223d3570a7aabf0ab","analyzedAt":"2026-08-16T10:14:26.282Z","schemaVersion":2},"datasetVersion":"2026-08-16T13:17:31.715Z"}