{"record":{"id":"44b3f7e6e9cbc9af","repo":"sigoden/aichat","slug":"invalid-github-repo-tree","errorCode":null,"errorMessage":"Invalid github repo tree","messagePattern":"Invalid github repo tree","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"src/utils/request.rs","lineNumber":348,"sourceCode":"\n    let sha = res_body[\"object\"][\"sha\"]\n        .as_str()\n        .ok_or_else(|| anyhow!(\"Not found branch or tag\"))?;\n\n    let url = format!(\"https://api.github.com/repos/{owner}/{repo}/git/trees/{sha}?recursive=true\");\n\n    let res_body: Value = client\n        .get(&url)\n        .header(\"User-Agent\", USER_AGENT)\n        .header(\"Accept\", \"application/vnd.github+json\")\n        .header(\"X-GitHub-Api-Version\", \"2022-11-28\")\n        .send()\n        .await?\n        .json()\n        .await?;\n    let tree = res_body[\"tree\"]\n        .as_array()\n        .ok_or_else(|| anyhow!(\"Invalid github repo tree\"))?;\n    let paths = tree\n        .iter()\n        .flat_map(|v| {\n            let typ = v[\"type\"].as_str()?;\n            let path = v[\"path\"].as_str()?;\n            if typ == \"blob\"\n                && (path.ends_with(\".md\") || path.ends_with(\".MD\"))\n                && path.starts_with(&root_path)\n                && !should_exclude_link(path, exclude)\n            {\n                Some(format!(\n                    \"https://raw.githubusercontent.com/{owner}/{repo}/{branch}/{path}\"\n                ))\n            } else {\n                None\n            }\n        })\n        .collect();","sourceCodeStart":330,"sourceCodeEnd":366,"githubUrl":"https://github.com/sigoden/aichat/blob/82976d349ad97ac9aae0655ad631dace5e2a6385/src/utils/request.rs#L330-L366","documentation":"After fetching `/repos/{owner}/{repo}/git/trees/{sha}?recursive=true`, `crawl_gh_tree` expects the JSON body to contain a `tree` array and returns `anyhow!(\"Invalid github repo tree\")` when `res_body[\"tree\"].as_array()` is None. This indicates the API replied with an unexpected payload shape — usually a GitHub error object (message/errors fields) rather than a tree object.","triggerScenarios":"The trees API returns an error/oversized-response body instead of a tree: too many files for `recursive=true` (truncated or rejected), wrong SHA, rate-limited response, or a malformed/non-JSON payload.","commonSituations":"Crawling a very large monorepo where the recursive tree request is rejected or truncated; hitting GitHub's unauthenticated rate limit mid-crawl; an invalid or stale SHA passed to the trees endpoint.","solutions":["Inspect the raw response body and HTTP status — handle GitHub error messages and rate limits explicitly before parsing.","For huge repos, avoid `recursive=true`: fetch the tree level by level or use the git archive/tarball endpoint.","Re-check authentication: adding a token raises rate limits and unlocks private repos.","Re-fetch the SHA (see 'Not found branch or tag' path) if the ref may have moved."],"exampleFix":"// before\nlet res_body: Value = client.get(&url).send().await?.json().await?;\nlet tree = res_body[\"tree\"].as_array()\n    .ok_or_else(|| anyhow!(\"Invalid github repo tree\"))?;\n// after\nlet resp = client.get(&url).send().await?;\nlet res_body: Value = resp.json().await?;\nif let Some(msg) = res_body.get(\"message\").and_then(|m| m.as_str()) {\n    anyhow::bail!(\"GitHub API error: {msg}\");\n}\nlet tree = res_body[\"tree\"].as_array()\n    .ok_or_else(|| anyhow!(\"Invalid github repo tree\"))?;","handlingStrategy":"retry","validationCode":"// pre-flight: confirm the repo is small enough for a recursive tree\nasync fn tree_fits(owner: &str, repo: &str) -> bool {\n    // GitHub truncates/errors on very large recursive trees; check repo size (KB)\n    let body: serde_json::Value = reqwest::get(format!(\"https://api.github.com/repos/{owner}/{repo}\"))\n        .await.ok()?.json().await.ok()?;\n    body[\"size\"].as_i64().map(|s| s < 500_000).unwrap_or(false)\n}","typeGuard":null,"tryCatchPattern":"for attempt in 0..3 {\n    match crawl_gh_tree(owner, repo, r, &client).await {\n        Err(e) if e.to_string() == \"Invalid github repo tree\" && attempt < 2 => {\n            tokio::time::sleep(Duration::from_secs(2u64.pow(attempt))).await; // likely rate limit\n        }\n        other => { other?; break; }\n    }\n}","preventionTips":["Check HTTP status and the error `message` field before parsing the tree JSON.","Use a token to stay far below the unauthenticated rate limit.","For monorepos, walk the tree non-recursively or use the tarball endpoint.","Log the raw response body when parsing fails to make the shape mismatch debuggable."],"tags":["github","api","json","crawler"],"backgroundTag":"unexpected-response-shape","analyzedSha":"82976d349ad97ac9aae0655ad631dace5e2a6385","analyzedAt":"2026-09-09T18:33:06.139Z","contentChangedAt":"2026-09-09T18:33:06.139Z","schemaVersion":2},"datasetVersion":"2026-09-16T09:17:16.951Z"}