sigoden/aichat · error · anyhow::Error

Invalid gh tree

Error message

Invalid gh tree {}

What it means

crawl_gh_tree parses a GitHub tree/blob URL into path segments to extract owner, repo, branch, and root path. It requires at least 4 segments in the URL path (origin/host + owner/repo/...); URLs that don't match a GitHub repository layout (e.g. a bare github.com page or a short URL) are rejected with 'Invalid gh tree {url}'.

Solutions

1. Pass a full GitHub tree or blob URL with owner, repo, and branch, e.g. https://github.com/owner/repo/tree/main
2. Check the URL for missing segments (repo name or branch omitted)
3. If you want to crawl general GitHub pages, use a non-GitHub crawl path or the HTML crawl flow instead of the gh tree mode

Example fix

// before
crawl_website("https://github.com/sigoden", options)?;
// after
crawl_website("https://github.com/sigoden/aichat/tree/main", options)?;

fn is_valid_gh_tree_url(u: &str) -> bool {
    url::Url::parse(u).map(|u| {
        u.host_str().map_or(false, |h| h.contains("github.com"))
            && u.path().split('/').filter(|s| !s.is_empty()).count() >= 3
    }).unwrap_or(false)
}
// if !is_valid_gh_tree_url(start_url) { /* fix URL before crawling */ }

match crawl_website(start_url, options).await {
    Err(e) if e.to_string().starts_with("Invalid gh tree") => {
        eprintln!("{e}: provide a full github tree/blob URL like https://github.com/owner/repo/tree/branch");
    }
    other => other?,
}

When it happens

Trigger: Calling crawl_website with a start_url whose path has fewer than 4 segments, e.g. https://github.com or https://github.com/owner — not a full repo tree/blob URL like https://github.com/owner/repo/tree/branch/dir.

Common situations: Passing a GitHub repository root or user profile URL instead of a tree/blob URL; a typo dropping the repo or branch segment; expecting generic web crawling on a GitHub URL that only supports tree crawling.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• Invalid crawl page at
• Not found branch or tag
• Invalid github repo tree
• Invalid extract selector

AI-assisted analysis of sigoden/aichat@82976d349a (2026-09-09). Data as JSON: /api/errors/51815f5e161b42a6. Report an issue: GitHub.

Appendix: source

Thrown at src/utils/request.rs:308

        }
        paths.extend(new_paths);

        index += batch.len();
    }

    Ok(result_pages)
}

#[derive(Debug, Deserialize)]
pub struct Page {
    pub path: String,
    pub text: String,
}

async fn crawl_gh_tree(start_url: &Url, exclude: &[String]) -> Result<Vec<String>> {
    let path_segs: Vec<&str> = start_url.path().split('/').collect();
    if path_segs.len() < 4 {
        bail!("Invalid gh tree {}", start_url.as_str());
    }
    let client = match *CLIENT {
        Ok(ref client) => client,
        Err(ref err) => bail!("{err}"),
    };
    let owner = path_segs[1];
    let repo = path_segs[2];
    let branch = path_segs[4];
    let root_path = path_segs[5..].join("/");

    let url = format!("https://api.github.com/repos/{owner}/{repo}/git/ref/heads/{branch}");

    let res_body: Value = client
        .get(&url)
        .header("User-Agent", USER_AGENT)
        .header("Accept", "application/vnd.github+json")
        .header("X-GitHub-Api-Version", "2022-11-28")
        .send()

View on GitHub (pinned to 82976d349a)