[Gitlab Loader]: Rate limit persists for ${sourceFilePath} a

Error message

[Gitlab Loader]: Rate limit persists for ${sourceFilePath} after ${retries} retries. Skipping.

What it means

GitlabRepoLoader.fetchSingleFileContents got HTTP 429 on every attempt; after MAX_RETRIES (3) recursive retries it gives up and returns null for that file, so the file is missing from the embedded workspace. Each retry waited for the Retry-After header or 60s by default before recursing.

Source

Thrown at collector/utils/extensions/RepoLoader/GitlabRepo/RepoLoader/index.js:346

   * Fetches the content of a single file from the repository.
   * @param {string} sourceFilePath - The path to the file in the repository.
   * @returns {Promise<string|null>} The content of the file, or null if fetching fails.
   */
  async fetchSingleFileContents(sourceFilePath, retries = 0) {
    try {
      const url = `${this.apiBase}/api/v4/projects/${
        this.projectId
      }/repository/files/${encodeURIComponent(sourceFilePath)}/raw?ref=${
        this.branch
      }`;
      const response = await fetch(url, {
        method: "GET",
        headers: this.accessToken ? { "PRIVATE-TOKEN": this.accessToken } : {},
      });

      if (response.status === 429) {
        if (retries >= MAX_RETRIES) {
          console.warn(
            `[Gitlab Loader]: Rate limit persists for ${sourceFilePath} after ${retries} retries. Skipping.`
          );
          return null;
        }
        const retryAfter = Number(response.headers.get("retry-after")) || 60;
        console.warn(
          `[Gitlab Loader]: Rate limit hit fetching ${sourceFilePath}. Waiting ${retryAfter}s...`
        );
        await this.#wait(retryAfter * 1000);
        return this.fetchSingleFileContents(sourceFilePath, retries + 1);
      }

      if (!response.ok)
        throw new Error(`Failed to fetch single file ${sourceFilePath}`);

      return await response.text();
    } catch (e) {
      console.error(`RepoLoader.fetchSingleFileContents`, e);

View on GitHub (pinned to 3aec848f28)

Solutions

Wait out the quota window (usually minutes) and re-run the sync — only skipped files remain missing.
Tighten include/exclude path filters so fewer single-file raw fetches are needed.
Use a PAT with higher rate limits, or relax limits on a self-hosted GitLab.
If it recurs consistently, raise the MAX_RETRIES constant or add jittered backoff between file fetches.

Defensive patterns

Strategy: retry

Prevention

Keep include/exclude path filters tight to minimize per-file raw fetches.
Track which files returned null and re-sync just those after the quota window resets.
Use a PAT with elevated limits for large projects.
Avoid running multiple collectors against one GitLab host simultaneously.

When it happens

Trigger: Bulk-fetching many individual files from a GitLab project (per-file raw fetches) against GitLab.com per-IP/per-token quotas; a throttled self-hosted GitLab; several concurrent syncs sharing one token.

Common situations: Deep repos with hundreds of files when ignore-filtering is loose; CI jobs syncing the same project repeatedly in a short window.

Related errors

AI-assisted analysis of Mintplex-Labs/anything-llm@3aec848f28 (2026-08-18). Data as JSON: /api/errors/347402084588c179. Report an issue: GitHub.