[Gitlab Loader]: Rate limit persists for ${endpoint} after $

Error message

[Gitlab Loader]: Rate limit persists for ${endpoint} after ${retries} retries. Skipping.

What it means

The paginated fetch helper fetchNextPage kept receiving 429 and exhausted MAX_RETRIES (3), so it returns null and pagination stops. Only the pages already fetched make it into the workspace; later pages (commits, tree entries) are silently dropped.

Source

Thrown at collector/utils/extensions/RepoLoader/GitlabRepo/RepoLoader/index.js:394

      if (requestData.page === -1) return null;
      if (!requestData.page) requestData.page = 1;

      const { endpoint, perPage = 100, queryParams = {} } = requestData;
      const params = new URLSearchParams({
        ...queryParams,
        per_page: perPage,
        page: requestData.page,
      });
      const url = `${this.apiBase}${endpoint}?${params.toString()}`;

      const response = await fetch(url, {
        method: "GET",
        headers: this.accessToken ? { "PRIVATE-TOKEN": this.accessToken } : {},
      });

      if (response.status === 429) {
        if (retries >= MAX_RETRIES) {
          console.warn(
            `[Gitlab Loader]: Rate limit persists for ${endpoint} after ${retries} retries. Skipping.`
          );
          return null;
        }
        const retryAfter = Number(response.headers.get("retry-after")) || 60;
        console.warn(
          `[Gitlab Loader]: Rate limit hit for ${endpoint}. Waiting ${retryAfter}s before retrying...`
        );
        await this.#wait(retryAfter * 1000);
        return this.fetchNextPage(requestData, retries + 1);
      }

      if (response.status === 401) {
        console.warn(
          `[Gitlab Loader]: Unauthorized request for ${endpoint}. Skipping.`
        );
        return null;
      }

View on GitHub (pinned to 3aec848f28)

Solutions

Re-run the sync after the rate-limit window resets so pagination can complete.
Shrink the working set: narrower repo path, specific branch, or stricter include/exclude filters.
Use a token with a higher limit or sync against a self-hosted GitLab with raised limits.
Stagger concurrent collector jobs so they do not share one quota window.

Defensive patterns

Strategy: retry

Try / catch

const page = await loader.fetchNextPage({ page: n });
if (page === null) { stopAndFlagIncomplete(resource); }

Prevention

Re-run the sync after throttle windows rather than retrying immediately by hand.
Reduce pagination volume with branch/path filters.
Distribute sync jobs across tokens or time windows.
Alert on 'persists after N retries' lines as the real data-loss signal.

When it happens

Trigger: Enumerating a very large GitLab project's repository tree or branches with tight per-minute limits; a token/IP already throttled by parallel jobs when pagination began.

Common situations: Monorepos with thousands of paths; multiple AnythingLLM instances syncing the same GitLab host concurrently.

Related errors

AI-assisted analysis of Mintplex-Labs/anything-llm@3aec848f28 (2026-08-18). Data as JSON: /api/errors/66a6d8288c86146c. Report an issue: GitHub.