{"record":{"id":"bf3a6156dfca6258","repo":"jackwener/OpenCLI","slug":"twitter-collection-limit-reached","errorCode":"twitter_collection_limit_reached","errorMessage":"twitter_collection_limit_reached: pagination cannot prove completion","messagePattern":"twitter_collection_limit_reached: pagination cannot prove completion","errorType":"error_code","errorClass":"CommandExecutionError","httpStatus":null,"severity":"error","filePath":"clis/twitter/collection.js","lineNumber":270,"sourceCode":"    };\n}\n\nasync function paginateCollection({ until, limit, maxPages = MAX_USER_TWEETS_PAGES, fetchPage, wait }) {\n    const seen = new Set();\n    const seenCursors = new Set();\n    const posts = [];\n    let cursor = null;\n    let oldestSeenAt = null;\n    for (let pageIndex = 0; pageIndex < maxPages; pageIndex++) {\n        if (pageIndex > 0 && wait) await wait();\n        const payload = await fetchPage(cursor, USER_TWEETS_PAGE_SIZE);\n        if (payload?.error) {\n            throw new CommandExecutionError(`twitter_collection_request_error: UserTweets returned ${payload.error}`);\n        }\n        const [pagePosts, nextCursor] = parseCollectionPage(payload, seen);\n        for (const post of pagePosts) {\n            if (posts.length >= limit) {\n                throw new CommandExecutionError('twitter_collection_limit_reached: pagination cannot prove completion');\n            }\n            const createdAt = parseCreatedAt(post);\n            if (!oldestSeenAt || createdAt < oldestSeenAt) oldestSeenAt = createdAt;\n            posts.push(post);\n            if (createdAt <= until) {\n                return {\n                    posts,\n                    receipt: completedReceipt('time_boundary_reached', until, pageIndex + 1, oldestSeenAt),\n                };\n            }\n        }\n        if (!nextCursor) {\n            return {\n                posts,\n                receipt: completedReceipt('cursor_exhausted', until, pageIndex + 1, oldestSeenAt),\n            };\n        }\n        if (nextCursor === cursor || seenCursors.has(nextCursor)) {","sourceCodeStart":252,"sourceCodeEnd":288,"githubUrl":"https://github.com/jackwener/OpenCLI/blob/49907e53dc3ade5c223ff0c4c2c2785687cec4e6/clis/twitter/collection.js#L252-L288","documentation":"When a page pushes the collected posts to the configured limit, paginateCollection throws instead of stopping gracefully — because cutting off mid-pagination means the run cannot prove it reached the requested 'until' timestamp. The library treats an uncompleted collection as an error so callers never mistake a truncated set for a complete one.","triggerScenarios":"The user's limit (or default) is smaller than the number of posts published by the account since the 'until' cutoff — pagination would need to stop mid-run without reaching until.","commonSituations":"Collecting the last N hours from a very high-volume account with a small limit; reusing a limit tuned for one account against a far more prolific account; passing an 'until' far in the past so the whole timeline must be walked.","solutions":["Increase the limit option so it exceeds the expected post count between now and 'until'.","Move the 'until' cutoff closer to the present to reduce the number of posts that must be walked.","Incremental collection: store the newest timestamp seen and use it as the next 'until' so each run walks fewer posts.","Accept partial data explicitly (catch this error and keep posts accumulated so far, understanding completion is unproven).","Raise maxPages in tandem with limit so pagination is actually allowed to reach the cutoff."],"exampleFix":"// before\nawait cliRun({ limit: 100, until: sevenDaysAgo });\n// after\nawait cliRun({ limit: 5000, until: oneHourAgo }); // limit large enough to cover posts since cutoff","handlingStrategy":"validation","validationCode":"const estimate = (now - until) / averageIntervalMs;\nif (estimate > limit) {\n  console.warn(`limit ${limit} likely insufficient (~${Math.ceil(estimate)} posts expected)`);\n}","typeGuard":null,"tryCatchPattern":"try {\n  const { posts, receipt } = await collection(...);\n} catch (err) {\n  if (String(err.message).startsWith('twitter_collection_limit_reached')) {\n    // accept partial data explicitly, or rerun with larger limit / nearer until\n  } else throw err;\n}","preventionTips":["Size limit above expected post volume between now and 'until'.","Collect incrementally using the newest timestamp as the next 'until'.","Raise maxPages together with limit.","Never treat a thrown limit error as a complete collection.","Tighten 'until' for high-volume accounts."],"tags":["twitter","pagination","limit","completeness"],"backgroundTag":"pagination-incomplete","analyzedSha":"49907e53dc3ade5c223ff0c4c2c2785687cec4e6","analyzedAt":"2026-08-29T08:14:47.543Z","schemaVersion":2},"datasetVersion":"2026-08-29T12:17:43.993Z"}