{"record":{"id":"e2770ecc030c9539","repo":"cube-js/cube","slug":"the-maximum-number-of-source-rows-this-options-m","errorCode":null,"errorMessage":"The maximum number of source rows ${this.options.maxSourceRowLimit} was reached for ${this.preAggregation.preAggregationId}","messagePattern":"The maximum number of source rows (.+?) was reached for (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"packages/cubejs-query-orchestrator/src/orchestrator/PreAggregationPartitionRangeLoader.ts","lineNumber":404,"sourceCode":"    const { data } = await this.queryCache.renewQuery(\n      query,\n      <string[]>values,\n      cacheKeyQueries,\n      60 * 60,\n      [query, <string[]>values],\n      undefined,\n      {\n        requestId: this.requestId,\n        skipRefreshKeyWaitForRenew: false,\n        priority: this.priority(QueuePriority.Interactive),\n        dataSource: this.dataSource,\n        external: false,\n        useCsvQuery: true,\n        lambdaTypes,\n      }\n    );\n    if (data.rowCount === this.options.maxSourceRowLimit) {\n      throw new Error(`The maximum number of source rows ${this.options.maxSourceRowLimit} was reached for ${this.preAggregation.preAggregationId}`);\n    }\n    return {\n      name: `${LAMBDA_TABLE_PREFIX}_${this.preAggregation.tableName.replace('.', '_')}`,\n      columns: data.types,\n      csvRows: data.csvRows,\n    };\n  }\n\n  public async partitionPreAggregations(): Promise<PreAggregationDescription[]> {\n    if (this.preAggregation.partitionGranularity && !this.preAggregation.expandedPartition) {\n      const { buildRange, partitionRanges } = await this.partitionRanges();\n      return this.compilerCacheFn(['partitions', JSON.stringify(buildRange)], () => partitionRanges.map(range => this.partitionPreAggregationDescription(range, buildRange)));\n    } else {\n      return [this.preAggregation];\n    }\n  }\n\n  private async partitionRanges(ignoreMatchedDateRange?: boolean): Promise<PartitionRanges> {","sourceCodeStart":386,"sourceCodeEnd":422,"githubUrl":"https://github.com/cube-js/cube/blob/7d981676b36392fec34088b9afab6bdcad40207c/packages/cubejs-query-orchestrator/src/orchestrator/PreAggregationPartitionRangeLoader.ts#L386-L422","documentation":"When building a lambda union pre-aggregation, Cube downloads the source (base) table as CSV (useCsvQuery) and enforces a hard cap on the number of source rows via options.maxSourceRowLimit. downloadLambdaTable throws when the fetched rowCount exactly equals this limit, indicating the source data likely exceeded the cap and the result may be truncated, so it aborts rather than silently building an incomplete rollup.","triggerScenarios":"Lambda union build where the base pre-aggregation/source query returns rowCount === maxSourceRowLimit; maxSourceRowLimit set too low for the dataset (default is small); unpartitioned lambda union over a very large source table.","commonSituations":"Large source tables rolled up via rollup_lambda without pre-filtering; deployments where maxSourceRowLimit was left at the default; accidental lambda (multi-stage) rollup over millions of rows instead of chaining pre-aggregations.","solutions":["Increase maxSourceRowLimit in preAggregations options to a value above your source row count (with attention to memory).","Add a filter/partition to the base pre-aggregation so the source data for the lambda union is smaller.","Restructure to a two-stage (chained) pre-aggregation: build a base pre-aggregation first, then roll up from it instead of raw rows.","If the limit is being hit coincidentally (rowCount exactly equals limit), add a sanity filter and re-run."],"exampleFix":"// before\npreAggregationsOptions: { }\n// after\npreAggregationsOptions: { maxSourceRowLimit: 20000 }","handlingStrategy":"validation","validationCode":"const rowCount = await sourceDriver.query(`SELECT COUNT(*) AS c FROM ${baseTable}`);\nif (rowCount[0].c >= maxSourceRowLimit) {\n  throw new Error(`Source rows (${rowCount[0].c}) exceed maxSourceRowLimit (${maxSourceRowLimit})`);\n}","typeGuard":null,"tryCatchPattern":"try {\n  return await loader.loadPreAggregations();\n} catch (e) {\n  if (String(e.message).includes('maximum number of source rows')) {\n    // raise maxSourceRowLimit or filter/partition the base data\n  }\n  throw e;\n}","preventionTips":["Set maxSourceRowLimit generously above expected source sizes.","Prefer chained pre-aggregations over raw-row lambda unions for large data.","Add filters/partitions to the base pre-aggregation feeding the lambda union.","Monitor base table row counts in CI for rollups using rollup_lambda."],"tags":["pre-aggregation","lambda-union","row-limit","csv-download"],"backgroundTag":"source-row-limit-exceeded","analyzedSha":"7d981676b36392fec34088b9afab6bdcad40207c","analyzedAt":"2026-09-02T03:45:10.400Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-08T15:18:49.778Z"}