apache/druid · info

fetchThreads [ ] being lowered to [ ]

Error message

fetchThreads [%d] being lowered to [%d]

What it means

KinesisIndexTask.computeFetchThreads() caps the number of fetch threads by available memory: maxFetchThreads = memoryToUse / GET_RECORDS_MAX_BYTES_PER_CALL. When the configured value exceeds this cap, it logs 'fetchThreads [%d] being lowered to [%d]' and silently reduces fetchThreads to the memory-safe maximum.

Solutions

1. Lower fetchThreads in the tuning config to a value within the memory-derived cap.
2. Increase the task's memory allocation (worker capacity / task runtime properties) if more threads are genuinely needed.
3. Rely on the automatic cap if acceptable — behavior stays correct, just fewer threads.
4. Recalculate: fetchThreads must be <= taskMemoryBytes / GET_RECORDS_MAX_BYTES_PER_CALL (and > 0).

Example fix

// before
"tuningConfig": { "type": "kinesis", "fetchThreads": 20 } // task has 2GB
// after
"tuningConfig": { "type": "kinesis", "fetchThreads": 4 } // <= memory / GET_RECORDS_MAX_BYTES_PER_CALL

int max = (int) (taskMemoryBytes / GET_RECORDS_MAX_BYTES_PER_CALL);
if (fetchThreads > max) { fetchThreads = max; } // pre-clamp before submitting spec

When it happens

Trigger: Configuring a large fetchThreads in the Kinesis tuning config while the task's memory budget (divided by GET_RECORDS_MAX_BYTES_PER_CALL, max bytes per GetRecords call) allows fewer threads.

Common situations: Setting fetchThreads=high on small task containers (low -Xmx or task memory); memory reductions after right-sizing; copied configs from larger clusters.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• The 'maxRecordsPerPoll' config property of the kinesis…
• The 'recordBufferSize' config property of the kinesis…
• Already closed
• buffer for list is too small, was
• can't reschedule fetch records runnable, recordsResult is…

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/035483a83f9adda7. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/kinesis-indexing-service/src/main/java/org/apache/druid/indexing/kinesis/KinesisIndexTask.java:216

      fetchThreads = runtimeInfo.getAvailableProcessors() * 2;
    }

    // Each fetchThread can return upto 10MB at a time
    // (https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html), cap fetchThreads so that
    // we don't exceed more than the least of 100MB or 5% of heap at a time. Don't fail if fetchThreads specified
    // is greater than this as to not cause failure for older configurations, but log warning in this case, and lower
    // fetchThreads implicitly.
    final long memoryToUse = Math.min(
        KinesisIndexTaskIOConfig.MAX_RECORD_FETCH_MEMORY,
        (long) (runtimeInfo.getMaxHeapSizeBytes() * KinesisIndexTaskIOConfig.RECORD_FETCH_MEMORY_MAX_HEAP_FRACTION)
    );
    int maxFetchThreads = Math.max(
        1,
        (int) (memoryToUse / GET_RECORDS_MAX_BYTES_PER_CALL)
    );
    if (fetchThreads > maxFetchThreads) {
      if (configuredFetchThreads != null) {
        log.warn("fetchThreads [%d] being lowered to [%d]", configuredFetchThreads, maxFetchThreads);
      }
      fetchThreads = maxFetchThreads;
    }

    Preconditions.checkArgument(
        fetchThreads > 0,
        "Must have at least one background fetch thread for the record supplier"
    );

    return fetchThreads;
  }
}

View on GitHub (pinned to 9b90983fd2)