apache/druid · error · ISE

Ran out of objects while reading aggregators from cache!

Error message

Ran out of objects while reading aggregators from cache!

What it means

fetchAggregatorsFromCache iterates the cached result objects in lockstep with the configured AggregatorFactory list; if the cache payload runs out of entries before all aggregators are populated, it throws an ISE. This signals that the cached blob's shape doesn't match the current aggregator spec — a cache-corruption or cache-key/versioning mismatch, never a user data problem.

Solutions

Clear the cache (or bump the cache identifierPrefix in the cache config) so stale entries created with a different schema/version are invalidated.
Confirm all Druid nodes sharing the cache run the same version; mixed-version clusters sharing one cache commonly produce shape mismatches.
Reproduce with caching disabled (set druid.caching.cache.enable=false or useCache/populateCache=false in query context) to confirm the query itself is fine.
If it recurs, check the external cache (memcached/Redis) for truncation/eviction bugs and enable cache-populate/populateCache logging.

Example fix

// before
cacheConfig = CacheConfig.builder().setIdentifierPrefix("druid-segment-v1")... // reused across upgrades
// after
CacheConfig cacheConfig = CacheConfig.builder().setIdentifierPrefix("druid-segment-v2-<newAggregatorSchema>")... // bump prefix when aggregator schema or Druid version changes

Defensive patterns

Strategy: retry

Try / catch

try {
  return cachedStrategyRead(cacheEntry, aggregators);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("Ran out of objects while reading aggregators")) {
    cacheClient.invalidate(cacheKey); // drop mismatched entry
    return executeWithoutCache(query); // recompute from segments
  }
  throw e;
}

Prevention

Bump the cache identifierPrefix whenever the aggregation schema or Druid version changes.
Keep all cluster nodes on the same Druid version when sharing a cache.
Periodically flush the cache after upgrades and schema migrations.
Validate external cache integrity (memcached/Redis) if truncation is suspected.

When it happens

Trigger: Reading a segment-level or result-level cache entry whose number of result objects is fewer than aggregators.size(): stale cache entries from before a query spec/schema change, a cache populated by a different Druid version, or corrupted/partially-written cache rows (e.g. in memcached/Redis).

Common situations: Upgrading Druid or changing an ingestion/aggregation spec without bumping the cache identifier prefix (druid.cache.identifier / cache config), so old cache entries deserialize against new expectations; hit-ratio misconfigurations mixing cache namespaces; truncated cache values from an eviction-buggy external cache.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ef8f9215d4e59172. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/query/CacheStrategy.java:157

   * When using the result level cache, the agg values seen here are
   * finalized values generated by AggregatorFactory.finalizeComputation().
   * These finalized values are deserialized from the cache as generic Objects, which will
   * later be reserialized and returned to the user without further modification.
   * Because the agg values are deserialized as generic Objects, the values are subject to the same
   * type consistency issues handled by DimensionHandlerUtils.convertObjectToType() in the pullFromCache implementations
   * for dimension values (e.g., a Float would become Double).
   */
  static void fetchAggregatorsFromCache(
      List<AggregatorFactory> aggregators,
      Iterator<Object> resultIter,
      boolean isResultLevelCache,
      AddToResultFunction addToResultFunction
  )
  {
    for (int i = 0; i < aggregators.size(); i++) {
      final AggregatorFactory aggregator = aggregators.get(i);
      if (!resultIter.hasNext()) {
        throw new ISE("Ran out of objects while reading aggregators from cache!");
      }

      ColumnType resultType = aggregator.getResultType();
      ColumnType intermediateType = aggregator.getIntermediateType();

      boolean needsDeserialize = !isResultLevelCache || resultType.equals(intermediateType);

      if (needsDeserialize) {
        addToResultFunction.apply(aggregator.getName(), i, aggregator.deserialize(resultIter.next()));
      } else {
        addToResultFunction.apply(aggregator.getName(), i, resultIter.next());
      }
    }
  }

  interface AddToResultFunction
  {
    void apply(String aggregatorName, int aggregatorIndex, Object object);

View on GitHub (pinned to 9b90983fd2)