apache/beam · warning

No shard found for [shards= ]

Error message

No shard found for {} [shards={}]

What it means

KinesisIO's shard-aware partitioner looks up which shard bounds contain the hashed partition key. When the cached shard map is non-empty but no shard's hash range covers the hashed key, it logs this warning and returns null, meaning the record cannot be mapped to a shard and will follow the fallback routing.

Solutions

1. Enable/shorten withRefreshPeriod so shard bounds refresh promptly after resharding
2. Ensure the stream's shards cover the full 0..2^128 hash range (a reshard in progress leaves gaps); wait or trigger a manual refresh
3. Reduce producers' reliance on explicit partition keys that hash into shrinking shard ranges, or use a wider key space
4. Upgrade Beam — shard map refresh handling has been improved in later versions

Example fix

// before
KinesisIO.write().withStreamName("s").withNumShards(10)
// after: enable periodic shard-boundary refresh so bounds stay current
KinesisIO.write().withStreamName("s")
  .withShardAwareRandomization(true)
  .withRefreshPeriod(Duration.standardSeconds(30));

// check stream shard coverage before writing
DescribeStreamSummary s = client.describeStreamSummary(b -> b.streamName(stream)).streamDescriptionSummary();
List<Shard> shards = listAllShards(client, stream);
BigInteger min = shards.stream().map(shard -> lowerHashKey(shard)).min(BigInteger::compareTo).orElse(null);
if (min == null || min.signum() != 0) log.warn("stream {} not fully covered by shards", stream);

When it happens

Trigger: Writing to a Kinesis stream with shard-aware hashing enabled (withShardAwareRandomization / refreshPeriodically) while the cached lower hash-key bounds are stale relative to the live stream, e.g. right after resharding (split/merge) so a hash key falls into a gap between stale bounds.

Common situations: Streams that recently split or merged shards while the writer still holds an old shard snapshot; explicit partition keys whose hash lands outside all cached ranges during the window before the next refresh tick.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• Executor service is taking long time to shutdown, will…
• Executor service was not completely terminated after
• Expected input schema with a 'data' (BYTES) field, a…
• Failed to refresh shards.
• Illegal shard number

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4b456c351cc05828. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java:1230

      class ShardRangesImpl implements ShardRanges {
        private static final Logger LOG = LoggerFactory.getLogger(ShardRanges.class);

        private final String streamName;

        private final AtomicBoolean running = new AtomicBoolean(false);
        private NavigableSet<BigInteger> shardBounds = ImmutableSortedSet.of();
        private Instant nextRefresh = Instant.EPOCH;

        private ShardRangesImpl(String streamName) {
          this.streamName = streamName;
        }

        @Override
        public @Nullable BigInteger shardAwareHashKey(BigInteger hashedPartitionKey) {
          BigInteger lowerBound = shardBounds.floor(hashedPartitionKey);
          if (!shardBounds.isEmpty() && lowerBound == null) {
            LOG.warn("No shard found for {} [shards={}]", hashedPartitionKey, shardBounds.size());
          }
          return lowerBound;
        }

        @Override
        public void refreshPeriodically(
            KinesisAsyncClient client, Supplier<Instant> nextRefreshFn) {
          if (nextRefresh.isBeforeNow() && running.compareAndSet(false, true)) {
            refresh(client, nextRefreshFn, new TreeSet<>(), null);
          }
        }

        @SuppressWarnings("FutureReturnValueIgnored") // safe to ignore
        private void refresh(
            KinesisAsyncClient client,
            Supplier<Instant> nextRefreshFn,
            NavigableSet<BigInteger> bounds,
            @Nullable String nextToken) {

View on GitHub (pinned to 12126d8942)