apache/beam · error · IllegalStateException

Expected state to be STARTED, but was COMPLETE_ERROR

Error message

Expected state to be STARTED, but was COMPLETE_ERROR

What it means

RpcQosImpl tracks the lifecycle of a Firestore RPC attempt through an internal state machine (STARTED, PENDING, COMPLETE_SUCCESS, COMPLETE_ERROR). Attempt-scoped methods like getMsToNextRttBreaker check via checkStarted() that the attempt is still in the STARTED state before acting. This IllegalStateException is thrown when the method is called on an attempt that already completed with an error, meaning the caller is reusing a finished attempt instead of starting a new one.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosImpl.java:206

        case COMPLETE_SUCCESS:
          throw new IllegalStateException(
              "Expected state to be PENDING or STARTED, but was COMPLETE_SUCCESS");
        case COMPLETE_ERROR:
          throw new IllegalStateException(
              "Expected state to be PENDING or STARTED, but was COMPLETE_ERROR");
      }
    }

    public void checkStarted() {
      switch (this) {
        case STARTED:
          return;
        case PENDING:
          throw new IllegalStateException("Expected state to be STARTED, but was PENDING");
        case COMPLETE_SUCCESS:
          throw new IllegalStateException("Expected state to be STARTED, but was COMPLETE_SUCCESS");
        case COMPLETE_ERROR:
          throw new IllegalStateException("Expected state to be STARTED, but was COMPLETE_ERROR");
      }
    }
  }

  private abstract class BaseRpcAttempt implements RpcAttempt {
    private final Logger logger;
    final O11y o11y;
    final StatusCodeAwareBackoff backoff;
    final Sleeper sleeper;

    AttemptState state;
    Instant start;

    @SuppressWarnings(
        "initialization.fields.uninitialized") // allow transient fields to be managed by component
    // lifecycle
    BaseRpcAttempt(Context context, O11y o11y, StatusCodeAwareBackoff backoff, Sleeper sleeper) {
      this.logger = LoggerFactory.getLogger(String.format("%s.RpcQos", context.getNamespace()));

View on GitHub (pinned to 12126d8942)

Solutions

Obtain a new attempt via rpcQos.newAttempt() inside the retry loop for every retry, instead of reusing the failed attempt
Inspect the stack trace to find which method was called after the attempt was already aborted/failed and move that call before the failure point
Ensure awaitOutOfBandPermission/next attempt logic follows the pattern in Beam's Firestore V1 connector (see RpcQosImpl usage in FirestoreV1Fn)
Update to a newer Beam version in case the error stems from a fixed QoS state-handling bug

Example fix

// before
RpcAttempt attempt = qos.newAttempt();
for (int i = 0; i < retries; i++) {
  try { attempt.pause(); doRpc(); break; } catch (Exception e) { /* reuse attempt */ }
}
// after
for (int i = 0; i < retries; i++) {
  RpcAttempt attempt = qos.newAttempt();
  try { attempt.checkStarted(); doRpc(); break; } catch (Exception e) { /* fresh attempt next loop */ }
}

Defensive patterns

Strategy: try-catch

Validate before calling

// before using an attempt, guard on state
if (attempt != null && isStarted(attempt)) { attempt.pause(); ... }
// implement isStarted by tracking state in your wrapper since state is private

Type guard

boolean isFreshAttempt(RpcAttempt a) { return a instanceof RpcQosImpl.RpcWriteAttempt || /* track via wrapper flag */ attemptUsed == false; }

Try / catch

try {
  attempt.pause();
  // rpc work
} catch (IllegalStateException e) {
  if (e.getMessage().contains("Expected state to be STARTED")) {
    attempt = rpcQos.newAttempt(); // get a fresh attempt and retry
  } else throw e;
}

Prevention

Create a new RpcAttempt per try iteration — never reuse an attempt after a failure
Wrap attempt usage in a small helper class that marks the attempt consumed after any exception
Follow the exact pattern in Beam's FirestoreV1Fn implementation

When it happens

Trigger: Calling a method on an RpcAttempt (from RpcQos.newAttempt) after a prior call on the same attempt threw an exception and moved the attempt into the COMPLETE_ERROR state — typically retry logic that keeps using the old attempt object instead of requesting a fresh one.

Common situations: Custom retry loops around Firestore RPCs that catch a GoogleApiException but retry on the same attempt; writing a custom DoFn that caches a single RpcAttempt across elements; upgrading Beam versions where the QoS layer began enforcing attempt state transitions strictly.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/431545985896f6bf. Report an issue: GitHub.