upload parked: credentials rejected (HTTP 401); retrying in

Error message

upload parked: credentials rejected (HTTP 401); retrying in background until auth recovers

What it means

This error is delivered when the storage service keeps rejecting uploads with HTTP 401 even after the queue refreshes credentials. The worker parks the item and keeps retrying in the background until auth recovers, notifying the completion channel with this descriptive error so the caller knows the upload is deferred, not permanently failed. It is an authentication-state signal rather than a data or network fault.

Source

Thrown at crates/codegen/xai-file-utils/src/queue.rs:2135

                            tracing::warn!(
                                attempt = item.attempts,
                                parked,
                                error = ?e,
                                "Auth error persists after credential refresh, aborting"
                            );
                            return Err(e);
                        };
                        if !parked {
                            parked = true;
                            stats.auth_parked.fetch_add(1, Ordering::Relaxed);
                            tracing::warn!(
                                attempt = item.attempts,
                                gcs_path = %item.gcs_path,
                                "401 persists after credential refresh; parking item until auth recovers"
                            );
                            notify_completion(
                                item,
                                Err(anyhow::anyhow!(
                                    "upload parked: credentials rejected (HTTP 401); \
                                     retrying in background until auth recovers"
                                )),
                            );
                        }
                        if item.enqueued_at.elapsed() >= policy.max_age {
                            tracing::warn!(
                                attempt = item.attempts,
                                age_secs = item.enqueued_at.elapsed().as_secs(),
                                "Parked item exceeded max_age waiting for auth recovery, aborting"
                            );
                            return Err(e);
                        }
                        wake = wait.await
                            || (last_wire_attempt.elapsed() >= policy.auth_park_probe_interval
                                && resolver.has_usable_credential());
                    }
                    if let Some(p) = permit.as_deref_mut() {

View on GitHub (pinned to bc7f02eddd)

Solutions

Verify and rotate the credentials file / token source the client uses; replace revoked service-account keys.
Check system clock sync (NTP) to avoid freshly issued tokens being rejected.
Confirm IAM permissions for uploads on the target bucket are intact for the authenticated identity.
Treat this completion error as 'parked, will retry' and monitor stats; re-check auth config while it retries in background.

Example fix

// before
// running with stale key
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/old-key.json

// after
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/rotated-key.json
gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS

Defensive patterns

Strategy: try-catch

Validate before calling

// Validate credentials are usable before enqueueing
async fn auth_ok(client: &StorageClient) -> bool {
    client.check_auth().await.is_ok() // or a cheap authenticated probe
}

Try / catch

// Recognize the parked-upload error and defer
if e.to_string().contains("credentials rejected (HTTP 401)") {
    tracing::warn!("upload parked pending auth recovery; will retry later");
    schedule_reauth_and_reenqueue();
}

Prevention

Rotate service-account keys before expiry and reload them promptly.
Keep system clocks NTP-synced so fresh tokens validate.
Verify IAM upload permissions after role/policy changes.
Watch the 401 attribution hooks/fire_401_attribution signals to react early.

When it happens

Trigger: Calling upload/enqueue with credentials the GCS backend rejects (401) and the refresh path produces still-invalid credentials; expired service-account keys or revoked tokens persisting past refresh; clock skew making freshly issued tokens appear invalid.

Common situations: Expired or rotated service-account key files on disk; environment with stale cached tokens after a tenant/role change; system clock drift; IAM policy removing upload permission from the identity used.

Understand the failure class

HTTP status errors: handling 4xx and 5xx responses — how to handle 4xx and 5xx responses properly.

Related errors

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/87e632c56e780a13. Report an issue: GitHub.