xai-org/grok-build · error

expired

Error message

expired

What it means

This error is raised when an upload queue item exceeds the retry policy's max_age and is dropped as expired. The worker removes the item's files, marks it failed in stats, settles any accounted bytes, and delivers this error to the item's completion channel. It means the upload never succeeded within its allowed lifetime and was intentionally discarded.

Source

Thrown at crates/codegen/xai-file-utils/src/queue.rs:1934

    stats: &Arc<UploadQueueStats>,
    consecutive_failures: &Arc<AtomicU32>,
    draining: &Arc<std::sync::atomic::AtomicBool>,
    mut permit: Option<ConcurrencyPermit>,
) {
    let size = file_size(item.source.path());
    let accounted_bytes = item.source.disk_bytes(size);
    stats.inflight.fetch_add(1, Ordering::Relaxed);
    stats.notify_transition();
    if item.enqueued_at.elapsed() > retry_policy.max_age {
        tracing::warn!(
            age_secs = item.enqueued_at.elapsed().as_secs(),
            outcome = "expired",
            "Dropping expired upload queue item"
        );
        remove_item_files(&item, Some(stats));
        stats.failed.fetch_add(1, Ordering::Relaxed);
        settle_pending(stats, accounted_bytes);
        notify_completion(&mut item, Err(anyhow::anyhow!("expired")));
        return;
    }
    let result = upload_with_retries(
        &mut item,
        resolver,
        retry_policy,
        size,
        stats,
        draining,
        permit.as_mut(),
    )
    .await;
    match result {
        Ok((url, compression, stored_size)) => {
            let compressed = matches!(compression, BlobCompression::Zstd);
            tracing::info!(
                attempts = item.attempts,
                size_bytes = size,

View on GitHub (pinned to bc7f02eddd)

Solutions

1. Increase the retry policy's max_age so long-running uploads are not expired during outages.
2. Monitor queue stats.failed and completion receivers; re-enqueue dropped items after the transient issue resolves.
3. Shorten wait_for_network_retry backoff or cap attempts so items finish within max_age.
4. Fix the underlying auth/network problem quickly; the queue intentionally gives up at max_age.

Example fix

// before
let policy = RetryPolicy { max_age: Duration::from_secs(300), .. };

// after
let policy = RetryPolicy { max_age: Duration::from_secs(3600), .. };

// Ensure the retry budget fits the worst-case queue wait
if policy.max_age < retry_policy.backoff * (retry_policy.max_retries as u32) {
    anyhow::bail!("max_age too small for retry policy; items will expire");
}

// Treat expiration as a skip-and-re-enqueue
match completion_rx.await {
    Ok(url) => use_url(url),
    Err(e) if e.to_string() == "expired" => {
        tracing::warn!("upload expired; re-enqueueing");
        queue.enqueue(item).await?;
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: An upload retrying repeatedly due to transient network failures or 401 auth outages until enqueued_at.elapsed() reaches policy.max_age; a queue worker processing a deeply backlogged item whose deadline already passed; extremely long retry backoff configurations exceeding max_age.

Common situations: Prolonged credential outage (401 parking) that outlasts the item's max age; sustained network partition while the queue keeps retrying; misconfigured max_age too short for large-file uploads; service suspension that lets queued items age out.

Related errors

• reference snapshot did not match expected sha256; upload ski
• upload parked: credentials rejected (HTTP 401); retrying in
• {} failed: {}
• blocking pool pre-warm stalled after {started} of {n} thread
• wait failed: {body}

AI-assisted analysis of xai-org/grok-build@bc7f02eddd (2026-08-31). Data as JSON: /api/errors/6fdc2d30c23c0e9e. Report an issue: GitHub.