containerd/containerd · warning · errdefs.ErrUnavailable

push is on-going: %w

Error message

push is on-going: %w

What it means

dockerPusher.push checks the StatusTracker; if another push of the same ref is currently active (a status exists that is not committed and not closed, and unavailableOnFail is set for the Writer path), it returns errdefs.ErrUnavailable with "push is on-going". This lets concurrent clients know an upload is already in progress elsewhere.

Source

Thrown at core/remotes/docker/pusher.go:100

	if err != nil {
		return nil, err
	}
	if p.dockerBase.warningHandler != nil {
		ctx = context.WithValue(ctx, warningSourceKey{}, WarningSource{
			Desc:   &desc,
			Digest: &desc.Digest,
		})
	}
	status, err := p.tracker.GetStatus(ref)
	if err == nil {
		if status.Committed && status.Offset == status.Total {
			return nil, fmt.Errorf("ref %v: %w", ref, errdefs.ErrAlreadyExists)
		}
		if unavailableOnFail && status.ErrClosed == nil {
			// Another push of this ref is happening elsewhere. The rest of function
			// will continue only when `errdefs.IsNotFound(err) == true` (i.e. there
			// is no actively-tracked ref already).
			return nil, fmt.Errorf("push is on-going: %w", errdefs.ErrUnavailable)
		}
		// TODO: Handle incomplete status
	} else if !errdefs.IsNotFound(err) {
		return nil, fmt.Errorf("failed to get status: %w", err)
	}

	hosts := p.filterHosts(HostCapabilityPush)
	if len(hosts) == 0 {
		return nil, fmt.Errorf("no push hosts: %w", errdefs.ErrNotFound)
	}

	var (
		isManifest bool
		existCheck []string
		host       = hosts[0]
	)

	if images.IsManifestType(desc.MediaType) || images.IsIndexType(desc.MediaType) {

View on GitHub (pinned to 4246446a2b)

Solutions

1. Serialize pushes per ref (single-flight/mutex) so only one push runs at a time
2. Wait for the other push to finish, then retry — treat ErrUnavailable as transient
3. Check whether concurrent workers are deduplicating refs properly (InFlightTracker usage)
4. If the other push crashed without closing status, restart the client/process to clear tracker state

Example fix

// before
go pushLayer(ctx, desc); go pushLayer(ctx, desc)
// after
var mu sync.Mutex
go func(){ mu.Lock(); defer mu.Unlock(); pushLayer(ctx, desc) }()

// check for an active push before starting another
if st, err := tracker.GetStatus(ref); err == nil && !st.Committed && st.ErrClosed == nil {
    return fmt.Errorf("push active for %s, wait", ref)
}

null

err := pusher.Push(ctx, desc)
if errdefs.IsUnavailable(err) {
    time.Sleep(backoff)
    return pusher.Push(ctx, desc) // previous push likely finished
}
return err

When it happens

Trigger: Calling Pusher.Writer(ctx, ...) (unavailableOnFail=true) while another goroutine/process holds an active, unclosed status for the same ref in the tracker; concurrent pushes of the same layer from parallel jobs in one process using a shared InFlightTracker.

Common situations: Parallel layer uploads in one containerd client; retry storms re-entering push while the first attempt is still live; multiple controllers sharing an in-memory status tracker pushing the same manifest concurrently.

Related errors

• short write copying file
• ErrInvalidArgument
• ref %v: %w
• failed to get status: %w
• bufpipe: read/write on closed pipe

AI-assisted analysis of containerd/containerd@4246446a2b (2026-09-02). Data as JSON: /api/errors/a75151d71577e4ea. Report an issue: GitHub.