What causes "failed to publish '%v'" in apache/beam?

Read the wrapped underlying error to see the Pub/Sub API cause (permission, quota, not found).

failed to publish

Error message

failed to publish '%v'

What it means

pubsubx.Publish publishes each message to a Google Cloud Pub/Sub topic and blocks on the result. When a message's publish future returns an error, it is wrapped as "failed to publish '%v'" so the failing message payload is identified.

Solutions

Read the wrapped underlying error to see the Pub/Sub API cause (permission, quota, not found).
Verify the topic exists and the service account has roles/pubsub.publisher.
Increase the context timeout or retry with backoff for transient unavailability.
Confirm the data payload encodes correctly; check for oversized messages (>10MB).

Example fix

// before
sub, err := pubsubx.Publish(ctx, client, topic, sub, messages)
// after (ensure valid ctx with timeout and permissions)
ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
defer cancel()
sub, err := pubsubx.Publish(ctx, client, topic, sub, messages)

Defensive patterns

Strategy: try-catch

Validate before calling

if topic == nil || topic.String() == "" { return errors.New("topic must be initialized before publishing") }

Try / catch

sub, err := pubsubx.Publish(ctx, client, topic, sub, messages)
var apiErr *googleapi.Error
if err != nil {
    if errors.As(err, &apiErr) {
        // inspect apiErr.Code: 403 permissions, 429 quota, 404 topic
    }
    // retry transient failures with backoff
}

Prevention

Grant roles/pubsub.publisher to the publishing service account.
Use a context with a sane timeout and retry policy.
Validate topic existence with client.Topic(...).Exists(ctx) before publishing.

When it happens

Trigger: Calling pubsubx.Publish (or TestPublish) when pubsub.Client Publish fails for any message — e.g. topic doesn't exist, quota exceeded, permission denied, or context canceled while waiting on Get(ctx).

Common situations: Wrong GCP project or topic name; missing pubsub.publisher IAM role on the credentials; publishing after ctx deadline exceeded in slow integration tests; Pub/Sub service outage.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/36b41b7a649b8d20. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/util/pubsubx/pubsub.go:115

}

func publish(ctx context.Context, client *pubsub.Client, topic string, messages ...string) (*pubsub.Subscription, error) {
	t, err := EnsureTopic(ctx, client, topic)
	if err != nil {
		return nil, err
	}
	sub, err := EnsureSubscription(ctx, client, topic, fmt.Sprintf("%v.sub.%v", topic, time.Now().Unix()))
	if err != nil {
		return nil, err
	}

	for _, msg := range messages {
		m := &pubsub.Message{
			Data: ([]byte)(msg),
		}
		id, err := t.Publish(ctx, m).Get(ctx)
		if err != nil {
			return nil, errors.Wrapf(err, "failed to publish '%v'", msg)
		}
		log.Infof(ctx, "Published %v with id: %v", msg, id)
	}
	return sub, nil
}

View on GitHub (pinned to 12126d8942)