instructure/canvas-lms · warning

dropped pandapub notification for #

Error message

dropped pandapub notification for #{channel}

What it means

CanvasPandaPub::Client#post_update publishes a notification by pushing a proc onto a background worker queue. When the worker's queue is full or shut down, push returns false (or nil) and the client logs this warning and silently drops the notification — the HTTP request is never sent to PandaPub.

Solutions

1. Increase the worker queue size or drain rate if drops occur under normal load
2. Check PandaPub endpoint health/latency — slow pushes starve the queue
3. Add retry/fallback (e.g., synchronous publish or re-enqueue) for critical notifications instead of accepting the drop
4. Monitor the warning rate to detect saturation before messages are lost in production

Example fix

// before
unless @worker.push(channel, proc { http.request(request, body) })
  @logger.warn("dropped pandapub notification for #{channel}")
end
// after
unless @worker.push(channel, proc { http.request(request, body) })
  @logger.warn("dropped pandapub notification for #{channel}; publishing synchronously")
  http.request(request, body)
end

# check worker health before publishing
unless CanvasPandaPub.worker&.alive?
  Rails.logger.warn('pandapub worker down; using fallback channel')
end

unless worker.push(channel, proc { http.request(request, body) })
  logger.warn("dropped pandapub notification for #{channel}")
  fallback_publish(channel, payload) # e.g., synchronous send or delayed job
end

When it happens

Trigger: @worker.push(channel, proc { http.request(request, body) }) returns falsy — the worker queue is saturated (max queue size reached) or the worker thread is not running (shutdown/not started) — while post_update attempts to publish to channel.

Common situations: Publishing bursts faster than the single worker thread can drain the queue; PandaPub endpoint slow/unreachable causing queue backlog; app shutting down while live-event updates are still being published; worker never initialized in some boot paths.

AI-assisted analysis of instructure/canvas-lms@1c9f0bb801 (2026-09-15). Data as JSON: /api/errors/6d61dc6fb5947da4. Report an issue: GitHub.

Appendix: source

Thrown at gems/canvas_panda_pub/lib/canvas_panda_pub/client.rb:86

    #
    #  channel - A String representing the PandaPub channel to post to. It should
    #    not include the application id, as that will be added by the library.
    #  payload - A Hash with the payload. It will be converted to JSON with JSON.dump.

    def post_update(channel, payload)
      path = "/channel/#{@application_id}#{channel}"
      request = Net::HTTP::Post.new(path, {
                                      "Content-Type" => "application/json"
                                    })
      request.basic_auth @key_id, @key_secret

      body = JSON.dump(payload)

      http = Net::HTTP.new(@uri.host, @uri.port)
      http.use_ssl = (@uri.scheme == "https")

      unless @worker.push(channel, proc { http.request(request, body) })
        @logger.warn("dropped pandapub notification for #{channel}")
      end
    end

    # Generate a token for subscribing to a channel.
    #
    # channel - A String with the channel to be subscribed to. Don't include the application
    #   id.
    # read - true if this token should allow reading from the channel.
    # write - true if this token should allow posting to the channel.
    # expires - A Date object specifying when the token should expire.
    #
    # Returns a String token.

    def generate_token(channel, read: false, write: false, expires: 1.hour.from_now)
      JSON::JWT.new({
                      keyId: @key_id,
                      channel: "/#{@application_id}#{channel}",
                      pub: write,

View on GitHub (pinned to 1c9f0bb801)