Topics
Publish one event to many queues, and subscribe to it from a Ruby job class.
A topic sends one event to many queues. Each subscription of the topic gets one copy of the event, as a normal job in its own queue, with its own retries and Dead set. Topics need server v0.3.0 or later.
Subscribe with topics: in anway_options:
class BroadcastMessage
include Anway::Worker
anway_options topics: %w[chat.message], queue: 'realtime'
def perform(message)
ChatChannel.broadcast_to(message['room'], message)
end
endPublish from any app:
Anway.publish('chat.message', [{ 'room' => 'general', 'body' => 'hi' }], id: 'msg-42')
# => #<data Anway::Client::Published event="msg-42", jobs=3, duplicate=false>Refer to Topics for the server rules.
Example: one event, apps in many languages
In the Anway demo, a Go app publishes to the product.created topic. A Rails app subscribes to it, and creates the product:
class Subscribe::CreateProductWorker < ApplicationSubscribe
anway_options event: 'product.create'
anway_options queue: 'products', topics: %w[product.created]
def perform(attributes)
Product.find_or_create_by!(external_id: attributes.fetch('id')) do |product|
product.name = attributes.fetch('name')
product.price = attributes.fetch('price')
product.status = 'published'
end
rescue ActiveRecord::RecordNotUnique
nil
end
endApplicationSubscribe is a plain class with include Anway::Worker.
- Each event of
product.createdbecomes oneproduct.createjob in theproductsqueue. - Other apps can also push
product.createjobs to the same queue directly. - Delivery is at least once. The external id makes a second run do nothing.
Subscribe
anway_options topics: %w[chat.message]
anway_options topics: %w[chat.message chat.edited], queue: 'realtime', retry: 3, reserve_for: 60- Each topic in
topicsmakes one subscription. - The job type of each copy is the
eventof the class. The default is the class name. - The copy has the
queue,retry, andreserve_forof the class. Without a queue, the copy goes todefault. topicsfrom more than oneanway_optionscall add up, with no duplicates. For a single value, such asqueue, the last call wins. So all topics of a class use the same queue. To send the events of one topic to another queue, use another class with another event.- A subclass does not get the
topicsor theeventof its parent.
When the subscription starts
The worker sends SUBSCRIBE for the subscriptions of all job classes when it starts, before its first fetch.
SUBSCRIBEis idempotent, so each start sends all subscriptions again.- A change of
retryorreserve_forapplies at the next start. - An event that you publish before a subscription exists does not go to that subscription. Start the subscriber worker one time before the first publish.
Queues of the worker
Without queues in config/anway.yml and without -q, the worker also fetches the queues of the subscriptions.
With a queue list, the worker fetches only that list. For a subscription queue that is not in the list, the worker logs a warning at start:
anway does not fetch queue realtime, so the events of topic chat.message wait there. Add the queue to queues.Add the queue to the list.
Publish
Anway.publish(topic, args = [], id: nil, unique_for: nil, custom: nil)argsis an array, as in a job. Every copy gets the same args. Args must be JSON types with string hash keys. Other values raiseArgumentErrorbefore the gem sends the event.idis the event id. With an id, a second publish of the same topic and id withinunique_forseconds makes no job, and the reply hasduplicate: true.- The default of
unique_foris 86,400 seconds (one day). Setunique_for: 0to turn off the check. - Without an id, the server makes one, such as
evt-…. There is no duplicate check. customgoes to each copy.- The reply is an
Anway::Client::Published, withevent,jobs(the number of copies), andduplicate.
Inside a database transaction, Anway.publish waits for the commit and returns nil. A rollback cancels it.
Anway.publish('chat.message', [{ 'body' => 'hi' }], id: 'msg-42', unique_for: 3600, custom: { 'source' => 'web' })Publish in bulk
Anway.publish_bulk(events)Anway.publish_bulk sends many events with PUBLISHB, 1,000 in each command. The server writes each command in one write. Each event is a hash with topic: and args:, and the options of publish: id:, unique_for:, and custom:. It needs server v0.5.2 or later.
results = Anway.publish_bulk(
products.map { |product| { topic: 'product.created', args: [product.as_json], id: product.id.to_s } }
)
results.select(&:error).each { |result| Rails.logger.warn("event #{result.event} refused: #{result.error}") }- The reply is an array of
Anway::Client::PublishResult, one for each event, in the same order, withevent,jobs,duplicate, anderror. - A refused event has
error. The other events still go. Anway.client.publish_bulksends one command, and raisesArgumentErrorfor more than 1,000 events.- In the test mode, each event is published as with
Anway.publish.
See Bulk publish.
Make the handler idempotent
The server adds topic and event to the custom field of each copy. event is the event id. Use it as the key to skip an event that already ran:
class StoreMessage
include Anway::Worker
anway_options topics: %w[chat.message], queue: 'cassandra', retry: 10
def perform(message)
event_id = job.custom['event']
return if Message.exists?(event_id: event_id)
Message.create!(event_id: event_id, body: message['body'])
end
endAn event with no subscription
The server does not lose an event with no subscription. It puts the event in the Dead set as one job, with the topic name as the job type, and the reply has jobs: 0. When you requeue this job, the server publishes the event again to the subscriptions of that moment.
Unsubscribe
The worker does not send UNSUBSCRIBE. The server keeps a subscription on disk, so copies go to the queue while the app is down.
When you remove a topic from topics, remove the subscription with the client. If you do not, its copies wait in the queue:
Anway.client.unsubscribe('chat.message', queue: 'realtime', event: 'BroadcastMessage')unsubscribe returns true. A missing subscription is not an error.
You can also subscribe with the client:
Anway.client.subscribe('chat.message', queue: 'realtime', event: 'BroadcastMessage', retry: 3, reserve_for: 60)Delivery
- Each copy is delivered at least once. Make each handler idempotent.
- Copies in different queues have no order between them. A queue keeps its own order.
- There are no filters. Each subscriber gets every event of the topic, and skips the events that it does not need.
Plain Ruby
Anway::Runner sends the subscriptions of all registered job classes. To send other subscriptions, give a list of Anway::Worker::Subscription:
runner = Anway::Runner.new(
queues: %w[realtime],
subscriptions: BroadcastMessage.subscriptions
)
runner.runAnway::Worker.subscriptions returns the subscriptions of all registered classes.
Problems
| Problem | Cause and fix |
|---|---|
publish returns jobs: 0 | The topic has no subscription. Start the subscriber worker one time, so that it sends SUBSCRIBE. The event is in the Dead set. Requeue it after the subscription exists. |
| Copies wait in a queue and do not run | The worker does not fetch that queue. Look for the does not fetch queue warning, and add the queue to queues. |
| Copies still go to a queue after you removed a topic | The server keeps the subscription. Call unsubscribe. |
Anway::ServerError at worker start | The server is older than v0.3.0 and does not know SUBSCRIBE. Update the server. |
| A handler runs the same event two times | Delivery is at least once. Make the handler idempotent with job.custom['event']. |
To test topics, refer to Testing.