Monday, June 25, 2007

Asynchronous Messaging with Rails

When asked how to integrate with other applications, the vast majority of Rails developers would answer "REST". And when they answer REST, they almost always mean synchronously. Now I have nothing against REST per se, but I will always favour asynchronous communication over synchronous. An asynchronous messaging solution such as JMS has well-documented advantages such as:

  • The message producer can "fire and forget", sending the message and then moving on to more important work.

  • The mechanisms required for reliability are relatively simple in the producer and consumer, and the more difficult tasks (such as persistence) can be handled in the message broker.

  • It is easy to add redundancy into the messaging infrastructure


For further reference, a good book on the topic is Gregor Hohpe's Enterprise Integration Patterns

On a number of occasions I've started out with synchronous messaging, only to say to myself "I really want this to be reliable". So I fire off an http request and wait for a 200. If I don't receive a 200 after a certain amount of time, then I try again. I'll repeat this for a few times, then I'll give up, and log the failure somewhere.

Then I say to myself "It would be really nice to have this re-trying out-of-process". So I add another process that receives a request for a message to be sent, sends the message, and performs any re-trying that is required. And all of a sudden, I have asynchronous messaging. Now, there's nothing about REST that prevents asynchronous messaging, but I rarely see anyone from the Rails community talking about anything but synchronous messaging. So I thought I'd talk about how we've solved the problem on our project. We've used the ActiveMessaging plugin, originally written by some people from Thoughtworks. I'll demonstrate ActiveMessaging here, and then over the next week or two I'll contrast it with a JMS JRuby solution I've been playing around with.

Let's say we have two Rails applications - a Customer Management application and an Order Management application. The Customer application is responsible for managing all things to do with Customers - their personal details, any communications between our business and the customers, and the customer's user preferences. The Customer Management app has a rich view of a customer with fields such as:

  • Name

  • Address

  • Telephone Number, etc.


The Order Management application needs to associate Orders with Customers, so that even when the Customer application is down for scheduled maintenance, orders can still be taken. The Orders app has a simple view of a Customer, with just a name and an id.

We will make the Customer application responsible for creating, updating and deleting Customers. Whenever any of these actions are performed, the Customer application notifies the Orders application with a message. The message indicates the type of action performed (create, update, or delete), and the id of the customer, so that the two 'views' of the Customer object (in the two different applications) can be kept in sync. The Orders application then creates, updates, or deletes its view of the customer according to the type of message that is sent.


First, we will need to setup an ActiveMQ Server. I've set it up locally on my machine using these instructions (I used the Linux instructions for my Macbook Pro):

http://activemq.apache.org/getting-started.html

In both of our Rails applications we will need to install the ActiveMessaging plugin:

script/plugin install http://activemessaging.googlecode.com/svn/trunk/plugins/activemessaging

ActiveMessaging provides the ability to send and receive messages. Our Customer application will send messages, and our Orders application will receive them.

For the purposes of this discussion, I will only show the 'create' messages, though the process would be very similar for 'update' and 'delete'.

Customer application


Our Customer application has a Customer object that inherits from ActiveRecord::Base:

class Customer < ActiveRecord::Base
end


It has the following fields:

:name, :string
:address, :string
:telephone_number, :string
:created_at, :date_time
:updated_at, :date_time


Our message will be a YAML serialized message of the following simple data object:

class CustomerPayload
attr_accessor :id, :name

def initialize(params)
@id = params[:id]
@name = params[:name]
end
end


We use a rails observer to observe the 'create' event of the Customer object, construct a CustomerPayload object, serialize it, and send it via ActiveMessaging:

require 'activemessaging/processor'
class CustomerObserver < ActiveRecord::Observer
include ActiveMessaging::MessageSender
observe Customer

publishes_to :customer_queue

def after_create(customer)
payload = YAML.dump(CustomerPayload.new(:id => customer.id, :name => customer.name))
publish :customer_queue, payload
end
end


To configure connection to the ActiveMQ server, I'll need the following configuration files in both of my rails applications:

#config/broker.yml
development:
adapter: stomp
login: ""
passcode: ""
host: localhost
port: 61613
reliable: false


#config/messaging.rb
ActiveMessaging::Gateway.define do |s|
s.queue :customer_queue, '/queue/Customer'
end


And to get the observer to work, I'll need this line in environment.rb

# Activate observers that should always be running
config.active_record.observers = :customer_observer


Orders application


For consuming messages, ActiveMessaging provides a directory under app called 'processors'. In this directory of my Orders application I place my CustomerMessageProcessor:

#customer_message_processor.rb
require 'processors/application'
class CustomerMessageProcessor < ApplicationProcessor

subscribes_to :customer_queue

def on_message(serialized_message)
customer_payload = YAML.load(serialized_message)
customer = Customer.new(:name => customer_payload.name)
customer.id = customer_payload.id
customer.save!
end

end


This class will need access to the same CustomerPayload object as the Customers application for deserialization.

In ActiveMessaging, a poller is used to poll the queue and trigger the on_message method shown above. We'll need to start the poller in the Orders application using the following command:

script/poller run

And that's it. If both the Orders and Customer applications are running, and have access to the ActiveMQ server, then when we create a Customer in the Customer application, it should show up in the Orders application a few moments later.

15 comments:

James Strachan said...

Great post; I've added it to the ActiveMQ articles page: http://cwiki.apache.org/ACTIVEMQ/articles.html

Incidentally mentioning Enterprise Integration Patterns; have you seen this?

http://activemq.apache.org/camel/enterprise-integration-patterns.html

its an easy way to use EIP using a Java DSL.

It might be interesting to try out a Ruby DSL version of Camel so folks can write the routing rules in JRuby?

kookster said...

just saw this via jon tirsen, I'm adding a link to it from the activemessaging wiki as well.


- Andrew Kuklewicz

Shane Harvie said...

Thanks James. Will be sure to check out camel.

Ola Bini said...

Hi, have you noticed that ActiveMessaging have support for JMS when running on JRuby, nowadays? =)

Shane Harvie said...

Hey Ola, that sounds awesome. I'll check it out.

Ramesh said...

Hi james , i tried out this as a tutorial and i have a doubt that where to place that Payload class in my application directory structure. and while i have started that poller it shows me an error that gems required. plz notice me that need to install any gem for this .. i'm expecting ur reply
Thanks in advance
Ramesh , India

Ramesh said...

Hi james, i found that stomp is the gem required and i installed that but again here's a doubt that what i should place in the login and passcode of my broker.yml file ? Where to place that payload file

Plz help me James
Thanks
Ramesh, India

Shane Harvie said...

Sorry for not replying sooner. (It's shane, by the way). Your login and password are for the ActiveMQ server - I don't have a login set up for the server, so I've just left them blank. Your payload object can be put wherever you like - it just needs to be accessible to the Customer observer (in the customer application) and the CustomerMessageProcessor (in the orders application). If you put it in one of the folders that are automatically loaded by rails (eg models) and name the file after the object (eg customer_payload.rb) then you shouldn't have to explicitly require it. But you do have to have the payload file in both projects. To reduce duplication, we've created a rails plugin that has our shared objects (such as the payload). Each project installs the plugin to get the common code. We make the plugin an svn:external so that we reduce duplication. Hope that helps...

Tom said...

It is probably good to keep in mind that ActiveMQ does not currently support any sort of authentication for Stomp clients yet.

So even if you configure a username and password for your queues in ActiveMQ, a Stomp client can connect to any queue with any username and password.

Avant said...

Tom, seems like the 'missing' security implementation that you speak of is gonna be available from activemq v 5.1 onwards.

http://activemq.apache.org/stomp.html

Nice post btw!

Avant

docsharp01 said...

Great blog with lots of useful information and excellent commentary! Thanks for sharing. DSL internet service providers

Nadav said...

Thanks for the post. It's great.

I am looking for some mechanism to implement reliable communication between browser and server - so that the browser will resend requests upon communication failure.

I can write my own ajax calls with 'onfailure' clauses, however I am looking for some kind of js framework for that purpose.

Do you guys know of such a thing ?

Thanks
Nadav

Shane Harvie said...

Hi Nadav, I don't know of anything off-hand. But that doesn't mean it doesn't exist. The usual suspects like scriptaculous and prototype don't offer anything?

Overbryd said...

Hy,

this is really a nice tutorial. But you should check your serialization:

http://www.pauldix.net/2008/08/serializing-dat.html

YAML has really poor performance.

Cheers,

Lukas

Filip said...

Hi,

I have tried the tutorial, but I am facing some problems with deserializing the objects with YAML.load()

In the on_message() method of the processor if I print the objectto console via puts it seems to be allright, but as soon as I extract the object with YAML.load() and then access the attributes it doesnt work any more. I tried the dump and load in ruby console and there was no problem also. Only after reading from MQ it can't be deserialized.
Anybody knows a solution for this problem?
Best regards, Filip