« OpenSource contributions on your resume, a good thing? | Main | Startup beans in WAS 6.0 »

June 22, 2005

Total JMS Message ordering

A lot of customers think JMS products provide message order. I haven't found one yet that really does it.

Naively, you'd think that if a single publisher sends messages then a single receiver will process those message in the published order.

We make an MDB with a single listener thread and run it on a single server. If nothing goes wrong then this is possible. But, in the real world, sadly, things do go wrong.

TCP Sockets

First problem is a crashed client where the socket to the JMS server remains open. This can happen when a network breaks or the server physically crashes. The message(s) buffered on that client stay locked until the socket is closed. This can take a while depending on TCP KEEP ALIVE settings, if you haven't tuned this then it's likely 2 hours. In the meanwhile, if you restart your client on a different server and it asks for messages then it gets the next messages. Oops. Out of order. When the socket is eventually closed, the 'trapped' messages are unlocked and delivered to the client. Busted.

Indoubt Transactions

Next up is indoubts. If an application server was processing messages and then crashed or the box it was running on failed then we may have some indoubt transactions. These are transactions interrupted during the XA commit phase. Normally, these transactions will need to be recovered either by a peer cluster member (WAS v6 in 15 seconds) or by restarting that server on the same or a different box (minutes). During this time, if you started receiving messages on a different server then it will receive the messages after the ones trapped by the indoubts. Oops. Out of order. When the transactions are recovered, it may be that the transactions will be rolled back as a result of recovery. Now, the old messages in use by that transaction become available again and are delivered to your client out of order.

Edge Conditions

This kind of requirements comes up a lot but it's complicated to get right as you should be realizing from the above. There are also interesting edge conditions. JMS offers a time to live on messages. If it's not delivered within 5 tries then the message moves to a dead letter queue. But, we want in order messaging. When this happens, do we suspend the queue until an administrator looks at the suspect message and tells the JMS server put it back or discard it? Poison messages would have the same impact. Then, when we want message order do we mean maintain message order per individual publisher or for the order the queue received the messages. What makes the order? Commit time or publish time? It can turn in to a rat hole pretty quickly.

Summary

I don't think people realize what 'I must have message order' means in terms of implementation. You will need application logic to do this correctly for now. I don't know of any product that can reliably do this at the moment. Anyway, it's a heads up.

June 22, 2005 in J2EE | Permalink

Comments

Billy

If using XA

One of the limitations of using MDBs is that you cannot hook the Message Listener so as to test for in-doubts before starting the message processing loop. Using an AsyncBean would allow you to write this sort of test prior to listening.

However, the next issue is that XA is not particularly helpful when it comes to telling you which objects (queues / tables etc) are involved in the in-doubt. The AsyncBean could enquire whether there were ANY in-doubt messages (transactions), but it would not know which queues the in-doubt GETs and PUTs were on. So I assume it would have to wait (back off for a period) if any in-doubts were returned by the XAResource. Would this be better than waiting on the HA Manager (if indeed it is even possible to wait on the HA Manager – hint hint)

If not using XA

Then you don't have all the in-doubt business to deal with. Instead – as you say "You will need application logic to do this correctly for now". This means accepting that duplicate messages can occur, and therefore writing idempotent receivers. As one might argue that you should do this anyway (as upstream systems may have to re-send) – then why bother with XA in the first place (plus you save the 2(n+1) disk forces).

Perhaps we should be thinking about how to cope with duplicates, rather that trying to figure out how to workaround around XA.

BTW, as of Oracle 9.2.0.6, Oracle no longer block level locks rows involved in XA transactions. Before this, the demise of an app server that left an in-doubt could render remaining app servers in-operable as they would be unable to insert new rows (either the data block would be locked, or an index leaf block). Cool huh!

Andrew

Posted by: Andrew Ward | Jun 24, 2005 12:57:26 PM

Billy

btw (just to check), I am assuming that for n XA Resources, you end up disk forcing twice for each resource (prepare + commit), plus twice to the TM's tran log (so it knows the state of the tran). Hence 2(n+1). Without XA you end up forcing n times (but have to code the idempotent logic in 'return' for this saving). So with 2 resources thats 2(2+1)=6 disk forces @ 5ms (at least) each = 30ms? Does the combination of WebSphere, Oracle and MQ do better than this? I presume if you use the WebSphere 6 SIB (and back the messages to the same Oracle db), that the XA 1-phase optimisation is used - hence leaving you with half (3=tran log + db commit + log) the number of disk forces.

When using CMP with SIB on 6.0 then indeed we use 1PC when the CMP database amd SIB databases are the same. You have to use CMP for this optimization to work.

Posted by: Andrew Ward | Jun 24, 2005 1:11:15 PM

This is an excellent summary of the issues related to message ordering in all messaging systems. I've never had a short conversation about message ordering!

In addition to the points you mentioned, once you get into high availability, things get even more complicated. For instance, if you use multiple message queues on multiple servers to enhance availability, then order goes out the window as well.

There is no solution to this problem that works for all of the people all of the time. However, we've thought about this a lot at BEA, and JMS in WebLogic Server 9.0 includes a "unit of order" feature. This lets customers attach a "unit of order" name to each message. We then guarantee that for a particular unit of order, only one message will be processed at a time. So for instance, if a message is delivered for a consumer for unit of order X, and the server crashes after the transaction prepares, no other messages for unit of order X will be delivered to anyone until the transaction is resolved.

This way, you can guarantee absolute one-at-a-time message processing for all the messages on a queue by giving them all the same unit of order, but you'll slow down performance because everything will be serialized. Or, you can assign different units of order to different messages, according to business requirements, and get more parallelism if you tell us it's OK.

This doesn't solve all the message ordering problems in the world, and since it's new we don't really know how customers will use it. But we think it's an interesting new feature that addresses this issue in a new way.

Greg
BEA
(gbrail at bea.com, as if this will confuse the spammers!)

Posted by: Greg Brail | Jun 24, 2005 1:59:35 PM

But life is better on the mainframe

Laugh - I nearly cried when I read this little gem ;-)

Redbook - WebSphere MQ Queue Sharing Group in a Parallel Sysplex environment
http://www.redbooks.ibm.com/redpieces/pdfs/redp3636.pdf

Serialized applications (Page 17)

Any application working in strict sequence on a queue, and using shared queues for availability may be subject to the following race condition
1. Application instance 1 runs on queue manager A.
2. Either queue manager A, application instance 1, or both abnormally terminate, causing a rollback of the current unit of work.
3. An external monitor (e.g. ARM) notices the abnormal termination and reschedules instance 2 of the application on queue manager B in the same queue sharing group.
4. Application instance 2 restarts and performs MQGET from the queue BEFORE the rollback of the unit of work that was created by application instance 1 has completed. Instance 2 of the application may now have retrieved a message out of sequence.

... discussion of the MQCONNX function - only available on zOS ;-) ...

Therefore, serialized applications can be used to protect against the race condition described above, and to preserve the sequence in which messages are retrieved from a queue. By ensuring that both instances 1 and 2 of an application use the same serialization token, instance 2 can be prevented from starting until instance 1 has either successfully completed, or has been completely recovered.

Billy - IBM have spotted this race condition already, but it uses the zOS CF - which is not available on distributed platforms. Outside the JMS spec sure, but perhaps this use case should be added?

Andrew

Posted by: Andrew Ward | Jun 24, 2005 3:41:41 PM

I am facing an issue similiar to this discussion. Thought if someone could help? Its kind off urgent..

This issue is that I have two java threads writing on MQ (using JMS). The first thread writes messages 1,2,3 and then the second thread starts to write message 4 but before committing the transaction, the first thread writes message 5 and commits it. So when the second thread later on commits the message, its position on the queue is not behind message 5 but in between 3 and 5. i.e. the position at which it started to write and not the position at which it committed(i.e. after 5). So the position of a message on MQ is getting assigned at the time the thread starts to write on the queue and not when the message is committed. Can someone please suggest whether there is any MQ property that needs to set so that the position of the message from different threads is decided at the time of commit and not when it starts to write?

Will appreciate if someone can advice on this issue. Thanks.

Posted by: Manish | Sep 29, 2005 8:11:55 AM

What if you have a cluster of websphere application servers however you must ensure that particular trans are only processed after others, but how can you when a MQ workload management cluster round robins the messages to all the different WAS JVM's...

can you spin those two particualr trans to a broker to let them group those and then send them to the was instances?

message order is tricky..

Posted by: ryan | Jun 27, 2006 8:30:58 PM

I have an issue too with message ordering. The scenario is that we have a cluster consisting of two Websphere Application servers. Webmethods would be the sender of those messages which need to be processed by the Application servers. We have configured MDB's to listen to the queue in which Webmethods put their messages into.
Our application is message sequence specific. Say for example, I receive Student data in message no. 1 and Student marks in messag no.2. Now, since there are 2 servers, both the messages get processed concurrently causing the second message to fail as there will be no Student marks without the master data of the student.

So, is there any way in which we can make the servers process the messages in a sequential manner and , the messages get evenly distributed across both the servers so that one server is not heavily loaded than the other ?

Friends, Looking forward for your comments.......

Posted by: Rajesh | May 13, 2007 8:47:51 AM

Post a comment