« Openspaces is more testable, not true. | Main | ObjectGrid brings CEP and XTP together »

August 25, 2007

XTP: Pat Helland, Square Architectures and Distributed Transactions evolve

It's becoming more and more clear than conventional tiered architectures have a lot of trouble scaling. Conventional tiering uses a triangular shape. If requests come in from the left then you tend to see multiple tiers moving right until you hit the data source on the right. The tiers are widest in terms of number of processors on the left where the web servers are and then the number drops steadily as we move right closer to the data source. This is why these architectures won't scale indefinitely. People attempt to help scalability by including data source offloading technologies such as caching on the left hand tiers, off loading the data source and allowing it to support more processors in front of it. These triangular architectures will not support XTP (extreme transaction processing).

XTP needs square architectures. A square architecture is one where the right hand side gets wider as the left hand side gets wider. The whole architecture scales from left to right. This is necessary to maintain growing transaction rates with constant transaction times. All resources needed by the system must have a way to scale out.

Products such as ObjectGrid can provide a framework in which to host XTP applications. The applications run using a partitioned model. There is no two phase commit. Transactions span only a single partition. A partition provides a unit of scalable state management for the application. This state includes both relational data as well as messaging based data. ObjectGrid unifies both types to a single data model and API. Messages are data and vice versa, there is no distinction.

If a transaction needs to change data across partitions then it's done in multiple steps. The first step is a transaction on partition A. Partition A sends a message to partition B. This message is sent from A to B using single phase commit. As a result, B may receive the same message multiple times and B must be able to handle this. B processes the message asynchronously.

ObjectGrids job is to provide the infrastructure for hosting and placing the partitions, providing fault tolerance and data integrity and support applications using the above application architecture. The application developers role is to architecture the system as a set of partitions and write the business logic using a one transaction per partition model with messaging hooking up multiple partitions. The developer doesn't need to worry about the infrastructure needed to host the application; this is ObjectGrids job in this.

Pat Helland from Amazon wrote an excellent paper on this type of architecture. ObjectGrid is designed around and embraces this type of architecture when hosting XTP applications. Pat describes keyed entities. Transactions cannot span entities. Logical transactions spanning entities happen instead with a series of independent entity scoped transactions connected using idempotent messaging. This approach expressed in ObjectGrid terms has an entity equal to a partition. An entity is an object graph with a hashable root object. Applications route messages to the root object using a hashing approach. ObjectGrid provides relational management for the entities and a messaging mechanism for communication between partitions.

As companies move to a real time rather than a batch enterprise architecture then we'll see XTP start to spread from telco and financial companies today to more and more enterprises as they try to make their business models real time. This is necessary for companies to stay competitive. Waiting for batch jobs to complete etc was fine several years ago but won't help them be competitive moving forward. No one wants to wait. Everybody wants everything immediately. Conventional triangular architectures will not cope as companies move to this model. The front end of these applications will use square architectures and XTP patterns and middleware.

There will still be conventional back-ends for reporting hosting on data ware houses, at least until XTP data stores evolve to support distributed reporting etc. These back-ends will basically be asynchronous replicas of a subset of the state in the XTP system.

To summarize, as we move towards real time enterprises, XTP will start to take hold and applications will need to be designed around new architectures as described here in order to scale effectively, middleware will need to evolve to support these new architectures. These types of architecture and exactly why we’re building ObjectGrid.

August 25, 2007 in XTP | Permalink


Post a comment