December 20, 2007
Major update of ObjectGrid just shipped
We just shipped ifix 3 for ObjectGrid which despite being packaged as an ifix is basically a major release for ObjectGrid. We added some amazing new features in this build including:
- Spring Container Managed Transaction Support
- Spring Bean Factory integration
- Support for zone or multi-data center replication (industry first)
- Zones allow customers to easily separate primaries and replicas spacially from each other for better high availability.
- HTTP Session support with multi-data center support
- Better integration with WebSphere
- FIFO style access to query result sets (use Maps as queues)
You can read more about it on the wiki. The update can be applied to XD DataGrid installs and then copied to WAS V6.0.2 or customers using J2SE.
You can find it here.
December 18, 2007
HStore at MIT, very interesting and very similar to ObjectGrid
This paper written about a prototype database at MIT called HStore provides a lot of the architectural principles behind ObjectGrid's design. We're very close to whats described in that paper. If you want to use ObjectGrid as an in memory database for XTP applications then you need to have a database schema that is basically a partitioned object graph. This means it's an object graph with a single root and you can partition the root objects using the key to distributed data. Examples are "Customers have orders, orders have order items". Another example is "Portfolios have positions, positions have fills". We can partition using customer and portfolio. Applications transactions only involve records within a single graph.
This is described in the paper as a constrained tree application or CTA. CTAs are what run best on ObjectGrid when you use it for XTP style applications. The paper speaks of CTA applications having transactions and transactions being coded as a 'stored procedure'. ObjectGrid uses its DataGrid agents to implement stored procedures and indeed these will offer the best performance.
Anyway, take a look, it's very interesting. There are a couple of things that we're not doing yet but the overall approach is very similar to how ObjectGrid works today.
December 15, 2007
Great time at the Spring Experience
I just returned from the Spring Experience where I'd been invited to speak. I did a session on the new data center support for ObjectGrid as well as an overview of an XTP application built using J2SE, OpenJPA, ObjectGrid and Spring. The conference was very well attended, I think there were around 600-700 attendees which is great. I like nothing better than making a single trip and being able to communicate to lots of customers with that single trip :)
I met a lot of customers at the sessions and got a very good response on the new Spring integration features in ObjectGrid V6.1 ifix 3 which should be public any day now.
I had a good chat with Rod and I think we'll be doing more to make ObjectGrid work even better with Spring moving forward.
The batch programming sessions were very full which shows that just because something is viewed as old tech, i.e. batch, it's very popular. I had people ask about ObjectGrid with Spring WebFlow (works fine), ObjectGrid with Spring native transactions (works in ifix 3) and questions on building ObjectGrid applications that use Spring services and configuration capabilities. I'm going to try to do more blog entries with examples/specific use cases and I'll work on putting them in to the official wiki also.
I bumped in to Floyd from infoq and recorded a video interview on virtualization so that should interesting for some, I hope. Floyd and I go way back and we'll probably work together on some content around multicore for infoq in the near future.
Bob Lozano was there and fun to hang around with as always. He works at Appistry which is a very cool product for organizing grids and running workflow style tasks with existing code on those grids. ObjectGrid would work very nicely with it. I had customers at one session ask me if ObjectGrid competed with Appistry and, of course, nothing could be further from the truth, they are very, very complimentary technologies. Appistry would be a great solution to provision ObjectGrid applications and submits jobs to them or use them as part of other jobs.
I met some old friends from my consulting days and that was great also. The hotel was very nice and the conference rooms were nice. The event seemed well run and they provided lots of cool prizes to attendees also. iPods, iPhones etc.
Unfortunately, I couldn't stay for the scalability expert panel which was on Friday at 9:30pm. Gigaspaces Nati was there as well as Patrick from Oracle Coherence so that would have been fun to attend but I had a personal appointment on Saturday and had to fly out right after my session on Friday.
I'm very much looking forward to doing more of these in the future.
November 29, 2007
Demo of creating JavaEE applications hosting ObjectGrid
ObjectGrid is designed to work in J2SE environments as well as run within WebSphere application server. WebSphere clusters can be easily used to host ObjectGrids and then clients can access the data stored within that grid. Clients can be applications within the same cluster using servlets or EJBs, J2SE clients or J2EE clients in different clusters or cells. This allows large datasets to be stored in the total memory of all cluster member JVMs. This can be used for XTP type applications, network attached caches or to host HTTP sessions. Several clusters can also be used as single grid even if the clusters are in different cells. This allows advanced topologies that we can go over in future blog entries.
This is a set of four screen casts. I recorded them at 1280x1024 so hopefully it fits on most screens. The intent is to show how easy it is to create a JavaEE EAR file and then add a couple of xml files to it which will cause ObjectGrid to start a container within the application server JVM the EAR is eventually deployed to. The end result is it shows how ObjectGrid can be embedded in WebSphere to provide a complete environment for hosting an objectgrid within a WebSphere cluster.
The demo uses a WebSphere V22.214.171.124 ND application server that has XD DataGrid (ObjectGrid) installed on top of it. The first video shows how to create the EAR in Rational Application Developer V6.0 and then export the EAR to the file system.
The next video shows how to start an ND cell on my laptop, define a single node. The cell topology on my laptop will be a single dmgr, a node agent and two cluster member JVMs. I'm using a 1GB RAM virtual machine running Windows XP to host the demo.
Next, we will show how to create the cluster and add two cluster members. We will then deploy the EAR file that we exported in the first step to the newly created cluster.
Finally, we will start the cluster members on the command line and show how to look at the SystemOut.log file from the cluster member to look for ObjectGrid messages that show that the grid is working and fails over correctly.
November 11, 2007
Network attached cache/memory for lower consolidation/virtualization costs
Many companies are consolidating their servers today. Rather than watch servers go underutilized whilst running a single application, they are consolidating multiple applications on to fewer boxes to save money and increase the average utilization of the servers.
This increases the CPU usage but memory can be a problem. Memory is not cheap. Even on blades, maxing out the memory capacity can be very expensive. Running multiple Java applications in their own JVMs on a single box uses up memory fast. Each one can take a GB or more of memory. If the application uses 5% of CPU on average then consolidating can push up the CPU usage but you may not be able to achieve 40-50% CPU usage because you may run out of memory first. 10 JVMs can easily take 5-10Gb of memory when running a biggish application. If the applications use caching then the memory usage may be considerable and result in either very expensive memory costs or fewer applications per server which works against a consolidation strategy.
Network attached caches such as ObjectGrid can help reduce memory costs considerably and thereby allow a successful consolidation strategy without incurring unreasonable memory costs. A cluster of ObjectGrid servers can run on a set of machines and provide a large network attached memory based cache for applications. This means that if an application needs a Gb of cache to run well then rather than have a Gb of memory per JVM, we can have a Gb of memory in total across the cluster which can result in a massive saving. If you have 20 machines then you may be able to run 20 objectgrid servers and use 500MB per box and still have 10Gb of network attached cache available for applications to use. This allows you to run more applications per server because the per application memory load is significantly lower.
ObjectGrid can easily pay for itself quickly when used in this manner. It cuts memory costs massively and allows more consolidation of applications on fewer boxes.
November 08, 2007
Caching read/write data using a network attached cache
This audio podcast discussing why traditional caching approaches fail when customers attempt to cache read/write data and how newer caching technologies like ObjectGrid, through the use of network attached cache architectures, allow read/write data even at high update rates to be effectively cached by a cluster of applications.
You can easily subscribe to all audio podcasts using iTunes by clicking on this link.
Network attached caches or memory - Audio Podcast
This is an audio pod cast discussing using ObjectGrid as a network attached cache rather than the traditional view of caches as an in memory cache. A network attached cache has numerous advantages over an in memory cache in that it's populated by all clients concurrently and what can been cached by one client is immediately available to service cache requests from its peer clients.
There is also no stale data in that data in a network cache is stored in one location and all clients seen the same record as there is only one copy in the network.
A near or local cache can be specified to filter cache requests to the network cache to improve performance when some smaller subset make sense to keep close to a particular client. Staleness can be handled using a local evictor or optimistic locking or row versioning to detect if the locally cached data was changed by a peer client while it was cached. The client can then invalidate the local entries and retry the transaction.
If you want to subscribe to audio podcasts using iTunes then click on this link.
October 01, 2007
Event Stream processing with ObjectGrid V6.1
ObjectGrid V6.1 shipped with a substantial upgrade in the streaming event space. We added the capability to define changes to a Map as a stream of insert/update events and then create derivative streams based on these fundamental streams. The derivative streams can do aggregation, filtering, processing of events.
The streaming capability allows events from a single partition to be combined/processed. Layers of event processing can be used to further aggregate these per partition results using the DataGrid APIs to aggregate the resulting derivative streams in parallel.
If you would like more information then head over to our wiki to read about it.
September 25, 2007
Spring WebFlow and ObjectGrid
Spring Webflow looks like a very nice Apache alternative to the LGPL competitors in the same space and looks to be extendable to a non web specific lightweight conversation manager also which is cool.
I had a conversation with Keith Donald yesterday about it and it looks like it should work with ObjectGrid backing the flows out of the box since it usually stores the flow state as an attribute in the users HTTP Session. ObjectGrid already provides a HTTP Session Manager servlet filter to attach to web applications persisting HTTP sessions to an ObjectGrid. If the web application is using flows then those flows will be persisted to the HTTP Session (and therefore the grid) along with the other application user state. So, there is nothing special to do for Webflow users when running in an ObjectGrid backed session environment.
I guess implementing an independant ObjectGrid backed flow repository would be interesting also so that users can use it to store conversations that happen outside a web context and these could be checked the next time they log in without needing to write the flows to a database or similar non scalable persistent backend.
Anyway, Webflow 1.0 looks pretty cool and Webflow 2.0 shows a lot of promise also.
September 21, 2007
Podcast available on iTunes now
I'm going to try doing an audio podcast as well as some video ones showing us discussing various scenarios. You can subscribe to the podcast using iTunes here.