« Experience with wiki for documentation development | Main | Phantom Fighter lives again »

September 12, 2005

ObjectGrid as a high performance distributed data integration layer

The ObjectGrid as a data abstraction layer seems pretty useful. Applications can be written to use the ObjectGrid APIs to access data. Data thats already in the ObjectGrid is returned back to the application quickly. Data that isn't in the ObjectGrid is requested from a Loader plugin.

Each Map in an ObjectGrid can have its own Loader. Map A might use a Loader with a CICS adapter, Map B may use a JDBC and Map C has no Loader (it's just a local cache). The Loaders job is to pull data from a backend when asked by the ObjectGrid. The Loader also receives the changes for a given transaction so that those changes to be applied to the backend. Loaders can be written to get data for any backend with a Java API transactionally. Examples would be:

  • JDBC
  • Hibernate through JDBC
  • JDO
  • CMPs
  • JCA Resource adapters to mainframes or CRM/ERP applications
  • RMI to other servers (which may also use an ObjectGrid).
  • ESB/SOA products that offer access to the backend through a Java API.
  • Any backend with a Java API for query and update.

When a given JVM pushes changes to the backend then this information can be replicated to other JVMs with ease using either JMS, the HAManager or any message transport of the customers choice. The addition of the ObjectGrid to this solution can significantly lower the cost of the data tier (hardware + licenses) especially if there is a lot of query load against it.

Asymmetric tiered caches can also be built. This is a scenario where, supposed the Web tier used Tomcat and sent RMI calls to a WebSphere cluster when ever it needs to persist data or obtain data from a backend. The WebSphere cluster then accepts these RMI calls and sends either queries or updates to a backend. It's asymmetric in the sense that only the WebSphere ObjectGrid can actually access the backend. The 'front end' cache in the Tomcat server just forwards its Loader traffic using RMI to the WebSphere layer.

So, we can introduce the ObjectGrid here. The ObjectGrid on the Tomcat side has Loaders which send the RMI calls to the WAS side. The WAS side has Loaders which access and update the backends. This is a multi-tiered cache. Now, the Tomcat side needs to access the WAS side less frequently due to the cached offered by the Tomcat ObjectGrid.

The WAS side in turn now accesses the backend less on queries as many results are cached. When the WAS side updates the backend, the set of changes in that transaction can be replicated to the other WAS Objects AS WELL AS the Tomcat ObjectGrid. This gives us a mostly up to date cache.

This kind of scenario can be used to significantly lower the cost of deploying a solution as well as improve its scalability enormously. We can effectively move MIPs from the most expensive tier, the data tier, to less expensive tiers as we move up closer to the client.

September 12, 2005 | Permalink


TrackBack URL for this entry:

Listed below are links to weblogs that reference ObjectGrid as a high performance distributed data integration layer:


Post a comment