Networking frameworks revisited

Talking to a former colleague the other day I learned about recent changes in the world of NIO frameworks. The guy who led the MINA project joined JBoss and designed a new framework known as Netty. Previously, JBoss Remoting 3 was supposed to be based on MINA2.

Now the original MINA and Netty designer is on Remoting team as well so I feel I cannot make sense of all this proliferation anymore (even not taking into account Grizzly). It remains to be seen if some grand unification/survival of the fittest will ensue.

While looking at JBoss documentation I also found that they have a new NIO-related project. It makes me wonder how dead Java is if a single company can produce so much new stuff in a pretty obscure niche.

Terracotta at JavaOne

Today I went to see what was going on at JavaOne which is held a couple of blocks from my office. The Pavilion was not much different from last year so I ended up talking mostly to Terracotta guys and brainy Taylor in particular. I was asking different nasty questions about their approach and he was kind enough to talk to me for at least half an hour. Right after he talked to Brian Goetz himself.

My opening salvo quite naturally was about their messaging framework. I was essentially told that JGroups clusters of more than 4 servers had been widely known and Apache Tribes did not fit exactly and bug-fixing was insufficiently quick. From what I know the whole JBoss stack (at least in its JBoss AS 5 reincarnation) is supposed to depend on JGroups and so I am at a loss to reconcile such contradictory statements. Probably the key is the number of servers in a cluster.

At face value it means there is no open source reliable messaging framework capable of scalability beyond a couple of servers. Taking into account that pretty much anything which is not yet "in the cloud" is clustered nowadays that sounds odd. And it drives home the idea of reliable messaging as a truly challenging thing to make of production quality.

I learned more about there positioning as well. They are after the middle market of, roughly speaking, up to 50 servers in domains such as web application. Which I guess implies that JBoss is a competitor more than Coherence/Gigaspaces going after larger clusters in Finance.

An active-active L2 server configuration is expected by the end of the year although the common belief is that the 10 seconds required to switch to a backup server now are tolerable. From how I understood it they are planning to send separate updates to both L2 servers instead of multicast or replication between the two. I might have misconstrued something though.

We talked about their paradigm a little bit. I admit to being rather uncomfortable with it because they are the only company I know literally exploiting the conceptual similarity of concurrent and distributed systems (i.e. CPUs sitting on the same bus differ from servers in a cluster only by communication delays much more pronounced in the case of a LAN). It is so different from pretty much any product (exposing a real API in terms of actual interfaces in, say, JCache as opposed to delimiting transactions with monitorenter/monitorexit pairs) that either they have invented the best thing since sliced bread or they are likely to fail as mavericks. They might as well be the next "the network is the computer" after all.

The foundational paradigm of Terracotta as a distributed JVM (complete with a DGC) evokes the same kind of argument as JVM used to ten years ago. Back then the idea of Java performance comparable with C++ was ridiculous although it was said at the very beginning that JIT-style dynamic optimizations would do the trick one day. It looks like JVM guys have pulled the trick after all so this lesson may have significant implications for Terracotta.

As an example, it can detect that a particular instance is used exclusively by one L1 server and transfer lock ownership to that L1 server from the central L2 host and so effectively avoid using distributed locking. As a result we have a sort of buddy replication (between L1 and L2). Like a JVM silently eliminating synchronization in a sequential program. Theoretically neat :)

One thing I can say safely is that in contrast to many companies in this field Terracotta is not afraid to share its source code. They do not pretend like Coherences of this world do that someone can steal anything from the code base and ruin their empire (anybody heard of a new M$ after the windows source code was leaked on the net?). As a developer I believe that code quality says a lot about the corresponding system (not to mention things one can learn from a large successful system) and I applaud Terracotta for their bravery.

NIH and reliable messaging

We are planning to use Terracotta as our distributed caching infrastructure. I always thought this framework was rather odd - pretty much every one else follows the usual "JGroups-like" approach of providing an API manipulating with such abstractions as Channel/Group/Member or the recent Map/Reduce-inspired analogues.

For historical reasons I am curious about different transport-level frameworks. Last year I was looking closely at JBoss and was amazed by their indiscriminate use of a few overlapping technologies such as JGroups and JBoss Remoting (which they were planning to migrate to MINA anyway). Coherence has its ownTCMP protocol and so does Gigaspaces. And a new messaging standard will likely influence the field. I certainly understand why developers are so inclined to re-implement the same functionality. Although there are well-known patterns for designing such a component there is a lot of hard-core fun left in all the intricate details of harnessing NIO and multithreading.

The flip side is that this kind of software is extremely difficult to get right. Even years later there could be elusive bugs (I heard JGroups still fails when there are a few dozen severs) . From my experience hunting them even in a 4-server configuration can be nightmarish to put it mildly. So I guess for those who do it implementing such a framework is a major professional self-esteem booster.

Naturally, one of the first things I was looking at in the Terracotta code base was their clustering. Curiously enough, they borrowed their transport layer - I guess they were having enough fun with higher-level state clustering. But the layer itself proved my point again - it's the framework used in Tomcat 6. So ASF seems to be following the same route of developing multiple overlapping technologies as JBoss is.

Consistent Hashing

An interesting and conceptually simple approach to choosing what to put on a new cluster member or where to move the state of a failed one.

Distributed algorithms

Distributed algorithms are a fascinating but often overlooked area in which academic research meets real-life challenges. Well-known practical applications such as the JGroups framework tend to simplify (for performance reasons obviously, certain levels of consistency are prohibitively expensive in terms of network traffic or latency) the approaches suggested by academicians. On the other hand, virtually all such frameworks are firmly rooted in hard-core research (e.g. Bela Ban was influenced by research done in Cornell where Ken Birman seems to be the founding guru) and you won't go far on common sense and re-invented wheels.

In mainstream development layers upon layers of middleware protect programmers from the challenges of real engineering. Quite few are lucky enough to closely work with JGroups-level components and even fewer use NIO and MINA-like frameworks (BTW MINA's lead developer has just been hired by JBoss) to build infrastructure of their own. In my experience, even among CS graduates not everyone has a good grasp of this field. Once upon a time I struggled mightily to learn basics of reliable messaging and its internals. As a result, I have been keeping my eye on the literature but surprisingly enough for our age of grid computing there are only a few sources catering to practicing engineers. Most CS books tend to jump to the theoretical aspects and never come back to something more practical.

So far I can truly recommend only a couple of books paying attention to real code.

• Concurrent and distributed computing in Java - a very good introductory book covering both essential concurrency issues and distributed algorithms. The author did a very good job of actually explaining many concepts and their simple Java implementations (the code is probably not perfect but it makes so much easier to grasp the ideas) without overwhelming the reader with gratuitous mathematical rigor. The book consists of chapters of roughly equal size which makes it possible to read in manageable chunks. Each chapter refers to selected original articles and so explains the history of the field, again very succinctly.

• Introduction to Reliable Distributed Programming - this book is based on a real-world messaging framework and as such provides both an in-depth discussion of algorithms and their implementation. The only thing I can say is that I wish there were a comparable book on JGroups which is arguably the leading Java framework.If I live long enough to overgrow those two books I will probably read also Design and analysis of distributed algorithmsbut I am not entirely convinced it will happen before I retire :)

As a side-note, I found the JGroups source code rather messy. I frequently saw places where refactoring was long due. Consequently, I doubt that framework is a good educational instrument at least for low-level mechanics especially if you do not have a pretty good idea of what exactly to expect. My UML sketch of TCP/UDP stacks looks rather unwieldy but they say most highly useful pieces of software look puzzling inside.