We are planning to use Terracotta as our distributed caching infrastructure. I always thought this framework was rather odd - pretty much every one else follows the usual "JGroups-like" approach of providing an API manipulating with such abstractions as Channel/Group/Member or the recent Map/Reduce-inspired analogues.
For historical reasons I am curious about different transport-level frameworks. Last year I was looking closely at JBoss and was amazed by their indiscriminate use of a few overlapping technologies such as JGroups and JBoss Remoting (which they were planning to migrate to MINA anyway). Coherence has its ownTCMP protocol and so does Gigaspaces. And a new messaging standard will likely influence the field. I certainly understand why developers are so inclined to re-implement the same functionality. Although there are well-known patterns for designing such a component there is a lot of hard-core fun left in all the intricate details of harnessing NIO and multithreading.
The flip side is that this kind of software is extremely difficult to get right. Even years later there could be elusive bugs (I heard JGroups still fails when there are a few dozen severs) . From my experience hunting them even in a 4-server configuration can be nightmarish to put it mildly. So I guess for those who do it implementing such a framework is a major professional self-esteem booster.
Naturally, one of the first things I was looking at in the Terracotta code base was their clustering. Curiously enough, they borrowed their transport layer - I guess they were having enough fun with higher-level state clustering. But the layer itself proved my point again - it's the framework used in Tomcat 6. So ASF seems to be following the same route of developing multiple overlapping technologies as JBoss is.