Transactional saga in Scala IV

Part V : Mapping future

.. in which Japanese researchers and a mad Russian scientist continue

With a reasonable Scala implementation and its Java approximation the original problem was solved. What kept nagging me was a feeling that something "monadic" could be possible. Sagas are not new, a lot of functional idioms are about sequences. Surely some prior art must exist.

As luck would have it I found an interesting paper almost immediately. It was both very relevant (see "2.1 Atomic Action and Workflow") and accompanied by Scala code. Fundamentally the paper considers some advanced features but it starts from a key idea of applying the Continuation Monad to chaining saga transactions. The bad news is that the original code relies on scalaz and does not work with Futures.

At first I stared at the code in despair. Then I remembered that I happen to sit nine feet from a legit mad scientist who is really into FP. He got interested in my use case and graciously adapted the code to the problem at hand. Behold the monadic version:
If you are like me it'll take you a while to meditate on the simplified code above. First continuation machinery is defined, then a means of converting an action into a continuation is provided (notice the "a: => Future[A]" syntax), and finally our favorite saga is actually composed.

Now about that continuation monad. Apparently it is classical but not really discussed even in the red book. Enough people were confused at work about it that the very same scientist came up with a lengthy explanation. I couldn't recommend it enough. Also now I know that implementing a monad in Scala requires three methods that satisfy the monadic laws.

This post is unusual because I don't fully understand the stuff I am writing about. I am still trying to wrap my head around it. My lame excuse is that very few people at work are comfortable with this code. It has nice type safety and composability. For me writing the monadic wiring from scratch would be a stretch. Lots of intellectual stimulation but merging something like that into a real codebase is a very open question for me.

What is worse is that for those who are unprepared (empirically, the vast majority) the core of this code is unmaintainable, pretty much "write-only" incantations. Concurrency could be hard but one can always go read JCiP one more time. For this stuff there is no bible. As a matter of fact I know only one book that successfully tries to put FP into practical use but it's too basic anyway.

P.S.
Speaking of monads and scientists, if you are into akka-http go look at the pull request making Directives monadic. Personally I expected the Akka guys to be much more impressed.

Transactional saga in Scala III

Part III : Back to the Future

.. in which we return to a Futures-based design but this time compose them idiomatically

Now that a naive Future-based version is out it's time to make it look idiomatic in the third iteration.  I was thinking about composing Futures with flatMap. I did not know how to chain multiple transaction rollbasks though. What I learned in the code review was unexpected.

A colleague caught me by surprise by showing that in a chain of Futures a recoverWith clause is called not just for the Future that failed but also for all the preceding ones.

Another small difference is the nested exception class. The problem it solves is logging the transaction that failed. Without it every rolled back transaction would be logged. This cosmetic improvement doubles the size of the recoverWith body though. Without it this version looks really slick.

At this point I was happy with my pull request. But one thought spoiled the pleasure. We are dealing with a sequence of monadic abstractions. We use flatMap. Could there be some truly monadic approach to composing sagas? Yes, we are finally leaving common sense territory and going some controversial places.

Transactional saga in Scala II

Part II : Future Imperfect

.. in which Future-based signatures are introduced but the Futures are orchestrated naively 

Now that the problem is understood it's time to consider the second iteration. This time all transactions return Futures. It is very convenient from the perspective of building a reactive service.  Compensation actions still return Tries. Partially for the sake of simplicity, partially to reduce potential for additional errors in the error handling path. The whole saga is wrapped into a Promise to make possible for the client to wait for completion.
As you can see, the conversion to Futures was done rather literally. On the positive side, it's so imperative one can easily see how it works. On the other hand, it looks more like Java. Stay tuned for what I learned during the code review.

Transactional saga in Scala I

Part I : Tried But Not True

.. in which we come across a real-life problem, establish its context, and try a naive approach to set the stage for the following posts

Recently I faced a practical problem at work which resulted in four iterations of a promising idea. I found the process instructive enough to share some details with the world. Individual iterations could be not particularly interesting but it's relatively unusual to see a head-to-head comparison of this kind. In this and following posts I am going to embed only skeleton implementations from a gist. If you are interested in detail, please see the real code with tests. 

The problem I was working on was making an operation spanning HDFS and RDBMS transactional. Imagine a (micro)service dealing with file uploads. In a typical flow you first put a file into a temporary location, then register it in the DB, and at last move the file to its real location based on a version-like value returned by the DB. We've got three atomic transactions. They should either all succeed or the ones that have been executed by the time of a failure must be rolled back.

My first and, frankly, only idea was to use sagas. It sounds epic :) but it was not supposed to be about fancy CQRS frameworks. I was just looking for a clean way to compose a few basic Scala types such as Try or Future. Also notice that the entire operation is stateful - each transaction returns some values and usually uses the values produced by previously executed transactions. We'll refer to that sate as transaction context. We can represent the basic abstractions like that: 

• TxContext represents shared state with type-safe access
• ObjectStore represents operations with HDFS files; some of them return a Future because input could be an akka-stream Source, others take an array and so return a Try
• ObjectCatalog represents operations with RDBMS rows

My first iteration was about sketching the general flow of a saga. To begin with there were individual transactions doing actual work. Then there was some looping machinery to execute them and support automatic rollback. 

The ObjectStoreTx trait represents a transaction that can be executed with a context or rolled back. The TryListSaga object implements a loop that executes transactions from a list sequentially and can trigger rollback in case of error.

One obvious problem that the code surfaced is very common in Scala if you mix APIs based on Futures and Tries. No only they do not compose easily but also "waiting on a Future" should happen only at the top level. I had to find a way to rely on Futures everywhere.

Scala is hot in analytics companies

It is hard to believe it's been six years since I started worrying that Java is obsolete :) We lived long enough to see Java8 in production. A couple of years ago I thought that Scala flat-lined. I remember keynotes on never getting to mainstream and compiler folks leaving. But this year I can see a very different picture. Most of the companies dealing with analytics are writing Scala. A great many are either already running Spark in production or prototyping it. At this point it feels like a very good time to join such a company. There are too few people around with real Scala experience to make it a strict requirement.

It is interesting to observe that some arguments against Scala sound so familiar from good old C++ days. Some companies ban use of a certain feature subset (e.g. implicits). Some folks have trouble with a multi-paradigm language where there is so much choice. Few people are excited about function signatures in the standard library (remember STL? partially specialized template anyone?). I know good engineers are supposed to value simplicity. I must be a bad one, I am attracted to complexity. Scala reminds me C++ in this respect. There is always some language tidbit to pick up every time you read a book. In Java only the j.u.c package could fill one with so much joy for years :)

Programming in Scala on Actors

I am through my first reading of the book and somehow it seems to be less helpful than I hoped it would be. I do not think it is bad or merely repackaged online documentation but I guess I expected more from a book written by the language author. Sublime wisdom or something :)

I believe most people are interested in Scala because it is an OOP-friendly tool for learning practical FP and Actors. So I was particularly irritated by chapters on GUI (who cares?) or minute details on integration with Java (better to be online instead of wasting precious book space). On the positive side, my feeling is that FP is covered in the book adequately.

What truly disappointed me was the chapter on Actors. To begin with, why only 1 chapter on this exciting topic? It is not like best Actor practices are better known than FP among us mere mortal with mostly C++/java background. And in my opinion that event simulator example is actually a good illustration of what not to do.

What I wanted to see was a clear comparison with traditional concurrent designs. So some universally-known concurrent system (e.g. a network /web server) would not require any explanation what it is and would immediately conjure up many examples from java world. Some code snippets would be probably even useful in real life. Not only modelling circuitry won't, it actually complicates the discussion with essentially unrelated clutter.

In addition, it would be really useful to learn how all this Actor machinery actually works. Is it some native magic or just a Scala library? How does it compare with j.u.c-based abstractions, at least performance-wide? Not a single word.

As a matter of fact, there is a much more elucidating paper named Actors that Unify Threads and Events. It really helps to read it before you delve into the dark internals of Actor.scala. It turns out that some interesting (and somewhat dirty? or truly ingenious? exceptions for control flow ..) design decisions were made to make it work.

As for the Actor-related code itself, I found it quite messy. Makes me wonder if the root cause was extreme performance fine-tuning or certain sloppiness. There is even a recent post where an independent Scala enthusiast comes up with a more orderly design of Actor.scala (and provides a few more hints on the official design). Hopefully Martin&Co will listen to him.

Is Java obsolete?

This topic seems to be nearly obligatory nowadays in Java circles. Is Java a mature language about to enter its years of slow but inexorable decline? Can it be resuscitated? Should it be? Does anybody care? :)

Clearly, future will tell and there are more experienced people out there who can better articulate their predictions. As most people in my generation I moved to Java from C++ and from the very beginning I knew how many sophisticated features were missing in Java. Back then I was still learning the ropes so it is safe to say that Java is the first major language I have seen evolving from its early years. When I started using C++ it was already king of the castle . Although reading about its history was amusing I was too young to truly appreciate the concept of language life cycle.

Harking back to Java, I am more fascinated by historicity and sociological/psychological roots of programming language preferences and fashions. There are usually a few competing schools of thoughts in most technical areas. Thus, even if I tend to align with some of them I do not believe there is the only right answer. And I hear that quite often technical prowess does not always decide the fate of a technology. The longer I live the more I tend to see heavy influence of wetware on seemingly semi-scientific matter.

I remember talking to a colleague about it a couple of years ago. Under influence of the first(?) book on Java obsolescence I told him that we were not likely to write in Java in, say, ten years. I recollected the year 2000 when I myself was surprised why anybody cared about that obscure thing called Java. I was even more stunned when MSFT embarked upon cloning Java under amusical brand. Why anybody would want anything else if they already had C++?

Fast forward 8 years and more than once I have seen blogs saying that in a few years Java will be actively used only for "low-level, system development". The notion of "low-level" has changed a tad since the time I was young :) And to my personal taste, I'd rather stay in Java than do whatever will be classified as "high-level".

Lacking a CS degree I hardly heard about functional programming or LISP in my youth. And certainly never encountered such things in my professional life. When the recent FP fad started spreading through blogoshere I was mostly taken aback because I did not make much sense to me. And I was pretty irritated with the idea of learning a new language a year. Frankly, I still do not count SQL or Ant scrips as languages (unless implying DSLs) and I do not think a year of even full-time development is enough to learn a serious language (as my experience with C++ and Java tells me).

Actually what is meant by that suggestion is that there is apparently no general metamodel for features found in different languages. So there is no way to learn about a useful concept or idiom without actually spending a lot of time on learning to read languages one is unlikely to use in real life. Which sounds like wasting a lot of time to me. Actually, I prefer frameworks to languages. As an example, deep inside the Erlang runtime all those marvelous message queues and clustering features are actually implemented. I'd rather read a book on algorithms used in that implementation than learn Erlang simply to familiarize myself with those concepts. Alas, I have never seen any Erlang runtime design documentation.

I am also yet to find a paper clearly explaning how OOP and FP could be two sides of the same coin. As they intuitively seem to be (e.g. think about Erlang actors or modules as classes .. yes, strictly speaking it's far-fetched at best). Judging from the dearth of literature on FP design or UML analogs (or at least profiles) for FP people somehow manage to survive with methodologies originated in OOA&D. Which implies certain affinity.

Anyway, I am reading the first book on Scala. It's fun, it's not weird the way Erlang or LISP are, it definitely feels as Java 2.0 and it makes all kinds of Java deficiencies clearly visible. A few things look rather unnatural or necessary only for particular corner cases at first glance. But in general the language is definitely more sophisticated than Java. It also makes me believe that there is no reason to introduce major language features to Java anymore. It would feel as a different language and we would still have tons of legacy code which often does not use even generics. I did not think so even a year ago but I guess now I get what people were talking about for the last couple of years :)

There is no reason to cry about throwing closures out of Java7. Not everyone migrated even to Java6, not to mention those cursed with J2EE stuff chained to jdk1.4. It would be better to officially introduce, say, Scala as the next main JVM language and let Java peacefully lose its mind share. My guess is that we have learned enough from Java to not only wish for but also have a good idea of how a major overhaul of the JVM world should look like.

Improved Scala support in Idea

The other day I noticed that IntelliJ had issued an updated Scala plugin. Unfortunately it works in Idea 8 only (with a good justification) so I had to install its beta as well. The good news is that it was worth it. I remember the original plugin which did not qualify even as a toy. This time they made real progress.

On the positive side now there is decent syntax highlighting, Ctrl-click navigation and other basic Idea facilities actually working. Clearly there is a lot of space for improvement from better code correctness checks to Scala javadoc support in Ctrl-Q but at least now it is possible to play with the language in a meaningful way.

This year I have seen so many blogs telling about Java decline and the rise of dynamic and/or functional languages that I decided it's time to extend my horizons. I decided to look at Scala (as a "better C++ than Java") and Erlang (because of its telco roots and the availability of a very good book). So far I have been struggling mightily with weird syntax of functional languages.

What I really find puzzling is that despite all the buzz about superiority of dynamic/functional languages I have not seen a single book on design with them. The standard OOA&D cannot be completely applicable (e.g. there are no classes in Erlang and no interfaces in a typical dynamic language) and even UML is likely to be more convenient for pure OO. Taking into account how long and hard it was for most people to master true OOP I am surprised with the lack of such discussions.