Transactional saga in Scala IV

Part V : Mapping future

.. in which Japanese researchers and a mad Russian scientist continue

With a reasonable Scala implementation and its Java approximation the original problem was solved. What kept nagging me was a feeling that something "monadic" could be possible. Sagas are not new, a lot of functional idioms are about sequences. Surely some prior art must exist.

As luck would have it I found an interesting paper almost immediately. It was both very relevant (see "2.1 Atomic Action and Workflow") and accompanied by Scala code. Fundamentally the paper considers some advanced features but it starts from a key idea of applying the Continuation Monad to chaining saga transactions. The bad news is that the original code relies on scalaz and does not work with Futures.

At first I stared at the code in despair. Then I remembered that I happen to sit nine feet from a legit mad scientist who is really into FP. He got interested in my use case and graciously adapted the code to the problem at hand. Behold the monadic version:
If you are like me it'll take you a while to meditate on the simplified code above. First continuation machinery is defined, then a means of converting an action into a continuation is provided (notice the "a: => Future[A]" syntax), and finally our favorite saga is actually composed.

Now about that continuation monad. Apparently it is classical but not really discussed even in the red book. Enough people were confused at work about it that the very same scientist came up with a lengthy explanation. I couldn't recommend it enough. Also now I know that implementing a monad in Scala requires three methods that satisfy the monadic laws.

This post is unusual because I don't fully understand the stuff I am writing about. I am still trying to wrap my head around it. My lame excuse is that very few people at work are comfortable with this code. It has nice type safety and composability. For me writing the monadic wiring from scratch would be a stretch. Lots of intellectual stimulation but merging something like that into a real codebase is a very open question for me.

What is worse is that for those who are unprepared (empirically, the vast majority) the core of this code is unmaintainable, pretty much "write-only" incantations. Concurrency could be hard but one can always go read JCiP one more time. For this stuff there is no bible. As a matter of fact I know only one book that successfully tries to put FP into practical use but it's too basic anyway.

P.S.
Speaking of monads and scientists, if you are into akka-http go look at the pull request making Directives monadic. Personally I expected the Akka guys to be much more impressed.

Transactional saga in Java

Part IV : Intermezzo

.. in which we try and fail exceptionally to have Futures more complete

Having developed the third Scala version I got curious about doing the same in Java.  I more or less expected the CompletableFuture to be a more verbose flavor of the Scala Future. It had also been a year since I stopped writing Java and I wanted to check how I felt about it. If you remember the last Scala iteration the following should look familiar:

At first glance "thenCompose" is like "flatMap" and "exceptionally" resembles "recoverWith". Close but no cigar. You cannot return a CompletableFuture from a function taken by CF::exceptionally. It does not cancel the rest of saga execution either. Not even in JDK10. So I had to throw an exception and to manually roll back transactions the way it worked in the second Scala version. It's tolerable but not exactly elegant.

One lesson here is that Scala Futures are at least a little more powerful than the Java ones. Once you get used to monadic composition two dozen Java signatures do not look all that readable anymore. While typing Java code I also noticed how tedious it is in comparison with Scala. All trivial bits such as constructors, semicolons, and no variable type inference result in constant friction. I am afraid I don't feel like writing Java anymore.

Transactional saga in Scala III

Part III : Back to the Future

.. in which we return to a Futures-based design but this time compose them idiomatically

Now that a naive Future-based version is out it's time to make it look idiomatic in the third iteration.  I was thinking about composing Futures with flatMap. I did not know how to chain multiple transaction rollbasks though. What I learned in the code review was unexpected.

A colleague caught me by surprise by showing that in a chain of Futures a recoverWith clause is called not just for the Future that failed but also for all the preceding ones.

Another small difference is the nested exception class. The problem it solves is logging the transaction that failed. Without it every rolled back transaction would be logged. This cosmetic improvement doubles the size of the recoverWith body though. Without it this version looks really slick.

At this point I was happy with my pull request. But one thought spoiled the pleasure. We are dealing with a sequence of monadic abstractions. We use flatMap. Could there be some truly monadic approach to composing sagas? Yes, we are finally leaving common sense territory and going some controversial places.

Transactional saga in Scala II

Part II : Future Imperfect

.. in which Future-based signatures are introduced but the Futures are orchestrated naively 

Now that the problem is understood it's time to consider the second iteration. This time all transactions return Futures. It is very convenient from the perspective of building a reactive service.  Compensation actions still return Tries. Partially for the sake of simplicity, partially to reduce potential for additional errors in the error handling path. The whole saga is wrapped into a Promise to make possible for the client to wait for completion.
As you can see, the conversion to Futures was done rather literally. On the positive side, it's so imperative one can easily see how it works. On the other hand, it looks more like Java. Stay tuned for what I learned during the code review.

Transactional saga in Scala I

Part I : Tried But Not True

.. in which we come across a real-life problem, establish its context, and try a naive approach to set the stage for the following posts

Recently I faced a practical problem at work which resulted in four iterations of a promising idea. I found the process instructive enough to share some details with the world. Individual iterations could be not particularly interesting but it's relatively unusual to see a head-to-head comparison of this kind. In this and following posts I am going to embed only skeleton implementations from a gist. If you are interested in detail, please see the real code with tests. 

The problem I was working on was making an operation spanning HDFS and RDBMS transactional. Imagine a (micro)service dealing with file uploads. In a typical flow you first put a file into a temporary location, then register it in the DB, and at last move the file to its real location based on a version-like value returned by the DB. We've got three atomic transactions. They should either all succeed or the ones that have been executed by the time of a failure must be rolled back.

My first and, frankly, only idea was to use sagas. It sounds epic :) but it was not supposed to be about fancy CQRS frameworks. I was just looking for a clean way to compose a few basic Scala types such as Try or Future. Also notice that the entire operation is stateful - each transaction returns some values and usually uses the values produced by previously executed transactions. We'll refer to that sate as transaction context. We can represent the basic abstractions like that: 

• TxContext represents shared state with type-safe access
• ObjectStore represents operations with HDFS files; some of them return a Future because input could be an akka-stream Source, others take an array and so return a Try
• ObjectCatalog represents operations with RDBMS rows

My first iteration was about sketching the general flow of a saga. To begin with there were individual transactions doing actual work. Then there was some looping machinery to execute them and support automatic rollback. 

The ObjectStoreTx trait represents a transaction that can be executed with a context or rolled back. The TryListSaga object implements a loop that executes transactions from a list sequentially and can trigger rollback in case of error.

One obvious problem that the code surfaced is very common in Scala if you mix APIs based on Futures and Tries. No only they do not compose easily but also "waiting on a Future" should happen only at the top level. I had to find a way to rely on Futures everywhere.