Transactional saga in Java

Part IV : Intermezzo

.. in which we try and fail exceptionally to have Futures more complete

Having developed the third Scala version I got curious about doing the same in Java.  I more or less expected the CompletableFuture to be a more verbose flavor of the Scala Future. It had also been a year since I stopped writing Java and I wanted to check how I felt about it. If you remember the last Scala iteration the following should look familiar:

At first glance "thenCompose" is like "flatMap" and "exceptionally" resembles "recoverWith". Close but no cigar. You cannot return a CompletableFuture from a function taken by CF::exceptionally. It does not cancel the rest of saga execution either. Not even in JDK10. So I had to throw an exception and to manually roll back transactions the way it worked in the second Scala version. It's tolerable but not exactly elegant.

One lesson here is that Scala Futures are at least a little more powerful than the Java ones. Once you get used to monadic composition two dozen Java signatures do not look all that readable anymore. While typing Java code I also noticed how tedious it is in comparison with Scala. All trivial bits such as constructors, semicolons, and no variable type inference result in constant friction. I am afraid I don't feel like writing Java anymore.

Asynchronous AWS S3 file transfer

I am probably ten years too late in writing about the oldest AWS service but the power of AWS SDK asynchronous file transfer is irresistible. Moving a 300MB file takes just a few seconds. It is very easy to plug this API into reactive backend services. For this exercise we assume that one service uploads a file to S3 and another service periodically checks for new files in that location.

Fundamentally, asynchronous operations require an instance of the TransferManager class and a callback to process status notifications. I wrapped the whole process into a few classes representing abstractions for uploading to, downloading from, and detecting newly uploaded files in a pre-configured location in some S3 bucket location.

The TransferManager API typically takes a Request object and an asynchronous status listener. It returns a Transfer instance that can be used to retrieve error message in case of failure. Polling an S3 location for available files requires a loop because the results are returned in batches. Checking if a file exists at a given S3 path  is implemented as an attempt to fetch the corresponding file metadata and treating a thrown exception as "FileNotFound".

S3 is a simple (duh!) service so there are only two additional notes. First, it is a good idea to encode some metadata into file names on S3. Things such as tenant id or version or video resolution. It helps with deciding how to handle a downloaded file by parsing its name. Second, it's convenient to superimpose a "directory structure" onto the flat namespace of the S3 bucket abstraction. 

So a reasonable file naming convention might include a three-part prefix appended to all relative file paths: a backend service name, a file schema version, and a namespace representing either an environment (e.g. PROD) or a developer (in development deployments). The version part in particular makes upgrades much easier in production. For example,
"$BUCKET/prod/somesvc/v2/relative_path/file.ext".

The digram below shows a typical sequence of operations for uploading a file and then finding it with S3 polling from a different service.

In my example,

• FileDownloader / FileUploader - abstractions used by the client to start a file transfer operation
• TransferCallback - the callback interface called by file transfer operations to report final status to the client asynchronously
• S3Destination - a way to specify a common bucket "subdirectory" for multiple files
• S3TransferProgressListener - converts progress events to success/failure notifications; makes possible to extract an error message 
• S3FileTransferClient / S3Client - TransferManager-based implementation of file transfer operations
• S3Uploader / S3Downloader - S3Client-based implementation of file transfer API     
• S3ChangeDetector - a job to be run periodically (e.g. with ScheduledExecutorService::scheduleWithFixedDelay) on the receiver side to look for new files on S3
• FileHandler - the callback interface called by S3ChangeDetector for every found file not seen before (the way of keeping track of previously seen files is likely to be application-specific)

Object block initializor

Object block initializor in Java is a code block in parenthesisexecuted before the class constructor. I always thought it was a bad idea because by default I expect all initialization code in a constructor. In addition, I have never seen such initializors used anywhere except for Java textbooks and interview questions.

Yesterday I for the first time read about a potentially useful way to utilize them. The idea is to save some typing (compare creation of b1 and b2):

Thunk

I first learned about this approach to lazy initialization a few months ago but every time I look at its Java implementation I think it's a really cute piece of code. I wonder why my book on Singletons did not mentions it :)

Sizing up Java employers

While trying to understand what kind of company I would like to work for next I made another observation on employers in the Java land. At first glance they fall into three categories:

• Typical Java shops doing something of little interest from the engineering point of view. Their job descriptions usually mention Spring and Hibernate. Frequently they also ask for either JEE or presentation tier technologies a-la JSP/Struts (IMO the latter is a particularly strong indicator you do not want to work for them).
• Companies working on something more challenging or in more exciting domains. For me a good indicator is that they ask for more advanced/less known technologies such as Lucene, Hadoop or one of the numerous DHT implementations.
• One-of-a-kind companies, GOOG-style. They do not list any particular technologies except for Java itself. They are explicitly after what they refer to as "bright people".

Seniors among Java engineers

If you were to trust what you read on the Net you would think Java is a brain-dead language and people who develop in Java are the alternatively gifted among software engineers. While the former is a multidimensional question the latter could make you nervous if you've been in the Java land for many years. Allegedly, in some circles a decade of predominantly Java experience raises a red flag.

I am inclined to think that Java is mostly infamous for its enterprise branch. Surely once fashionable J2EE technologies are good for outsourcing but the damage they inflict on developers is very real. But if you ask me, I'd rather do EJB2 than any kind of web UI from JSP to ajax. Luckily I have not had to deal with either for a long time.

Occasionally I participate in interviewing potential future coworkers. And some of them position themselves as Sr Java engineers and even have CS major backgrounds. You would think that in the Bay Area developers are likely to be a uniformly enlighten group. After all, this is arguably the most expensive place to hire and an international magnet for software engineers.

Not really if my experience is representative. I have interviewed and even worked here with people who demonstrated almost appalling lack of professional skills (e.g. decent OO design with proper decoupling and meaningful responsibility assignment) or relatively basic knowledge (the role of hashCode/equals in the HashMap context, anyone? optimistic/pessimistic locking strategies? ACID? mapping of class hierarchies on relational DB tables?). On a few occasions after a decade or two in the industry.

Or is it just me? Probably I am supposed to ask candidates to implement a RW lock (unless your name is Doug or Brian I'd rather know you are intimately familiar with the util.concurrent package) or recursive tree scanning? But those look like more advanced concepts to me than the difference between volatile and synchronized in protecting a long variable. And it's not like in their defense they could show years of non-OO (although I am pretty sure FP folks are into good design, only by different means) or non-Java/C++ experience (then again, what makes him a Sr Java engineer?).

Could it be the case that Java is merely so dominant that it covers wildly different domains (e.g. util.concurrent/NIO v J2EE; telecoms or financial middleware v a web UI for DB access) and we do not have specific terms for people with the corresponding expertise?

How to leak Linux pipes from Java code

Fixing a bug in legacy code I found a moderately counter-intuitive rule for dealing with Process instances returned by Runtime when one executes a process. It turns out that the three standard streams associated with each instance must be closed regarless of whether you use them anywhere in your code.

By default I expect that unless I explicitly open something there is no need to close it manually. Moreover, I double-checked javadocs and Java Programming Language and I have not found any related warning. On the other hand as it happens on the boundary between java world and OS it is probably natural to expect very strict requirements for disposing of unused resources. At least it was not me who wrote that code in the first place :)

An easy way to check if pipes are leaked is to run something like "/usr/sbin/lsof | grep pipe | grep java" and see if the number is growing. In my experience if you get a "too many open files" error it's likely to be either about leaking pipes or sockets (if you can see an unreasonable number of sockets in CLOSE_WAIT state in netstat output).

They knew it but they still ended up with Java as we know it

Ten years ago I did not know who Guy Steele was or that I would spend years programming in Java. Seven years ago I thought that a language without operator overloading and templates was an ugly toy. A few years later I got so used to it I mostly stopped worrying.

It turns out Java authors were not morons. They might have thought that the users of their language were. Or they just could not imagine that a language for programming refrigerators (or was it set top boxes?) would be of any wider use. What is painful to learn in hindsight is that they knew about all the good suff , liked it a lot themselves but somehow chose to ignore it.

Small language changes for JDK7

I am a thoroughly conservative guy and quite naturally tend to be a late adopter even if I am aware of a particular new technology in general. Watching a panel on future of Java on infoq.com a couple of years ago I was quite surprised by people of Joshua Bloch caliber discussing Java as a probably mature/declining language.

Back then I expected that somehow that language changes of Java5-magnitude would be added with each new release. I believe that only in Java5 the language was getting some of the C++ power it was originally deprived of (e.g. STL and all the features enabling it). So it felt as Sun finally decided to get serious about the language and stop dumbing it down to almost VB level.

Surely enough the titans were right and saw it coming even back then. In contrast to the JVM (as in garbage collectors or contested monitor performance) Java does not seem to have meaningful progress anymore. Syntactic sugar just is not worth it in my opinion. And Doug Lea seems to be the only one pushing Java's limits.

Things like catch blocks for multiple exceptions do not bring much to the table, not to mention all the legacy code which is not likely to be migrated. And to have mixed styles in the same code base is even worse than to have gratuitious blocks in the first place. As a creature of habit, I do not even notice such things anymore, I code them on auto-pilot.

Last year I was reading the Erlang book and got so confused with FP idioms that I had to digress and read the Scala book to better grasp what all the FP hype was about. It did not convert me from OOP but definitely made me appreciate FP (and incidentally reminded me about STL which enabled all the FP-style fun with lists fifteen years ago).

And somewhere down that road I found myself thinking that I do not want Java to be changed dramatically anymore. An attempt to make Java more sophisticated would result is a limited version of, say, Scala. It would be a new language and all the existing code would immediately become non-idiomatic. Or in the worst case there would be just a few special cases hard-coded into the language instead of meta-protocol-like facilities.

So, if even I have this funny feeling that any potential Java upgrades (years ahead, and some people have not migrated even to 1.6) are not as appealing as a modern language such as Scala probably the tide is actually turning.

Is Java relevant?

My previous post made me think about my attitude towards Java. Even if it happens to be behind its pinnacle, is Java any good beyond paying my bills for years? If we know what Java gave this world probably we will be better equipped to decide when to move to a new language and how to judge it.

History teaches us that non-technical aspects matter more than what is habitually referred to as technical excellence. Which is actually difficult to define. I have a growing suspicion that what matters a lot is something I would call, for a lack of standard term, "age and experience demographics". The idea is that certain technologies are perceived to be cool by people entering the field. Those technologies in a sense create an imprint on that particular generation.

As an example, in the early nineties Assembly was for real men and C++ for real programmers. At least how I felt it back then under certain circumstances (no influence from CS-savvy people, and extremely limited literature in the absence of the Internet, most people using and targeting MSFT platform). I know that years before I discovered computers people had been writing in HEX code and considered Assembly to be unmanly. It very well may be that nowadays something like scripting/dynamic languages are all the rage among the young.

There must be also a heavy influence of experience resulting in some kind of "shifting interest curve". You need to be through a complete life cycle of a major language such as Java to be able to see that not everyting is new under the sun. As an example of a self-made developer, I have progressed through Pascal, C++, OOP, OOA&D, RUP, UML, Java, algoritms and data structures 101, development methodologies, software architecture, really understanding OOA&D, really understanding Java concurency, TDD (disclaimer: for simplicity sake the sequence was artificially serialized and stripped of many things).

Right now I value good analysis and design (assuming OOA&D; I am not aware of FP/dynamic language alternatives to it; and yes, I appreciate UML) and best software engineering practices more than a particular language. I like reading about algorithms much more than I did ten years ago. I am dabbling with FP in its safely diluted incarnation (Scala) and planning to look again at Erlang later next year. Who knows, probably in my next ten years I will be able to get interested and even like highbrow stuff such as LISP. The point is that in addition to normal technological progress each individual has another learning curve which includes things not affected directly by his current job.

And last but not least, not everyone majored in CS. As a matter of fact, I believe I have worked with more people from Engineering/Physics backgrounds than properly educated CS graduates. Naturally, the former were not indoctrinated in seemingly non-practical subjects such as LISP, FP or even the details of compiler design. Some of those are probably later picked in spare time but I would still expect a gap between originally CS and non-CS folks.

So, what do I like about Java? I happen to maintain an Apache module written in C and every time I fix a bug there I appreciate our warm and fuzzy Java world more. This juxtaposition elucidates good things about Java although by no means it is even remotely scientific or exhaustive. Disclamer: borders between Java the language, the JVM and Java ecosystem at large are blurred to reflect real life.

• Java has a standard API (or a de facto standard API) for nearly everything. And innumerate open-source implementations of those.
  
• Java has multiple open-source components available for anything from reliable messaging and IoC to web UI and Map/Reduce on serious scale.
• We have a set of widespread mature technologies covering most important mundane areas (e.g. Maven/Spring/JUnit)
• Java has a memory model and an amazing util.concurrent library. It was Doug Lea and not James Gosling who was my youth hero :) And Brian Goetz wrote really nice javadocs :)
• Java has a decent Collections libraryJava dominates the server side. Jobs are abundant in interesting domains from Finance to Telecommunications
• Java promotes OOP and interface-based design. Not really design by contract though :(
• Because Java is a dumbed-down language, it is easy to read (well, multi-screeen methods are still found out there but I guess you catch my drift). Though I have always missed overloaded operators and proper generics (enabling the STL library in C++).
• Nowadays there are enough high-quality Java books
• Without IntelliJ IDEA I would go out, shoot myself in the foot and die :)
• Java does not look weird (presumably in comparison with the C branch of evolution)

Most of us would be able to continue this list for a long time. When I look at it the point seems to be indeed that it is not about Java the language. Libraries and the JVM sneak into the picture all the time. As a creature of habit I like Java because I am used to it but if a JVM language more sophisticated than Java comes along I would be all for it.

Harking back to Scala, the language looks smart to me (yes, I am enamored with C++ like complexity and things like covariant/contravariant type annotations attract me rather than scare away). I detest languages that restrict me (Java more than C++, Erlang more than Scala). I cannot imagine living with a language where I would not be able to do OOP even if I like the idea of trying out at least elements of FP. And Martin reminds meBjarne more than others do :)

Is Java obsolete?

This topic seems to be nearly obligatory nowadays in Java circles. Is Java a mature language about to enter its years of slow but inexorable decline? Can it be resuscitated? Should it be? Does anybody care? :)

Clearly, future will tell and there are more experienced people out there who can better articulate their predictions. As most people in my generation I moved to Java from C++ and from the very beginning I knew how many sophisticated features were missing in Java. Back then I was still learning the ropes so it is safe to say that Java is the first major language I have seen evolving from its early years. When I started using C++ it was already king of the castle . Although reading about its history was amusing I was too young to truly appreciate the concept of language life cycle.

Harking back to Java, I am more fascinated by historicity and sociological/psychological roots of programming language preferences and fashions. There are usually a few competing schools of thoughts in most technical areas. Thus, even if I tend to align with some of them I do not believe there is the only right answer. And I hear that quite often technical prowess does not always decide the fate of a technology. The longer I live the more I tend to see heavy influence of wetware on seemingly semi-scientific matter.

I remember talking to a colleague about it a couple of years ago. Under influence of the first(?) book on Java obsolescence I told him that we were not likely to write in Java in, say, ten years. I recollected the year 2000 when I myself was surprised why anybody cared about that obscure thing called Java. I was even more stunned when MSFT embarked upon cloning Java under amusical brand. Why anybody would want anything else if they already had C++?

Fast forward 8 years and more than once I have seen blogs saying that in a few years Java will be actively used only for "low-level, system development". The notion of "low-level" has changed a tad since the time I was young :) And to my personal taste, I'd rather stay in Java than do whatever will be classified as "high-level".

Lacking a CS degree I hardly heard about functional programming or LISP in my youth. And certainly never encountered such things in my professional life. When the recent FP fad started spreading through blogoshere I was mostly taken aback because I did not make much sense to me. And I was pretty irritated with the idea of learning a new language a year. Frankly, I still do not count SQL or Ant scrips as languages (unless implying DSLs) and I do not think a year of even full-time development is enough to learn a serious language (as my experience with C++ and Java tells me).

Actually what is meant by that suggestion is that there is apparently no general metamodel for features found in different languages. So there is no way to learn about a useful concept or idiom without actually spending a lot of time on learning to read languages one is unlikely to use in real life. Which sounds like wasting a lot of time to me. Actually, I prefer frameworks to languages. As an example, deep inside the Erlang runtime all those marvelous message queues and clustering features are actually implemented. I'd rather read a book on algorithms used in that implementation than learn Erlang simply to familiarize myself with those concepts. Alas, I have never seen any Erlang runtime design documentation.

I am also yet to find a paper clearly explaning how OOP and FP could be two sides of the same coin. As they intuitively seem to be (e.g. think about Erlang actors or modules as classes .. yes, strictly speaking it's far-fetched at best). Judging from the dearth of literature on FP design or UML analogs (or at least profiles) for FP people somehow manage to survive with methodologies originated in OOA&D. Which implies certain affinity.

Anyway, I am reading the first book on Scala. It's fun, it's not weird the way Erlang or LISP are, it definitely feels as Java 2.0 and it makes all kinds of Java deficiencies clearly visible. A few things look rather unnatural or necessary only for particular corner cases at first glance. But in general the language is definitely more sophisticated than Java. It also makes me believe that there is no reason to introduce major language features to Java anymore. It would feel as a different language and we would still have tons of legacy code which often does not use even generics. I did not think so even a year ago but I guess now I get what people were talking about for the last couple of years :)

There is no reason to cry about throwing closures out of Java7. Not everyone migrated even to Java6, not to mention those cursed with J2EE stuff chained to jdk1.4. It would be better to officially introduce, say, Scala as the next main JVM language and let Java peacefully lose its mind share. My guess is that we have learned enough from Java to not only wish for but also have a good idea of how a major overhaul of the JVM world should look like.

FP and LISP for the rest of us

As a developer with a traditional C++/Java background and a non-CS major I find the recent surge in popularity of functional programming pretty confusing. It turns out that there is a whole parallel universe of FP where most hard-won imperative experience is of little help in understanding new concepts. And those are frequently hidden behind bizarre syntax.

I have stumbled upon a couple of very well written articles on FP and LISP . The author starts from something as trivial as Java and Ant and ends up explaining, say, the power of LISP. Very approachable with quite a few insights.