Generating akka-http endpoints from protobuf declarations

When I started working with akka-http last winter I was surprised they had no better alternative to hand-crafting RESTful endpoints. The gRPC Gateway prototype inspired me to some experimentation. Back then I was still hoping to convince people at work to avoid REST. Recently I wanted to share what I had learned with a colleague and it pushed me to finish the first actually working version.

There are a couple interesting tidbits for protobuf connoisseurs in general. I picked them up while digging into the gateway project. To begin with, the protobuf specification has a description written in protobuf. It is important because the protoc compiler supports plugins which in turn operate in terms of those meta-level protobufs. So you don't have to start your protobuf transpiler from scratch. The ScalaPB project did all the hard work of integrating with protoc toolchain. I am still afraid of my SBT file which I had to mostly borrow.

Secondly, the protobuf specification defines a somewhat obscure feature known as extensions. GOOG itself used an extension to define a mapping of RESTful API elements onto protobuf declarations. It was a pleasant surprise to finally find a standard. You take a protobuf service declaration, augment it with an option clause following clear guidelines, and all of a sudden your still valid protobuf file can be a player in the RESTful world too.

All that remains is to walk an AST comprised of protobuf descriptor classes and generate some code.  On the one hand, it easier than writing a query parser. On the other hand dealing mostly with Strings feels less structured. The current approach was borrowed from the gateway project. I am unhappy with its structure but don't have a good enough alternative yet. Nevertheless the code is straightforward and can be extended.

In our case the generated code will be a skeleton of akka-http service. It starts with some boilerplate Akka initialization and then generates HTTP request processing directives. To illustrate the use of the code generator there is a small sample service project. It's pretty self-explanatory once you look at the protobuf file. 

I learned a few things from this PoC:

• the protoc toolchain is not particularly nice to JVM-based code; thank God for ScalaPB but your SBT file will be rather cryptic anyway
• now that we know how to write a scalac plugin the actual transpiler logic is quite easy to write
• the protobuf-to-REST mapping is very reasonable but has edge cases such as nested messages

One point of contention is streaming. It's well known that gRPC streaming is not synonymous with bulk data transfer. The former operates on individual messages, the latter deals with sending a large file without buffering the entire file in a byte array. The former is represented by some ScalaPB-generated case classes, the latter typically looks like an akka-http Source. So there is no standard way to express in protobuf the need to generate anything akka-streaming-related.