Are functions colored?

There is a blog post that I read some weeks ago when I was in a frenzy reading Rust Language RFCs, specifically this one (yes, I'm crazy and read the better part of that thread; the comment that mentioned the article is here).

I want to begin with a little context because my opinions here shifted a lot while thinking about the involved topics.

First: how Rust handles exceptional conditions. In a language such as Python, you could write a function like this:

def divide(divident, divisor):
    if divisor == 0:
        raise RuntimeError("Divide by zero")
    return divident / divisor

There's probably a DivisionByZero error in the standard library, but that's not my point. The important thing is that "raising" (or in other languages "throwing") an exception can be handled by the caller, or not (badness of code again not the point):

def foo():
    try:
        return divide(3, 0)
    except RuntimeError:
        return None

def bar():
    x = divide(3, 0)
    print(x)

foo provides a default value for division by zero, bar propagates the error to its caller, who then has to handle (or again propagate) it.

Rust takes a different approach. Rust doesn't have exceptions, instead it has a Result type that's defined roughly like this:

pub enum Result<T, E> {
    Ok(T),
    Err(E),
}

A result can either be Ok and contain a result of type T, or an Err, and contain an error value of type E. Our function could look like this:

fn divide(divident: u32, divisor: u32) -> Result<u32, ()> {
    if divisor == 0 {
        Ok(divident / divisor)
    } else {
        Err(())
    }
}

Here E is (), which means an Err doesn't carry any additional information. This is how a caller could use this:

fn foo() -> Option<u32> {
    match divide(3, 0) {
        Result::Ok(value) => Some(value),
        Result::Err(_) => None,
    }
}

fn bar() {
    let x = divide(3, 0);
    println!("{}", x.unwrap());
}

fn baz() {
    if let Result::Ok(x) = divide(3, 0) {
        println!("{}", value);
    }
}

Here the discrepancies between Python and Rust are very apparent:

• foo returns Option<u32>, not just u32. That means the caller again has to unpack that value if they need a u32. The Python code would probably lead to an exception later, if None can't be processed somewhere.
• bar uses a second error handling mode that is kind of similar to exceptions, but is not the first choice in Rust: it panics. unwrap() returns the value of a result, but if the result was an error, the panic (usually) kills the thread or process. Panics can be caught, but that's meant for infrastructure code such as thread pools, not for everyday application error handling.
• baz swallows the exception: the if let statement matches only a successful result, so if there's no match, nothing happens.

There is an additional way of handling errors that I'll need again later:

fn quuz() -> Result<(), ()> {
    let x = divide(3, 0)?;
    println!("{}", x);
    Ok(())
}

Notice the question mark. This operator - I'll call it the try operator - returns the result immediately if it was an error, or takes its content if it was successful. This early-returning behavior feels a little like raising an exception, but it requires that the function itself returns a result - otherwise it couldn't return an Err! A side effect is that, even though we don't have a real result, we need to end the function with Ok(()), i.e. return a successful result without any data.

Now back to code colors. This is the point where you should read the linked blog post, if you haven't already. It proposes a language where

• every function is either blue or red
• those colors are called with different syntax
• only red functions can call other red functions
• red functions are "annoying"

Those red functions are async functions that need async/await syntax to call. For the rest, read the linked post. An important point here is transparency: callers have to know function color; higher-order functions may work on all blue functions, but a separate function for red functions (and thus code duplication) might be necessary.

At first the argument that async poses a needless separation into two colors seemed intriguing, although I wasn't entirely convinced. But that doubt felt lazy, caused by habit rather than reason. I left it at that, followed some more discussions about possible syntactic options for async/await in Rust, and wrote some actual async Rust code.

Rust tries to get things right. The people behind Rust experiment with features a lot before they're stabilized, and they wouldn't just add an await keyword because that's what other languages do; Rust's syntax should not only be familiar, it should be practical and reduce the possibility for programmer errors.

A concern with await is the combination with the question mark. Async operations are often ones that could fail, so you would await them and then check for an error - like this:

(await do_something())?;  // or
await do_something()?;

The first is ugly, but would you expect the second to await first, try second? That's why people were discussing a postfix variant:

do_something().await?;  // or
do_something().await!()?;  // actually proposed postfix macro syntax, or even
do_something()~?;  // I made that up, but in principle, why not?

This reads properly from left to right. And, now I'm getting to the point, it shows that the blue/red distinction is not really unique to the topic of async functions. Error handling, at least the explicit kind using Result, is basically the same:

• every function is either blue or red - either you return T or Result<T>
• those colors are called with different syntax - you need to extract Results, but not Ts
• only red functions can call other red functions - not entirely. You can do one of three things:
• make your function red and use the try operator (quux);
• deal with the implications and keep your function blue (foo, bar, baz);
• write buggy code (bar, baz depending on requirements)
• red functions are "annoying" - Results are more annoying than Ts

Does that mean that Results are a bad way of handling errors? I don't think so. How do exceptions hold up in this comparison?

• every function is either blue or red - if you throw exceptions, that's a difference in contract; I take it as a yes
• those colors are called with different syntax - that's the plus, your higher-order functions don't see any color
• only red functions can call other red functions - nope, but I think that was already debatable
• red functions are "annoying" - not to call, but to call correctly

In fact, if we look at checked exceptions as they exist in Java, we get basically the same picture as with async/await or results: you can't put a function that may throw a checked exception somewhere no checked exceptions are expected.

So Results and (unchecked) exceptions are both valid mechanisms on a spectrum of explicit vs. implicit error handling. await is on the explicit side of the asynchronicity spectrum; it remains to be seen how implicit asynchronicity can be implemented properly (Go might qualify as an already working solution here). It may not be a good fit for Rust, which chooses the explicit path, but dynamic languages might benefit from a proven solution to this.

Type-level states in Rust

This is a pattern I saw recently while reading some hardware abstraction code for microcontroller programming with rust. On microcontrollers, you can access pins of your chip and use them for different kinds of work. For example, one pin could be used for input, another for output, an yet another pin could be configured for "alternate functions". Also, an input pin could have a pullup or pulldown resistor, or none ("floating").

So there's a hierarchy of configurations a pin could use:

• Input
• Floating (default)
• PullUp
• PullDown
• Output
• PushPull
• OpenDrain
• AF0 (alternate function 0)
• AF1
• ...

This state is stored in the microcontroller hardware, but this pattern could also be useful when the state is stored elsewhere, like connection state that's stored in the network card's driver, or state on a server.

A classical way to model this is using an enum, storing the state somewhere, and checking the state at runtime (or just trusting the programmer):

#[derive(PartialEq, Clone, Copy)]
pub enum Mode {
    InputFloating,
    InputPullUp,
    Output,
    // more features left out for brevity
}

pub struct Pin {
    mode: Mode, // just for checking
}

impl Pin {
    pub fn new() -> Self {
        Pin { mode: Mode::InputFloating, /* just for checking */ }
    }

    pub fn configure(&mut self, mode: Mode) {
        self.mode = mode;  // just for checking
        // configure the hardware
        match mode {
            Mode::InputFloating => {}
            Mode::InputPullUp => {}
            Mode::Output => {}
        }
    }

    pub fn read(&self) -> bool {
        assert!(self.mode != Mode::Output); // just for checking
        false // bit is read from hardware
    }

    pub fn write(&self, value: bool) {
        assert!(self.mode == Mode::Output); // just for checking
        // bit is written to hardware
    }
}

There are a couple of downsides with this approach:

• If we want checks, the value is stored twice: once in the struct and once in a register of the microcontroller
• If we want checks, we also slow down the actual program.
• If we trust the programmer instead, we can remove all the "just for checking" lines/items, but now we're trusting the programmer.
• Even with checks, we only see errors at runtime, and there could be lots of code paths that lead to writing or reading a pin.

Luckily, we can do better!

pub struct InputFloatingPin(());
pub struct InputPullUpPin(());
pub struct OutputPin(());

impl InputFloatingPin {
    pub fn new() -> Self {
        InputFloatingPin(())
    }

    pub fn into_input_pull_up(self) -> InputPullUpPin {
        // configure the hardware
        InputPullUpPin(())
    }

    pub fn into_output(self) -> OutputPin {
        // configure the hardware
        OutputPin(())
    }

    pub fn read(&self) -> bool {
        false // bit is read from hardware
    }
}

impl InputPullUpPin {
    // pub fn into_...

    pub fn read(&self) -> bool {
        false // bit is read from hardware
    }
}

impl OutputPin {
    // pub fn into_...

    pub fn write(&self, value: bool) {
        // bit is written to hardware
    }
}

Here, every configuration has its own struct type, and we use into_... methods to configure the pin and convert between the associated struct type. The struct definitions with (()) at the end look a little funny, that's for visibility reasons: the () is a 0-byte-sized private field that prevents the struct from being constructed outside the declaring module. Clients have to go through InputFloatingPin::new(), meaning they have to start with the reset configuration.

So we have a bunch of structs with no values in them, which seems kind of stupid. But as noted above, the configuration is in hardware anyway. Also, creating structs that don't need memory is not only cheap, it has no cost at all! We get all the checks from above, but at compile time without slowing the program. We can even use these types to specify that a function requires a pin in a certain configuration.

There is a downside however: code duplication. There's the into_... methods I left out, and there's the read method that's written twice.

So let's fix that with generics. The idea is to define a struct Pin<MODE>; all the into_... methods would be defined on Pin<_>, thus available in any configuration. Pin<Output> is trivial, but crucially, we model the proper configuration hierarchy for inputs: read is implemented for Pin<Input<_>>! So no matter what kind of input we have, we can read from it:

use std::marker::PhantomData;

pub struct Pin<MODE>(PhantomData<MODE>);

pub struct Input<MODE>(PhantomData<MODE>);

pub struct Floating(());
pub struct PullUp(());

pub struct Output(());

impl Pin<Input<Floating>> {
    pub fn new() -> Self {
        Pin(PhantomData)
    }
}

impl<MODE> Pin<MODE> {
    pub fn into_input_floating(self) -> Pin<Input<Floating>> {
        // configure the hardware
        Pin(PhantomData)
    }

    pub fn into_input_pull_up(self) -> Pin<Input<PullUp>> {
        // configure the hardware
        Pin(PhantomData)
    }

    pub fn into_output(self) -> Pin<Output> {
        // configure the hardware
        Pin(PhantomData)
    }
}

impl<MODE> Pin<Input<MODE>> {
    pub fn read(&self) -> bool {
        false // bit is read from hardware
    }
}

impl Pin<Output> {
    pub fn write(&self, value: bool) {
        // bit is written to hardware
    }
}

HTTP and Websockets

A quick epilogue to the service/topic story: what approach is used in HTTP (and many other Internet protocols)? Of course, it's a classical request/reply architecture, and that is too bad!

HTTP has become one of the most prevalent protocols, but it does not provide a way to initiate communication from the server. Back when browsers displayed static web pages, that was fine, but now the content that is served to browsers is highly dynamic. And browsers, for security reasons, can't just use any protocol. For a long time, HTTP was the only option.

An initial "fix" to this was HTTP long polling: the client would issue an HTTP request, but instead of responding immediately the server would wait for some event to occur. Once the client got a response, he would immediately make another request, which would then block again. That's not what HTTP was supposed to do, but it works.

Fortunately, we recently got WebSockets, which handle bidirectional communication in a clean manner. Now, we basically have this:

Browser   Webserver
   ---- http ---->    Browser sends requests, server replies
   <- websocket ->    Browser and server can both send messages at any time

And ta-da! Universality was restored!

The ROS Service/Topic abstractions

Although I quickly outlined the two messaging facilities of ROS (topics and services), I then focused on possible underlying I/O models, and said that a polling mechanism makes for a good primitive.

I think it's clear that without the ability to poll multiple event sources, it's hard to implement efficient asynchronous IO. On the other hand, the opposite direction is really easy. Let's take the code from last time:

rospy.poll(sub1, sub2)
data = sub1.receive()
if data is not None:
    rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)
data = sub2.receive()
if data is not None:
    rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)

To perform a blocking receive, all you have to do is write this:

rospy.poll(sub)
data = sub.receive()

So supporting polling does not push one into nonblocking IO, but not supporting it does push one into blocking IO.

Now let's look a level higher at the actual messaging. ROS provides us with two tools here:

• topics allow multiple subscribers to receive all messages from multiple publishers;
• services allow multiple clients to send requests and receive replies from a single server.

You will notice that both types of connections are asymmetric: topic subscribers and servers can't initiate communication (subscribers can't send messages at all). So this pushes us into a specific approach. These can't support arbitrary protocols, right?

Not on their own, but we could use a pair of topics or a pair of services to model a full-duplex connection:

Node 1          Node 2
   --- topic_12 -->    Node 1 publishes to node 2
   <-- topic_21 ---    Node 2 publishes to node 1

or

Node 1          Node 2
   -- service_12 ->    Node 1 is client of node 2
   <- service_21 --    Node 2 is client of node 1

In the first case, you basically send independent messages, in the second case each message gets a response as well. I also want to show a third topology:

Node 1          Node 2
   -- service_12 ->    Node 1 is client of node 2
   <-- topic_21 ---    Node 2 publishes to node 1

This is also full-duplex, and way better aligned to real-world needs. This approach comes rather naturally when designing a system:

• I (Node 1) want to give my robot (Node 2) commands, for this I need a service.
• I also want to know my robot's state, for this I need a topic.
• (and this approach, unlike the others, is easily extended to multiple clients)

That's the value behind the topic/service abstraction: although it seems to constrain you, it actually helps with the patterns that appear in the real world. Another aspect: topics and services are constrained to a single message type (or request/reply message type pair). How do you work with that? Easy, use separate services and topics for different kinds of data. What seems like a constraint at first is actually just pushing you into separation of concerns!

Approaches to IO

I'm currently looking at ROS which, despite its name (Robot Operating System), is neither an operating system nor at its core particularly specific to robotics. ROS includes libraries that help you model and program robots, but at its core, it is a framework for distributed applications.

A quick overview: a ROS application consists of several nodes (processes) that communicate via topics (broadcast) and services (remote procedure calls). Using a message definition language, you can define which data is sent between nodes. There's also a master node which provides name resolution, so that clients can find the topics and services they're interested in. Libraries can provide you with message definitions and utilities, and you will usually use third-party nodes in your application. I would say these nodes still count as libraries, but unlike a traditional library you don't call their code in the context of your process. Instead the node runs as its own process and you use topics and services for communication.

This may sound familiar because it's basically the same as a microservice architecture or probably dozens of other approaches to distributed applications. What follows is applicable to a lot of different systems, but I'll use ROS to present my code samples.

Topics, publisher:

import rospy
from std_msgs.msg import String

pub = rospy.Publisher('chatter', String, queue_size=10)
rospy.init_node('talker', anonymous=True)
rate = rospy.Rate(10) # 10hz
while not rospy.is_shutdown():
    hello_str = "hello world %s" % rospy.get_time()
    rospy.loginfo(hello_str)
    pub.publish(hello_str)
    rate.sleep()

Topics, subscriber:

import rospy
from std_msgs.msg import String

def callback(data):
    rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)

rospy.init_node('listener', anonymous=True)
rospy.Subscriber("chatter", String, callback)

# spin() simply keeps python from exiting until this node is stopped
rospy.spin()

The code is simplified and taken from the ROS Wiki. It should be pretty easy to understand: the publisher sends a message to the chatter topic ten times a second, and the subscriber registers a callback to handle these messages. There can be any number of publishers and subscribers on the same topic.

A different approach to the same problem might be a more traditional blocking I/O API:

import rospy
from std_msgs.msg import String

rospy.init_node('listener', anonymous=True)
sub = rospy.Subscriber("chatter", String, callback)

while not rospy.is_shutdown():
    data = sub.receive()
    rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)

This code is of course only hypothetical. The idea is that receive() would block until a new message arrived. What are the up- and downsides? One upside is that it is perfectly clear how the message is handled: right in the main thread. The original code only had a callback, and we had basically no idea where that is called. Does each subscription spawn a new thread? Are the callbacks called inside spin()? If so, what happens when we replace spin() by a busy loop? (for the record, my tests suggest one thread per subscription, but maybe ROS is smarter when the number of subscriptions rises. Or maybe not - another downside of the original approach?)

But of course there are upsides to the original code: for one, it hides the logic of receiving messages from the user. What you're interested in is the message payload, not the control flow you have to use to get it! And the most serious consideration: what if you have two subscriptions? You can't simply call receive for both, because it blocks. When subscription A doesn't get new messages, it won't let subscription B receive its messages, no matter how many there are.

This is a classic problem in blocking I/O, and the naive solution is multithreading: just block a dedicated thread that doesn't have to process anything else. For robotics, this is probably fine, but spawning and switching between operating system threads is relatively expensive and so not that appropriate for servers that operate under high load.

One step in the right direction would be a nonblocking receive:

import rospy
from std_msgs.msg import String

rospy.init_node('listener', anonymous=True)
sub1 = rospy.Subscriber("chatter", String, callback)
sub2 = rospy.Subscriber("gossip", String, callback)

while not rospy.is_shutdown():
    data = sub1.receive()
    if data is not None:
        rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)
    data = sub2.receive()
    if data is not None:
        rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)

Here, instead of blocking, receive will immediately return None if there is no message yet. If you haven't spotted it, the downside is that this is a busy loop: no matter whether there are messages or not, the CPU will constantly iterate through this code. In the blocking code, the operating system would just put the thread aside until the socket that receive is waiting for has new bytes. Now this is not the case and the OS will run the thread, even though it's actually blocked.

What we really want is some kind of multi-source polling. Let me show just one possible variant:

import rospy
from std_msgs.msg import String

rospy.init_node('listener', anonymous=True)
sub1 = rospy.Subscriber("chatter", String, callback)
sub2 = rospy.Subscriber("gossip", String, callback)

while not rospy.is_shutdown():
    rospy.poll(sub1, sub2)
    data = sub1.receive()
    if data is not None:
        rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)
    data = sub2.receive()
    if data is not None:
        rospy.loginfo(rospy.get_caller_id() + "I heard %s", data.data)

here we still assume receive is nonblocking, but before running the loop, we use poll. poll will block until data on either subscription is available. Afterwards, we still need to check the subscriptions ourselves, but performance-wise this works.

It turns out that this - at least the principle - is a rather useful primitive for the operating system to provide. It allows to build many different abstractions efficiently. receive is not enough, but poll is.

Software tooling

Learning a new programming language is hard. For the obvious reasons, but also because of tooling. (This post is mostly going to be a rant, ignore it if you're looking for practical information.)

I think nothing is more frustrating when learning a new language than not being sure where errors come from. Do you need to install another library that you're building against? Do you need to add paths to your environment? Are you passing incorrect flags to the compiler? Is your project definition missing some configuration?

And it goes on: What IDEs have good integration? Is the language or its community opinionated on project layout, documentation, testing? If not, are there still any best practices to follow? All this makes it hard to effectively use a new language at first, even if the language's syntax and semantics are familiar from the start. But all this should be trivial to take care of.

When I started programming (with Java), it was (as far as I could tell) the time of Ant build scripts. In its functionality, Ant is similar to Make, in that it defines rules for building specific parts of a project, and dependencies between rules to deduce which rules should fire and in which order. Unlike Make, Ant is Java based and its build files are XML (thus making Ant ugly to read or write).

When I heard about Maven, it was a huge revelation. Not because it made expressing build processes so much easier (or because it was less ugly - Maven still used XML build files), but because unlike Ant, Maven included what I consider the most important part of language infrastructure: dependency management! Tell Maven what libraries your project depends on, and it will download them and configure all paths so they're found. Nice!

So there's dependency management, there's building your project, and also importantly, there's installing the tools in the first place. I usually don't want to pollute my operating system by installing stuff when I have no idea what I'm actually doing.

Let's leave Java behind and look at Python. One major problem is the split between Python 2 and 3. They are largely incompatible and Python 3 has lots of cool stuff, but some software still requires Python 2. On the plus side, there are pyenv and virtualenv which let you install Python without root and Python libraries for specific projects, respectively.

Normally, your Python package will have a file setup.py which contains your project's description: name, version, dependencies, etc. As this is a Python file, you could think this is meant to contain build instruction, but that's not really the case. The file should contain a single call to setup(...), which uses the data provided in its parameters in a fairly declarative way. Further, in "modern" usage, you will likely use pip to handle dependencies. For example, pip install -e ./myproject will find ./myproject/setup.py and install the package (as editable, -e) as well as all missing dependencies.

So pip handles dependencies. What about building? Well, Python is interpreted, so normally not much. To make a "distribution" of your package, you can call python setup.py sdist and/or python setup.py bdist_wheel to make a source or binary distribution.

In JavaScript, the situation is more confusing. There are tools like bower, grunt, gulp, browserify, webpack, npm, yarn, etc. that all enter the picture, and tomorrow there will be even more. Also, there's three major ways to execute JavaScript: in a Browser, in Node and in Internet Explorer. But despite that, if you specify a package.json and use e.g. npm, you have a pretty good chance of distributing your packages to your users. (Does it show that I have not used JS as much as Python?^^)

My last look will be at Rust, because it's cool and is in a rather special situation: Rust is new, and its build tool, cargo, was developed from the beginning with the language and the rest of the tooling. It handles dependencies, building code and documentation, testing and publishing packages all in one. For installing Rust, there is the rustup tool, which fits approximately pyenv's profile: it lets you install and choose different versions of Rust. This is especially important as Rust has a six-week release cycle and a nightly channel that allows to use new unstable features.

Cargo is not required to build Rust, you can use the compiler rustc on its own as well, but usually there's no reason to go that way. That means that Rust is a rather homogenous environment (as long as you're not using unstable features) and you can focus on getting your code to compile at all; the rest of your environment will most likely work out of the box.

Interoperability

Looking back at my earlier posts, I'm surprised how utterly Java-centric everything I did was. Never mind things like bytecode instrumentation, of course that's JVM specific, and it's cool stuff!

But for other things, I think I would try to achieve more interoperability these days. Specifically, I'm looking at JGroups. It has some nice features in terms of consensus, making sure that all known nodes have the same view of the system, but looking at it purely as a networking library, it shockingly limits you to implement all your nodes in a JVM language.

I think there are generally three layers of interoperability. The JGroups example is one extreme, basically no interoperability. It has one Java implementation and its network protocols are specific to that implementation. I don't know the specifics of JGroups, but the protocol used may well be tailored to transmit very specific manipulations of JGroups nodes' states. In other word, there are lots of assumptions that make JGroups easy to implement in Java, but very, very hard to port to another language.

One kind of interoperability comes through dynamic linking. For example, through native methods, Java can call code written in C or another language that is compiled into machine code. As long as the JVM uses the ABI and calling conventions of the native library, access is no problem. The native code is executed in the calling process, so this kind of interoperability is strictly for libraries, not for client/server communication.

So if it was CGroups instead - written in C - then there could simply be a Java binding and a .Net binding etc., and all platforms using these bindings could cooperate. Unfortunately the reverse isn't true: the JVM is, quite obviously, a virtual machine, and you can't simply call code targeted at a VM. That VM first has to run, and then you have to interface with the VM before finally getting access to the library. This is true for all languages requiring a runtime environment (i.e. interpreted languages), but it gets worse the more complicated the runtime gets.

Which leads me to the third kind of interoperability: wire protocols, which solve the issue of having a protocol that is full of assumptions from the original implementation. Instead, the first step to designing the application or library is to design a binary or text based encoding and to clearly states what kinds of communication are allowed, in what order, at what time, etc. The protocol has to be simple and explicit enough to write down, so it should also be possible to implement in different languages. That doesn't allow you to use one library in different contexts, but it makes it feasible to port the library for different platforms.

If it weren't for that last kind of interoperability, the Internet would be a very different place. Imagine if websites used a browser-specific protocol instead of HTTP! Microservice architectures also rely on well-defined protocols to connect otherwise incompatible systems: of course, a service written in Java can expose some JGroups-derived functionality via, say, HTTP, and then any other application can access that service.

Blogging on a Time Budget

I've never been regular in writing blog posts, and in the past years I never found the time necessary to do serious Magic programming. I enjoy thinking about general ways to model the Magic rules in a program, but without a serious amount of time, it will always remain tinkering. It's an interest, but not serious enough to resolve to put out thousands of lines of code to make it work.

However, I'm still a passionate programmer and learner. While most of my posts so far have had a Magic context, they were always about programming. And this is where I want to shift the focus: non-MtG posts aren't off-topic, and that should encourage me to write more about what I'm currently doing. I know I won't start spewing out post after post, but as long as I find the time, I shouldn't run out of topics.

So what have I learned about in the three years since my last blog post?

I have shortly continued with Scala, but got demotivated by the compile times. A lot of time was spent working with Python, and it's currently my most fluent language. I have written web applications with Django, learned about generators and async/await, and constructed protocols and servers using ZeroMQ.

There has also been a considerable amount of JavaScript and some TypeScript, mostly using Vue.js to create reactive frontends for my web applications. I have also looked at web-technologies, such as Webpack and, although briefly, Websockets, PWAs (particularly service workers) and WebAssembly.

Recently, I re-discovered Rust after playing with it years ago. In terms of languages-to-use-if-you-want-to-get-things-done-in-a-reasonable-amount-of-time, Rust is about as far away from Python and JS as you can get (yes, there's also C and C++, but I don't think they're farther away, just slightly worse at getting things done... just my opinion though).

I intend to write about most of these things, if they're still relevant when I get around to them. If there be still readers here, let me know with what to start.

Oh, and I have watched tons of programming videos on YouTube! There's lots of great content. But there are also videos that suffer from what I call "boring tutorial syndrome": they try to teach good programming style (or something), but actually don't go far beyond how to write an idiomatic Hello World program (or whatever the basics of the respective topic are). The problem is that by starting from scratch, these tutorials don't manage to reach a reasonable depth. So whatever I write, don't expect start-at-zero tutorials, expect food for thought and self-study!

PS: if you're looking for videos that don't suffer from the boring-tutorial-syndrome, try LiveOverflow

Looking at IntelliJ - not the best start...

In a recent blog post, I wrote that IntelliJ IDEA is a popular choice for Scala developers. I wanted to try for myself, but had no luck - just as many others. While there are people who successfully code in Scala using IDEA 12, the most recent version 14 seems to have major problems in this regard. The bug I encountered was this one - unfortunately a show stopper; I can't even properly add the Scala SDK to IntelliJ. Others, who probably upgraded their install from a previous version of IDEA, seem to come a little further, and then run into other problems, like Gradle integration - which I want as well.

So, no trying IntelliJ with Scala for now. I still want to give it a try, however. With Laterna being Scala-only now, it will stay with Eclipse for a while though.

I think I'll try it with Android, as the official IDE for Android was recently migrated from Eclipse to IntelliJ. So it can't be just bad ;)

What is a card?

This post is about ambiguity, but this post is not about Ambiguity. (Don't forget to look at the card topsy turvy as well. And don't forget to look at the card Topsy Turvy as well)

Okay, enough with the puns... at least if I don't find any others while writing... what I really want to write about is how overloaded the term "Card" is in Magic. Let's see how many different meanings I can come up with.

Cards are those things that Wizards designs. For example, Grizzly Bears is a card, and Llanowar Elves is as well.

What about two Grizzly Bears? A card can also mean the physical object, sold in boosters and put in decks. But you can't even simply say that the physical card is "a Grizzly Bears", because there's a few different ones. And there's foil as well, and probably other things I'm forgetting. So the physical card is actually not an instance of what Wizards R&D designed, but of one translation of one printing of such a card, either in foil or not.

Getting to the actual game, cards are a kind of object, in contrast with tokens and copies of cards, which are not cards but serve almost the same purpose.

Noncard objects exist in a few different forms: emblems are always in the command zone; tokens can only exist on the battlefield; ability objects exist on the stack; copies of spells exist on the stack; casting a copy of a card creates that copy in the original card's zone, which in then put onto the stack as a spell while casting.

Permanents are objects that can be either cards or tokens, so another thing a card can be. Compared to other objects, permanents have state:  tapped/untapped, flipped/unflipped, face up/face down, and phased in/phased out.

The comprehensive rules on spells are worded a little strangely: "A spell is a card on the stack", "A copy of a spell is also a spell", "if the player does [cast a copy of a card], that copy is a spell as well". It sounds contradictory at first, but it's manageable.

So what have we got here:

• What Oracle says a card is, identified by the English name of the card.
• A printing of such a card, identified by English name together with an expansion. This extends the Oracle card by rarity, artist, illustration, collector number, card frame style, guild symbols and similar in the text box, ...
• A translation of a printing, identified by a multiverse ID, or alternately by English name, expansion and language; I wouldn't trust that the translated name is unique. The translation adds obviously anything language specific about a card. This includes printed, non-oracle wording including reminder text, as well as flavor text.
• A piece of cardboard that has a Magic card printed on it. To my understanding, this interpretation of "card" can be uniquely identified by a function defined on an interval of the timeline that maps every instant in that interval to the volume of space that is occupied by the card at that time. Or, a card is a card.
• A digital representation of such can also be considered a card in that sense.
• A card is an object that can be in a zone in a game of Magic.
• Some permanents are (represented by) cards.
• Some spells are (represented by) cards.

You can imagine that all this ambiguity makes it quite hard to come up with a proper software model for a game of Magic!

Ordering of replacement effects

Handling of multiple replacement effects is intricate, but well described in the comprehensive rules, so I'll link you to my favorite online version: rule 616

The code I presented in monday doesn't handle any of that, but it wouldn't be too hard to do. In my quick mockup, effects are stored in a list and applied in that order. For example:

Boon reflection: If you would gain life, you gain twice that much life instead.
Nefarious Lich: If you would gain life, draw that many cards instead. (and other abilities)

Clearly a life gain could be replaced by either, but only one of the resulting events is a life gain in turn. So, if I gained two life, depending on the order, I'd draw either two or four cards, and it should be my choice.

So how would that be written? Almost exactly like the comprehensive rules are formulated: Starting with the set of all replacement effects,

1. filter the set for those applicable to the event
2. if the set is empty, we're done
3. filter further
• for self-replacement effects
• if the result is empty, for "under whose control an object would enter the battlefield" replacement effects instead
• if the result is empty, for "cause an object to become a copy of another object as it enters the battlefield" replacement effects instead
• if the result is empty, don't filter at all
4. if there's more than one effect, let the applicable player choose one effect
5. replace the event and remove the effect from the unfiltered set
6. repeat

a partial function already has isDefinedAt, and filtering sets is easy in scala, e.g. effects.filter { _.isDefinedAt(event) }. I can even use groupBy to partition the effects into their four types in one line. What needs thought is how to ask the players for ordering effects.

 Edit: here is my new vesion for replace:

//applies all effects to the event.
@tailrec
def replace(event: Event, effects: Set[ReplacementEffect]): Event = {
  val filtered = effects.filter { _.isDefinedAt(event) }
  //TODO here I need to know what type an effect is
  //0... self-replacement, 1... control, 2... copy, 3... regular
  val byType = filtered.groupBy { effect => 3 }
  val sorted = byType.toSeq.sortBy { case (k, _) => k }
  val choices = sorted.collectFirst { case (_, v) if !v.isEmpty => v }
  choices match {
    case None => event
    case Some(effects) =>
      val effect =
        if (effects.size == 1) effects.head
        else {
          //TODO let player choose one effect
          effects.head
        }
      //only applicable effects here, so use apply directly
      replace(effect.apply(event), effects - effect)
  }
}

The only change to my description is that I don't do step 2, but instead do it in step 4. The collectFirst call returns the first nonempty collection, or None if they're all empty, so I get that check here for free anyway. Also note that I don't use a loop, but instead use recursion, which is preferred in functional programming. Some types of recursion, like this, can be optimized into a loop, so here it's just a matter of style. In fact, by adding the @tailrec annotation, the compiler would issue an error if this couldn't be optimized into a loop.

Replacement effects with partial functions

This is a very crude mockup of how replacement effects could look in Laterna Magica. Instead of any impressive effect, I'm just modifying a variable that represents a player's life total, but it does quite a lot for only being about 30 lines of code:

//a replaceable event
trait Event { def execute(): Unit }
//an effect that replaces an event
type ReplacementEffect = PartialFunction[Event, Event]

//applies all effects to the event. If a replacement doesn't match, the
//event remains unchanged
def replace(event: Event, effects: ReplacementEffect*) =
  effects.foldLeft(event) { case (event, effect) => effect.applyOrElse(event, identity[Event] _) }

var activeEffects: Seq[ReplacementEffect] = Nil
def execute(event: Event) = replace(event, activeEffects: _*).execute()

//example: A player's life points
var _life = 20
def life = _life

def gainLife(life: Int) = execute(GainLife(life))

case class GainLife(life: Int) extends Event {
  def execute() = _life += life
}

//execution:

//gain three life
gainLife(3)
println(life) //23

//gain twice as much life
activeEffects = List({ case GainLife(x) => GainLife(x * 2) })

//gain six life
gainLife(3)
println(life) //29

The trait Event is used to represent replaceable events, and effects are simply "partial functions": functions that can only be applied to some of the values that its parameter type would allow. For example, at the bottom there's the partial function:

{ case GainLife(x) => GainLife(x * 2) }

While ReplacementEffect is defined for all Events, this one only accepts GainLife instances. Using PartialFunction.applyOrElse, I handle the cases where a replacement effect does not apply to an event.

The example is self-contained, so you should be able to run it yourself. In any case, the output is next to the print statements: The first time, you only get 3 life, but after registering the effect, the amount is indeed doubled. In reality, execute (and replace?) needs to be a bit smarter to recognize the right effects to apply, but otherwise this could stay almost as it is.

Scala revisited: so much shorter!

I didn't even realize that I wrote about Scala before, until I logged into my blog and saw that it's apparently one of my most popular posts. My earlier flirt with Scala ebbed out rather quickly, and I think the reasons were twofold.

Firstly, back in 2010, IDE integration for Scala wasn't that great. While there's still much room for improvement, the situation is much better now. I only tried the Eclipse Scala IDE, and from what I heard, IntelliJ integration is nice as well.

The second reason is simply my lack of understanding of functional programming. I would say that I had a good concept of how functional programming is supposed to work back then, but I was surely lacking practice. My interest recently led me to a FP/Haskell class at university, and while it was only introductory level, I think it helped a lot.

As far as I know, the language itself has evolved quite a bit as well. There was a big cleanup of the collection API, and much more - it has been five years, after all. How time flies!

Anyway, I'm in love with that language! In the last three weeks, I converted Harmonic (and other projects) from Java to Scala - and from Maven to Gradle, by the way - and got from 1027 to 870 lines of code, about 15% less. Granted, that's not exclusively the language switch but also a partial redesign - hence the three weeks of work - but I don't think that lessens the message. It gets more impressive when I add that the Scala code includes about 300 lines of freshly written unit tests!

Scala gets rid of a lot of boilerplate code that bloats the line count of Java. For example, a typical property takes 7 lines of Java:

private int property;
public void setProperty(int property) {
    this.property = property;
}
public int getProperty() {
    return this.property;
}

Converted to Scala idioms, the code would look like this:

private var _property: Int = _
def property = _property
def property_=(property: Int) = _property = property

Well, that is a little shorter. Also note how def property omits the parentheses, and that methods ending in _= are called as if they were assignments. Now compare incrementing that property:

//Java
x.setProperty(x.getProperty() + 1);
//Scala
x.property = x.property + 1


That's just one example, I could go on: Instead of static members, Scala has objects, but they are actually singletons and can extend other classes and traits. Traits are like interfaces, can have method implementations like in Java 8, but are actually inherited as mixins. Case classes allow pattern matching, comparing that to switch is almost insulting given how much more flexible pattern matching is. Implicits allow the compiler to do even more of the work for you: implicit conversions are like user-defined autoboxing (another insult!), implicit parameters save you typing when a context variable is needed over and over in method calls. Need to pass a Game to every other method? Make it implicit!

I bet there are many more gems hidden in Scala, but I only had three weeks to look so far :P

Okay, one more thing. I won't even describe it, just show you code. This is how unit tests can look in Scala:

class EngineSpec extends FlatSpec with Matchers with GivenWhenThen {
  behavior of "An engine"

  it should "handle branches properly" in {
    Given("an engine")
    implicit val engine = new Engine()

    And("a PolybufIO for a custom action")
    engine.addIO(MyAction)

    {
      When("creating a branch at HEAD")
      val head = engine.head
      val branch = engine.Branches.createBranchHere("branch1")

      Then("the branch's tip should be the head")
      branch.tip should be(head)

      When("executing an action")
      engine.execute(new MyAction())

      Then("the branch's tip should still be the old head")
      branch.tip should be(head)
    }

    {
      When("creating another branch at HEAD")
      val head = engine.head
      val branch = engine.Branches.createBranchHere("branch2")

      Then("the branch's tip should be the head")
      branch.tip should be(head)

      When("making that branch current")
      engine.currentBranch = branch

      And("executing an action")
      engine.execute(new MyAction())

      Then("the branch's tip should be the new head")
      branch.tip should be(engine.head)

      When("moving that branch's tip")
      branch.tip = head

      Then("the new head should be the branch's tip")
      engine.head should be(branch.tip)
    }
  }
}

Next Step: Uno

I'm posting this here so that I have one reason more for not slacking off: the next thing I'm going to try is to make a game of Uno based on Harmonic. Why another dummy instead of Magic? Because, while Uno is of course much simpler than Magic, they share a few features that make it interesting as a prototype. And as Uno is much less complex, it's faster to spot where there are still features missing.

So what are these similarities? I identified the following few:

• it's a card game, duh
• unlike TicTacToe, Uno can have more than two players
• the turn order in Uno can change
• there are multiple zones in which cards can reside; the deck is hidden, the discard pile is public, and the hands are private to each player
• Uno has randomness for shuffling the deck
• some cards require additional decisions by the player

I also found that at least two things are very different from magic - at least in regard to Uno's applicability as a prototype: there is no such thing as tokens, i.e. no new objects are introduced into the game after it starts; and there is no significant strategic part to Uno, so testing the feasibility of an AI is hard. Still, Uno should be really easy and still provides a lot of the things I was hoping to find, so that's where I'll go next.

Multiplayer-TicTacToe

If you read my previous post and/or looked at my GitHub account, you already know that I'm programming a small game of TicTacToe to demonstrate and debug my state synchronization library, Harmonic. At first, I had two engines, where one actually played the game, and the other simply listened. Both engines were located in the same JVM, so identifying them was easy.

At that time, I didn't even have states yet, so the communication exchanged the actual action. This was a problem, because although both engines executed the same actions, there was no way that the application could really know. Only later, when the communication consisted of states, this could really be called synchronization.

The real fun began when I added a notion of branches. A branch points at a state, and can move to other states as well. An engine can publish changes to a branch, and the receiving engine will then ask for the missing states. Branches also make it possible to delve into alternate histories, for example for AI.

Although states and actions are defined as protobuf messages, nothing I described so far contained any networking or IO code. The BranchManager class contains three methods that do communication, but all they do is call a callback method and leave the actual networking to whatever the implementation of that callback sees fit. I added an implementation using JGroups; the same library I used in my last attempt.

And now, I have a graphical multiplayer Tic Tac Toe. Granted, it does not enforce that every client plays only his own pieces, but that's not the point and would be easy to do; it just requires a more sophisticated new-game negotiation. What does work out of the box is spectating. A newly started client does not yet ask for the current state, but as soon as one of the playing clients performs an action, the spectator is also updated.

Right now, to try it, you have to build Polybuf, Harmonic and TicTacToe yourself, which is not optimal. TicTacToe is meant to help me experiment, find out what's good practice and what needs improvement. When it comes to distribution, applications have different requirements than libraries, and fulfilling these requirements is the next thing on my schedule.

Okay, that's fixed now! I have added a git repository that contains a Maven repository. That means you can reference my libraries (for example, in building TicTacToe), and the dependencies can be downloaded from there, without the need for you to build them yourself.

The repository also contains a TicTacToe-0.1.0-jar-with-dependencies.jar, which... well. It also has a MainClass attribute, so it should be reasonably double-clickable. I have only tested it on one machine, and I don't know how the default JGroups-Stack behaves in LANs and WANs, but fixing issues arising from that is a matter of passing a configuration to a constructor...

Harmonic: Game replication and git?

So I took some time off of trying to implement game replication. Instead, I did some other stuff (like university), and gathered some experience with git. With time, I realized that git, and distributed version control in general, provides exactly those concepts needed for the model of replication I wanted. Looking back at an earlier post on that topic, I can't really say how I expected it to work out. In the end, it turned out that transmitting "unfinished transactions" was a really bad idea; in the end, the very essence of a transaction is that it executes either as a whole or not at all.

Well, anyway, now I had experience with git, and it contained exactly those concepts I needed for my replication as well:

• a repository of files (objects) in which changes are tracked
• a history of commits, where commits have a parent/child relation. The main attribute of that relation is what changes happened between those commits
• branches which point at specific commits and provide a comfortable way of traversing and manipulating the history
• definition of remote repositories for synchronization of branches

In contrast to the transaction approach I pursued earlier, where the parent/child relationship is in regard to application logic and does not really add to the synchronization capabilities of the framework, this version control approach is based around a tree of commits, where the parent/child relationship is a temporal one, and essential to the framework. The framework is less cluttered with concepts that are not important to it.

Of course, a few things are different: a git repository manages files, essentially byte sequences, while a replication framework manages objects. While these could be interpreted as byte sequences, it doesn't really add to the usability, because objects are usually manipulated on the level of fields and methods. Furthermore, files are organized in a file system with human-chosen names for every single file. Objects simply reside at some location in memory not even the program can control; some other sort of identification is needed. And finally, for file versioning, a human user can try to resolve conflicts. For replication, this is not possible. Conflicts can't be resolved automatically, and as different object store different data, even manual merging could result in inconsistent state.

Luckily, besides the last problem, these can be solved. And for the last problem, usually it shouldn't be a problem in the first place, at least when talking about Magic. Magic is not a game of reaction time, where all players try to do something at the same time, and in the end someone was the earliest. Magic's system of priority defines very clearly which player may take an action at what point, so the game logic can ensure that conflicts do not arise, simply by obeying the rules.

You can take a look at Harmonic if you want. I'm not happy with the name, but I couldn't come up with anything better. I wanted it to sound like multiple independent systems are in sync, like vibrating at the same frequency, or looking the same; I don't think I succeeded, but that's okay as long as it works.

An Entity compares to a file. Entities are objects managed by Harmonic and are identified by a numeric ID. IDs are created by simply incrementing a counter, so creating IDs in different instances of Harmonic is consistent.

An Engine compares to a repository; it is a store of entities. An Engine also manages the different states (analogous to commits), and its current state: in a repository, the files always reflect one specific commit. In the engine, the entities always reflect one state, and the engine knows how to move between the states. Engines are identified with random IDs; as engines are generated on different machines, there is no way to ensure that engine IDs are always different. So, random IDs are used.

States, as mentioned above, compare to commits. Git uses random commit IDs - at least they look random. In Harmonic, the engine ID is part of the state ID. Assuming the engine IDs are different, it's enough to simply increase the ID for each subsequently created state.

Last but not least, Actions and Modifications correspond to the changes that make up a commit. A modification is a single change that is easily reverted, like setting a variable (or creating a new entity ID). An action is a sequence of modifications, so the action can be reverted by reverting the modifications making it up.

In this model, states, containing actions, are transmitted between engines, but modifications are not. Actions are defined by the application, so they may be as small as needed and as big as possible. The modifications making up the action are only created during execution, so they don't need to be transmitted, making large actions more efficient. For example, instead of transmitting that a card is tapped and mana is added to a mana pool, it's now enough to transmit that a mana ability of a land was played.

I like the direction where this is going. I think distributed source control is an elegant concept, and I don't think I did a bad job at catching its essence, so this should turn out to be pretty usable. In fact, while developing Harmonic, I also wrote a game of Tic Tac Toe that uses it, so I have a proof of concept that Harmonic really works as intended in a turn based game.

git-flow

Check it out. Do. The "A successful Git branching model" post has, as far as I can tell, gone viral. I stumbled upon it some years ago, when git was almost new to me. I felt that it was a good idea to have some structure in your branching. Intriguing was the idea that, whenever you check out a repository, the head of the master branch is something stable, something you could immediately build and use. You don't have to search the history for an earlier commit where the code did work; if it's on the master branch, it works.

And I wasn't the only one intrigued by the branching model. Someone wrote a set of additional git commands that allows to use the high-level "git-flow" concepts such as "start feature" and "finish release" instead of initial branch and merge commands.

Despite the benefits, I wasn't really comfortable with git on the command line. I'm still not. I feel that for efficient code versioning, a concise representation of changes is essential. I might be hacking along, not sure whether the code is worth committing, before finally saying, "yes". Then, I start up my git tool of choice, SmartGit, and review what changes I actually made, figure out what that means, stage individual changes that are logically related, and commit them with relevant commit messages. My personal feeling is that I'm more productive with a graphical tool.

But recently, SmartGit started to support git-flow out of the box, and I'm happy to be able to finally use git-flow comfortably. Really, go ahead and read the original post and try git-flow or SmartGit. You'll like it, and I'll like it if I check out your repository and have runnable code right in front of me.

I'm back(?) and polybuf

I hope I am!

I was just thinking I was in the mood to write something for my blog. And I have things to talk about. I have some new code online at github; Polybuf and Harmonic might be intersting to you.

Mind you that I don't have any Laterna Magica code online there. I'm not sure about LM's future, besides that it will definitely have one! Looking at the history, it seems it was over two years ago when I added the last bit of real functionality. Somewhere in that time frame, I realized that the whole architecture of LM wasn't able to address all the stuff I wanted to get out of it - and that stifled the fun I had with the project.

But LM was always in the back of my head. I tried to figure out what I wanted, what I needed to do so, and what was fun for me to make at that moment. And here I am, writing a new blog entry, because it's fun to me. I hope it will stay fun to me, so I hope I'm back!


Now to make this post not entirely meta, let's talk about protobuf and polybuf.

Protobuf is a protocol library which is apparently widely used inside Google. You specify a protocol using .proto files, and protobuf creates code to read and write that protocol. It's basically a replacement for serialization, and provides code generators for different target languages, so that programs written in different languages can exchange data.

The protobuf code generator supports options that let you control how the code is generated; without losing protocol compatibility, you can code that is more or less optimized for memory footprint, code size, parsing time. Protobuf is only one serialization replacement library of many, and depending on how you run your benchmarks, you probably get different results as for which is the fastest. Anyway, protobuf definitely has the features I need, whether or not it's the "best", and it's supposed to have good documentation. I can't complain so far.


One thing I didn't like about serialization from the beginning is that you throw the Serializable interface right into your code. Exporting and importing state is a very specialized aspect of a software, and it's likely not directly related to the core classes of your software. The thought behind serialization is that adding the implements Serializable clause immediately enables you to serialize your objects - but that's only true if your object really correspond to whatever you would want to transmit over the network or store on this. Any discrepancies mean that you have to work around this central concept of serialization, often making the code further less readable, and further clogging your classes with logic that does not correspond to their original concerns.

And after you're finished with all this, you have a class that has logic for one specific serialization mechanism. If you want to switch that mechanism out, you have to plumb directly in your application logic classes, instead of replacing some auxiliary import/export classes.

And there comes the fact that Serialization semantics are... funny to begin with. Deserialization does not invoke a constructor of the class to deserialize. It creates an empty object without using constructors and then populates the fields manually from the stream. Deserialization can even modify final fields. But, after that, you yourself can't, even using the mechanisms that deserialization provides.


Protobuf (and its likes) kind of work around these problems. You're not serializing your logic objects, but dedicated messages that have well defined serialization properties. On the receiving end, you can restore your object using whatever constructors and methods you seem fit. That seems like more work, but only in the short run; your benefits are familiar semantics (constructors and methods), separation of concerns, easier reaction to changes (through the separation, and the fact that protobuf was created with the explicit goal of compatibility between protocol versions), and of course improved performance through dedicated code instead of reflection.

One thing, by the way, that protobuf does not support, is polymorphism. The reason is simple: there are other patterns in protobuf that enable the same functionality, and polymorphism is a feature handled differently in different languages (Java has no multiple inheritance, and C obviously has no inheritance at all). Making Polymorphism a feature on that level limits the interoperability of protocols.


And that's where polybuf comes into play, and where I'll finally show some example code. Polybuf is essentially a convention on how to format messages that contain polymorphic content, and a collection of classes that facilitate deserializing these messages. At the core is the polybuf.proto file, showing the relevant parts:

message Obj {
  optional uint32 type_id = 1 [default = 0];
  optional uint32 id = 2 [default = 0];
  
  extensions 100 to max;
}

Proto files contain message definitions. The messages declare fields that may be required, optional or repeated. required is actually not recommended, because it makes changing the message slightly harder. This message features only one data type: uint32. Protobuf encodes these as compact as possible; small numbers will take less than four bytes; big numbers may be larger. For values where big values are expected, there are other fixed length encoding types. Every field has a numeric tag that is used instead of encoding the field name. Fields can be removed and added without breaking compatibility, as long as these tags are not reused - at least on the protobuf level. Of course, the application must be able to process messages where some fields are missing; that's what the default values are for. These are not transmitted. The receiver simply fills in the values if the field was not present (very possible for optional fields).

Below these two fields, there is an extension declaration, and now it becomes interesting: 100 to max specifies that all field tags greater than 100 are free to use for extensions. There is an example directly in the file:

//  message Example {
//    extend Obj {
//      optional Example example = 100;
//    }
//    
//    optional string value = 1;
//  }

The Example message declares an extension to Obj, which is simply a set of additional fields, labeled with tags from the extension range. It has nothing to do with and extends clause, and is pretty independent from the Example message; it just happens to be there for logical grouping. The only consequence from the placement is the namespace used by the extension.

The Example message itself declares a field value, plain and simple. Now, where's the benefit? Imagine there's a class Example in your program and you expect it to be subclassed, but want to still support protobuf serialization. This structure allows you to

• reference an Example by declaring the field as Obj,
• create an Example by setting type_id to 100 (the extension tag used by Example), and filling the example extension field with an Example message,
• create an ExampleSub message that uses a different extension tag, and simply fill both the Example and ExampleSub extension fields. From the type_id field, you can see that it's an ExampleSub object, and since you know the ExampleSub class structure, you know that there will be an Example message containing the superclass fields.
• The same goes for multiple inheritance: If you know that the type_id-referenced class has two superclasses, simply read both superclass extension messages from the Obj.

Of course there's some Java code supporting this message structure to give a more serialization-like experience. Look at it. Input and Output are similar to the streams you're used to; however, they don't convert objects into bytes and vice versa, they convert between objects and Objs. From there, it's only a small step to networking and persistence. Config stores the IO objects that actually translate between objects and the corresponding extension messages. The Config is reused for subsequent Input and Output uses.

The IO interface is the most interesting one. Its three methods specify clearly the semantics that polybuf expects for (de)serialization, making it easy to understand what's going on. It extracts the logic from private methods in the class itself into a place that is actually meant to handle that concern, and gives the programmer full control over what data is (de)serialized, what constructors and methods should be called. And last but not least - these IO classes support inheritance! Deserializing a subclass can be as simple as subclassing the superclass IO, add your own subclass initialization and persistence, and delegate to the superclass IO for fields that belong to it.

Properties and relationships - it won't get easier

Okay, now to show you the peak of what I've done using ASM: Look at these classes:

@PropertyFactory("p")
public class Test {
    protected final PropertyChangeSupport s = new PropertyChangeSupport(this);
    protected final Properties p = new BoundProperties(s);
   
    @Property
    private String s;

    

    public void setS(String s) {this.s = s;}

    public String getS() {return s;}
    //add/remove PropertyChangeListener methods
}

//test code

Test t = new Test();

t.addPropertyChangeListener(...); //prints to System.out

t.setS("hello"); //listener fires!

With the right setup (and with setup I don't mean things you have to do every time; it's just either configuring a ClassLoader or transforming the bytecode directly after compilation), it's as simple as that to add property change support, and other things; in fact everything that can be done with a Properties object. (in case you never heard of that creation, look at the code) And there is another thing I have done using properties: bidirectional 1:1, 1:n and m:n relationships, which is pretty nice if it's as simple as this:

@PropertyFactory("p")
public class A {
    protected final Properties p = new BasicProperties();
   
    @OneToOne("a")
    private B b;

    //getters and setters
}

@PropertyFactory("p")
public class B {
    protected final Properties p = new BasicProperties();
   
    @OneToOne("b")
    private A a;

    //getters and setters
}

//test code

A a = new A(); B b = new B();

a.setB(b);

System.out.println(b.getA() == a); //true!

I'm skipping the other cardinalities, and the two examples here were a little modified, but they are functional! It's not simplified beyond what's necessary for the code to actually work as advertised. If you want to take a look at the complete test cases, look here. It also includes a ClassLoader that can do the job, as well as the setup for junit that invokes the test code on the modified class.

Okay, the advertising is over; one or two sentences about the internals:

• The class loader that is used prints all classes it loads (that is, the test classes) to a directory structure analogous to java's package structure. The printed text is the disassembled bytecode that is actually loaded - so not the same code as the compiled class files, and even some classes that are not present in the sources. If you can read bytecode, that's perfect to see every detail on how a simple this.b = b; comes to setting the private A a; in the newly referenced B instance.
• The relation annotations create a new attribute, for example OneToOne<A> a$o2o; (with default access). This is essentially the field you would name a if not making your life easier with ASM.
• That field is accessed by a class like B$$a$o2o_EntityFunction (For the OneToOne field named a in class B). It implements a simple interface and implements only one method, taking the B instance and returning the OneToOne object. Since the class is in the same package as B, the access is allowed, and any classes can be related without exposing any normal-java accessible fields for other classes.

I hope you can get your benefits out of this little gem!