Juggling & gluing with Extinct

Erlang/OTP is considered by its insiders as a fantastic coordination language. That means it makes a great glue for systems that need to juggle many moving parts: parts written in languages, other than Erlang.

It’s good at juggling because a concurrency model was built right into Erlang from day one. They chose asynchronous message passing as the means to perform such trickery, which makes OTP a great tool for gluing polyglot systems into one.

Interportability is something Erlangers should care about for the obvious fact that our community is small, and libraries can be quite mature in other communities that are teeming with people and activity.

Say, for example, there’s a powerful Machine Learning tool written in Python. Perhaps hundreds of programmers have contributed to years of development of this tool. Needless to say, it would be silly to rewrite its equivalent in Erlang. Rather, by leveraging one of the many Erlang interportability options, we can just interface with this Python library and get on with our day.

What is Extinct?

Extinct is a new tool written in C that digests OSC messages and spits out that same data in Erlang’s ETF (external term format). It’s a tool designed to mitigate the headache involved with connecting programs together. Extinct relies on Erlang’s ‘ei’ library to encode the ETF and also uses various open-source code to digest incoming OSC messages.

(OSC is a simple yet expressive protocol which can be used like one would use JSON and XML. The OSC protocol can be learned about here.)

The goal with Extinct is to treat your OTP hub system as a first-class citizen, and it’s quite accommodating for that reason. After your hub system executes the code of another language, by simply piping the yields of those programs through Extinct, your hub system gobbles up that data in ETF. Your OTP hub system doesn’t need to do any decoding — it just needs to do a single erlang:binary_to_term/1 and presto, the data is in your hands in easy-to-use Erlang terms.

Getting started

The public repository for Extinct is here:

https://bitbucket.org/nato/extinct

After running make, one can run programs that output valid OSC messages and simply pipe the output to the extinct executable. The executable can be found in ‘priv/bin.’ Here’s an example to drive the point further:

./runs-then-outputs-osc.sh | path/to/priv/bin/extinct

The extinct output is a binary ETF representation of the original OSC data, and with that, you’re half the way there.

The Erlang side

It’s assumed that anyone needing to use Extinct has a good grasp of Erlang, so with that, just a few words about the approach on the Erlang half of things.

If you take a look at ‘src/extinct.erl,’ you’ll be greeted with some familiar-looking textbook Erlang that is ready to digest ETF via Extinct using erl_interface — a methodology that mixes Erlang Ports with Erlang’s ‘ei’ library.

The Erlang Extinct source files (located in ‘src’) is a good place to peruse. Since everyone’s needs are different, the code in there is just for reference.

You can run an Erlang Extinct sample via make sample after the initial make has been performed.

Things to consider

This project is in its infancy, so be prepared for additional functionality as time goes by. For example, at the time of this writing, the only data types on the Erlang side are binaries, strings, integers and floats. This will improve over the coming months to include more data types.

Again, the OSC protocol can be learned about here. The next section is a breakdown of the types that Extinct supports.

Supported incoming OSC types

• b: binary blob
• f: float
• d: double
• i: int32
• h: int64
• s: string
• m: midi
• t: timetag
• T: true
• F: false
• I: infinitum
• N: nil

This list is more or less complete.

Supported outgoing ETF types

• binaries
• bitstrings
• integers
• strings
• floats

This list is a work in progress.