Erlang, the Unix way

Joe Armstrong once said that ‘Java runs everywhere. Erlang runs forever.’ For anyone new to Erlang/OTP, this is a beautiful reduction.

When used in anger, OTP is supposed to be started once, and never, hopefully, fall over. This is in the culture of this wonderful language, and that’s why when one uses Erlang in other ways, everything just feels a little off.

Links in a chain

Recently, I wanted to utilize Erlang’s robust support for parallel computing in a project that was mostly comprised of Unix scripts. I wanted to let Erlang handle the chunk that could be parallelized, then hand it back to the calling script after it was through.

The problem was, OTP’s ‘escript’ methodology wasn’t going to cut it. I needed it to be stateful (like ‘gen_server’) and wanted it to feel like a service. I didn’t want to use sockets; I simply wanted my Unix script to call my Erlang service. So with a little mishmash, this is what I came up with.

Overview

First, the interface for my Erlang service will be a small ‘escript’ named ‘cb.escript.’ This will be callable from a Unix shell script, and capable of taking args (a must), plus returning values if needed. Here is that code:

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -Bd -setcookie mnbvcxz -sname cb_script@foo

main([A, B]) ->
    ok = rpc:call(cb_system@foo, client, run, [A, B]),
    ok = rpc:call(cb_system@foo, client, clear, []),
    ok.

Not a heavy lift at all! You’ll notice the little nicety of passing OTP flags on the third line. An invaluable feature.

Next: the fake ‘gen_server’ which I named simply ‘server.erl.’ It’s a stripped down — albeit lacking — server, that is generic in that it can assume the roll of any server. TFTP, Key-Val, anything, actually.

It looks as follows:

-module(server).

-export([start/3, stop/1, loop/4, call/2]).

start(A, Fn, Ld) ->
    Pid = proc_lib:spawn(server, loop, [A, Fn, Ld, []]),
    erlang:register(A, Pid).

stop(A) ->
    Pid = erlang:whereis(A),
    erlang:exit(Pid, kill).

call(A, B) ->
    Pid = erlang:self(),
    A ! {Pid, B},
    receive
        {A, error} ->
            erlang:exit(server_error);
        A, {ok, U}} -> U
    end.

loop(A, Fn, Ld, Buff) ->
    receive
        {Pid, U} ->
            Buff1 = trim([ U | Buff ]),
            maybe(Pid, U, {A, Fn, Ld, Buff1})
    end.

maybe(Pid, A, {_, Fn, Ld, _}=B) ->
    U = (catch Fn(A, Ld)),
    maybe1(U, Pid, B).

maybe1({'EXIT', A}, Pid, {B, _, _, Buff}) ->
    Pid ! {B, error},
    error_logger:error_report([{server_error, {A, Buff}}]);
maybe1({Reply, Ld}, Pid, {A, Fn, _, Buff}) ->
    Pid ! {A, {ok, Reply}},
    loop(A, Fn, Ld, Buff).

%%
%% support routines
%%

trim([ A, B, C, D, E | _ ]) -> [A, B, C, D, E];
trim(A) -> A.

This server is probably overkill, even, for our purposes here. But you never know when you’ll want to, say, stop the server gracefully, or get a report on the last five requests.

The server’s specifics are left out, and placed in a ‘client.erl’ source file that will interface with the server, and the outside world. It’s as follows:

-module(client).

-export([start/0, stop/0, handle_event/2]).
-export([run/2, clear/0]).

-define(NAME, cb).

%%
%% api routines
%%

run(A, B) ->
    server:call(?NAME, {run, A, B}).

clear() ->
    server:call(?NAME, clear).

%%
%% callback routines
%%

start() ->
    Fn = fun(A, Ld) -> handle_event(A, Ld) end,
    server:start(?NAME, Fn, []).

stop() -> server:stop(?NAME).

handle_event({run, A, B}, Ld) ->
    Ld1 = [ {A, B} | Ld ],
    {ok, Ld1};
handle_event(clear, Ld) ->
    error_logger:info_report([{clear_loop_data, Ld}]),
    Ld1 = [],
    {ok, Ld1}.

Now, anyone familiar with OTP will immediately notice a striking similarity with any module using the ‘gen_server’ behavior. It’s uncanny, actually. And the reason is the mechanics in both are the same. It’s just that our client/server is missing those fine details that make the real ‘gen_server’ run like a tank!

Follow the bouncing ball

Before going further, this is what this code does — don’t worry, it’s not doing anything fancy at all. Calling our ‘escript’ makes a couple requests to a running server (It will be running when we try it out, but that’s just a little ways off). The client module is called with two args from the calling script and passed on to the ‘run/2’ routine. Our little system isn’t that productive: all it will do is store these args as state in the server. Right after, our ‘escript’ calls ‘clear/0’ which just eliminates anything in the server’s state. Making it fresh.

Copy and pasting commands into the terminal is never fun, so here is a POSIX Makefile that has the parts we care about:

.POSIX:
.SUFFIXES:

.DEFAULT: all
all: FRC
    erlc *.erl
    @echo '***' run as follows:
    @echo escript ./cb.escript foo-bar-baz 99

server: FRC
    erl -boot start_sasl -detached \
      -setcookie mnbvcxz -sname cb_system@foo -eval "client:start()"

ping: FRC
    erl -boot start_sasl -setcookie mnbvcxz -sname cb_ping_test@foo \
      -noinput -noshell \
      -eval 'A = net_adm:ping(cb_system@foo), io:format(user, "*** ~p~n", [A])' \
      -s init stop

clean: FRC
    rm -f *.beam

# Force Re-compile
# Targets that have this pre-req. will
# still be built, even if that file
# exists.
FRC:

From least important, we have a ‘make ping’ target that will test the node connectivity (our script is a node, and so is the server). The ‘make clean’ isn’t interesting, but the ‘make server’ is. This target sets up our server once and for all, and should put our stateful server handling requests for the duration of our needs. (You can kill it by using ‘top’ and grep’ing for ‘beam.’)

Now, let’s run ‘make’ to compile our beamcode. You can copy the suggested ‘escript’ one-liner that’s returned, too. (ie. ‘escript ./cb.escript foo-bar-baz 99’.)

After a quick ‘make server,’ a ‘make ping’ should yield a ‘*** pong’ for a reply. Anything else, and you’ve got troubles.

Finally, in that same directory as our beamcode, sourcefiles, etc, we can run this in a shell:

escript ./cb.escript foo-bar-baz 99

This will run, and clear in one go. As our ‘escript’ has so declared.

Summary

With this as a guide, it should be easy to have a client module do any number of nasty work. I’m going to use this scaffolding to run some massively parallel file IO work so that I don’t have to attempt the same in C/Unix. No way.

And this will all be something easily callable from a Unix shell script such as:

#

foreach i ( some/path/data/* )
    ...
    escript ./cb.escript ${i} 1001
end
...

Now this little working example is quite the square-peg-round-hole cope, but sometimes, the only way to get what you want is to bend your tooling into shapes the language designers never would have predicted.