Sync’ up! … without getting drained

dec 3

Last resort testing

Testing code is discussed infinitely, and every year more paradigms are introduced. It’s a trendy topic indeed, but at Syncpup, we take a slightly different approach thanks to OTP & ‘let it crash.’

My Erlang routines clock in around four lines of code each. With such terse functions, it’s tough to find the willpower to write even unit tests for them. To be truthful, I just don’t do it much any more. Hell, if a piece of code looks something like the following:

handle_something_complicated({complex, X}) ->
    X2 = prepare_complex(X),
    {ok, X2}.

prepare_complex(X) ->
    lists:map(fun other_mod:some_call/1, X).

I’m wary of wasting any time after it’s been refactored, considering it reads like English. In fact, I’d assert (no pun intended) that tests fly in the face of Erlang’s ‘let it crash’ credo: can we honestly protect our code from niche bugs with tests at all?

Reactive driven development

As code chugs along in production, things come up, and code gets fixed when it does. Since we can patch code while keeping our systems going thanks to OTP, it’s often easy to ship code overzealously.

So how does one hedge the dangers of aggresive shipping, you ask? Well, code coverage analysis, of course!

Here’s the pattern I tend to rely on:

  1. A target release gets shipped and testing has been performed in large part while in development (using the Erlang shell, dialyzer, etc)
  2. Errors (if any) rear their ugly head as is the case with any system
  3. Code coverage is run over a specified time to examine the code that’s never called
  4. Unit tests are written for that code to ensure this rare code isn’t buggy

In large part, step three is the trickiest to get right for a production target release. So let’s go over what’s involved so there’s no confusion.

YATS

There’s little to gain unless we have a system we can play with. So let’s go ahead and create Yet Another Toy System for the sake of a tutorial.

Since I hate math, we will create an OTP application called ‘maths’ which is riddled with problems. In a directory of your choosing, using your Unix shell, or rebar, create/touch the following layout:

mkdir maths
cd !$
mkdir src
touch src/maths.app.src
touch src/maths_app.erl
touch src/maths_sup.erl

Respectively, each of the the files contain the following…

maths.app.src :

{application, maths, [
  {description, "Faulty maths"},
  {vsn, "1986.a"},
  {registered, [maths_sup]},
  {modules, [maths_app, maths_sup]},
  {applications, [
    kernel,
    stdlib,
    sasl]},
  {mod, {maths_app, []}},
  {env, []}
]}.

maths_app.erl :

-module(maths_app).
-behavior(application).

%% callbacks
-export([start/2, stop/1]).

%% api
-export([add/2, subtract/2]).

%%
%% api routines
%%

add(N, K) -> N + K.

subtract(N, K) -> handle_complex(subtract, {N, K}).

%%
%% callback routines
%%

start(_StartType, _StartArgs) ->
    maths_sup:start_link().

stop(_State) ->
    ok.

%%
%% business routines
%%

handle_complex(add, {K, N}) -> K + N;
handle_complex(subtract, {K, N}) -> K + N.

maths_sup.erl :

-module(maths_sup).
-behaviour(supervisor).

%% api
-export([start_link/0]).

%% callbacks
-export([init/1]).

-define(REGISTERED, ?MODULE).

%%
%% api routines
%%

start_link() ->
    supervisor:start_link({local, ?REGISTERED}, ?MODULE, []).

%%
%% callback routines
%%

init([]) -> {ok, {{one_for_one, 5, 10}, []}}.

And it should be noted that ‘maths_sup.erl’ is just in place to adhere to OTP principles; it doesn’t do much in this toy application.

Now that we have the Erlang code written, let’s quickly put the few things in place relx needs in order to generate a release.

mkdir private ebin
erlc +debug_info -o ebin src/*.erl
cat src/maths.app.src > ebin/maths.app
echo "[]." > private/sys.config
echo "-sname maths\n-setcookie abc123" > private/vm.args

And finally, our ‘relx.config’ looks as follows:

{include_erts, true}.
{extended_start_script, true}.
{default_release, maths, "1986.a"}.

{release, {maths, "1986.a"},
    [maths, sasl, compiler, tools]
}.

{vm_args, "./private/vm.args"}.
{sys_config, "./private/sys.config"}.

With that, we can generate a release to play with. Note, the ‘compiler’ & ‘tools’ applications are necessary for our ‘cover’ tool to be on hand.

Release for play

Assuming relx is in our path, let’s generate a target release:

relx release

We are left with a new ‘_rel’ directory, which contains our release. Let’s play with the ‘cover’ tool, and generate a report for the maths_app module’s activity.

./_rel/maths/bin/maths console

1> cover:start().
2> cover:compile_beam(maths_app).
3> maths_app:add(3, 4).
4> maths_app:subtract(9, 1).
5> cover:analyse_to_file(maths_app).
6> cover:stop(), init:stop().

N.B. The output file will be located in ‘_rel/maths,’ not the project’s root.

When we take a gander at the generated file, ‘maths_app.COVER.out,’ we’ll notice a call count next to each function head. This count starts after the ‘cover’ application has been set in motion, of course. It’s with this simple counter that we can unveil potential problems.

Discovery

Quite often, code sits around either unused, or waiting ready to error out because of unexpected parameters/calls/state. With our last-resort approach, our system hums away and the code that isn’t called, gets investigated reactively.

Any code that isn’t called — having a zero pointing to it in our report — could be deprecated, buggy or even completely erroneous. We planted one such bogus clause in our ‘maths_app’ module. This will jump out at us when we view ‘maths_app.COVER.out.’

Notice the line beginning with a zero:

...

   |  %%
   |  %% business routines
   |  %%
   |
0..|  handle_complex(add, {K, N}) -> K + N;
1..|  handle_complex(subtract, {K, N}) -> K + N.

Indeed, our shoddy code (with all the runtime arithmetic funkiness aside) never calls the first ‘add’ handle_complex/2 routine. We could write a test for it at this point, or simply have our API call it as is the case with the ‘subtract’ routine. Either way, we have covered our butts a bit.

Nothing but the code

Obviously, the many ways to test are always available to us & not all situations work with this flavor of testing. But if you’re using OTP, reactive testing fits the bill.

After all, what’s the point of writing tests only to discover the ones you weren’t counting on are wreaking havoc on your system?

Thanks to Erlang’s solid ‘cover’ tool, we can test when we truly need it, while keeping our codebase sanitary all the while. This saves oodles of time, and keeps our releases always marching in the direction of correctness.