Hotfix howto

Let’s say your OTP release is humming along with confidence, and ‘nine nines’ up-time is looking like a piece of cake this year.

Then, alas, the phones start to ring.

The world’s smallest typo in some JSON message from the Erlang back-end is creating a headache for the entire QA team.

Here’s what happened…

In the latest release, instead of a UI message reading ‘Your Username is Taken’ as it should, it reads ‘Your Username is Tekken.’

This two-character mishap is making new sign-ups worried that their name gets saved as ‘Tekken’ — which isn’t true, but that’s how they are all interpreting it.

You look at your watch and put your head on the desk as you think of what’s involved with pushing a whole new OTP release.

What to do?

Small but urgent

Sometimes it’s just not worth fast-tracking an entire new release for such small changes.

When the bugs are pressing but minuscule — but they hardly warrant a whole new release — it’s good to know how to push small changes live to an OTP system.

Target system hotfixes

Pretend for a moment that our typo lives in a file named ‘foo.erl.’

“There it is. Line 833, Tekken.”

We make the spelling correction, and get the go-ahead to put a patch in place, as the next stable release is still a week out, and QA is looking tarred ’n’ feathered.

What now? Well, follow along and we will put this fix in place. (And without taking our OTP system out of service, of course.)

First, we shell into our production box(es), and create a new folder in our home directory called ‘hotfix.’ We can do that like so:

cd ~
mkdir ./hotfix

There. Next, we copy the corrected ‘foo.erl’ file into this new ‘hotfix’ directory (from source control or otherwise) :

cp /place/that/has/it/foo.erl ~/hotfix/foo.erl

We can now dive into our production Erlang shell. (Depending on your system, you can do that via ‘run_erl’ and pipes, or simply via Erlang’s ‘remote-shell’ mechanism.)

Let’s take a moment and examine what it is we want to accomplish:

1. Compile the new ‘foo.erl’ — the corrected Erlang source file
2. Tell our running system about it & load it
3. Purge the old typo version of ‘foo’ from existence

Before we do all that, in the Erlang shell, let’s glance at some information about the typo version of ‘foo’ to start off:

1> foo:module_info().

We do this so that after we successfully load the corrected version of ‘foo,’ it’s obvious that our efforts took effect.

Next, we need to compile our new source file. There’s a zillion ways to compile Erlang source code, but just go with the following for now:

2> compile:file("/home/jimbo/hotfix/foo.erl", [
  {outdir, "/home/jimbo/hotfix"}, debug_info, {i, "include"}]).

There. Our ‘hotfix’ directory should now have both the source and the compiled BEAM equivalent.

So far, our system is still unaware of the new version of ‘foo.’ To change that, first we must tell our system about this special code path, and then finally load the new ‘foo’ into our system. We do that as follows:

3> code:add_patha("/home/jimbo/hotfix").
4> code:load_file(foo).

You will notice that we went with the ‘add_patha’ routine rather than ‘add_pathz.’ In essence, we want our ‘hotfix’ directory to take precedence wherever ‘foo’ is found.

Presto change-o

Believe it or not, our production system has now been patched with the hotfix! The typo is history — which means the phones should have stopped ringing. And it only took a couple lines in Unix & a mere few lines in the Erlang-shell. How great is that?

Before we ping the boss about the good news, it’s good housekeeping to double-check our efforts, and, in one last step, to purge the old version of ‘foo’ from the runtime system. You can do that as follows:

5> foo:module_info(). % compare to previous
6> code:soft_purge(foo).

And with that, you can safely detach from the Erlang shell and get back to hacking the fun tricky stuff.

You can see that in a pinch, even complicated things like changing the code of a live system is a snap in OTP land…Isn’t it amazing how much power you can wield with just the knowhow and a few keystrokes when you have Erlang by your side?