Prod-to-dev with Mnesia

Back in my dark days of Rails development, it was essential to have a way to populate one’s local database, say SQLite, with production data. And for some reason, half a decade has gone by as an OTP engineer before I needed to perform this task in Erlang-land.

After looking into the matter, it was easy enough, but the documentation was scattered. So, I thought I would share my process for getting a production node’s db-data onto my local development machine.

Ramping up

First off, I want to define a few things for the sake of this guide. I will refer to our single production box via the name ‘jimbo@prod’ while our development name will be ‘jimbo@dev’ — these, of course, are the VM’s sname values, or short names, and in your case, you’ll probably be using fully qualified long names.

Also, we will assume our Unix user-name on production and development is ‘jimbo’ for both.

Lastly, we will assume you have a preferred strategy for attaching to production nodes — either via Erlang’s remote shell strategy, or via erl_pipes.

Dumping production

The first order of business, is to make a copy of the production database. We will want to output it somewhere, so follow along and we will take care of that. On our production box, let’s create some directories to aid us along:

cd ~
mkdir tmp

Next, attach to your production system Erlang shell and perform the following:

1> mnesia:backup("/home/jimbo/tmp/mnesia.prod.backup").

Done. You can now detach from the production Erlang shell.

Now, change into that dump directory and tar up the database contents for quick transfer to your development machine. Perform the following to do just that:

cd ~/tmp
tar cvfJ mnesia.prod.txz mnesia.prod.backup

And we can close our Unix session on production. We are all finished with it.

Dev prep

Our OTP release could be anything at all. From a simple server, to something quite complicated. But for the sake of this guide, we need to make an assumption that our system has an API resource whereby we can prime our Mnesia database for fresh installs of a target system.

There is no standard way of doing this, but to create the requisite database schema, I usually have some code that does this all in a deterministic way, such as foo_lib:init(db). which can be called once, manually.

Here is some example code on what this could look like. Here is a snippet from a fictional module called ‘foo_lib.erl’ :

...

init(db) ->
    Nodes = [node()],
    mnesia:stop(),
    mnesia:create_schema(Nodes),
    mnesia:start(),
    try
        mnesia:table_info(type, user)
    catch
        exit: _ ->
            mnesia:create_table(user, [
              {disc_copies, Nodes},
              {attributes, record_info(fields, user)}
            ])
    end,
    try
        mnesia:table_info(type, pets)
    catch
        exit: _ ->
            mnesia:create_table(pets, [
              {disc_copies, Nodes},
              {attributes, record_info(fields, pets)}
            ])
    end,
    Tables = [user, pets],
    Reply  = mnesia:wait_for_tables(Tables, 15000),
    {status, Reply}.

...

This code is just to give you a clue about how to set up your database schema. Be opinionated about how you like to do it, for sure. We will assume for the rest of this guide, a fresh Mnesia database can be primed after we start the OTP system, then call foo_lib:init(db).

With that lengthy aside out of the way, let’s get ready to transfer our production data to our local box. First, let’s create the familiar working space with mkdir ~/tmp and copy over the compressed database to there now:

scp jimbo@prod-box.net:~/tmp/mnesia.prod.txz /home/jimbo/tmp/

Let’s assume all went well with scp, ssh access and such, and we will proceed with un-tarring the dumped database:

cd ~/tmp
tar xvf mnesia.prod.txz

There. A good deal of the nitty-gritty is out of the way.

Introducing ‘convert.erl’

Now’s the time to write some Erlang. We are going to create a new module that reads the production Mnesia database and converts the naming of the production node, to that of our development one, wherever it occurs.

Recall, that our production sname is ‘jimbo@prod,’ but locally, it’s ‘jimbo@dev.’ That’s why we need to do this.

Here is the contents of the new ‘convert.erl’ source-file; this is on our development box & placed in ‘/home/jimbo/tmp’ :

-module(convert).

-export([new/0]).

%%
%% api routines
%%

new() ->
    From = 'jimbo@prod',
    To   = 'jimbo@dev',
    Path = "/home/jimbo/tmp",
    handle_node_name(
      mnesia_backup, From, To, 
      Path ++ "/mnesia.prod.backup",
      Path ++ "/mnesia.dev.restorable").

%%
%% business routines
%%

handle_node_name(Mod, From, To, Source, Target) ->
    Convert = convert_fun(From, To),
    mnesia:traverse_backup(
      Source, Mod, Target, Mod, Convert, switched).

%%
%% support routines
%%

convert_fun(From, To) ->
   fun({schema, db_nodes, Nodes}, Acc) ->
       L = [ maybe_convert(
         X, {From, To}) || X <- Nodes ],
       {[{schema, db_nodes, L}], Acc};
      ({schema, Tab, CreateList}, Acc) ->
       L = [ maybe_key(
         KeyVal, {From, To}) || KeyVal <- CreateList ],
       {[{schema, Tab, L}], Acc};
      (Other, Acc) -> {[Other], Acc}
   end.

maybe_convert(N, {N, To})    -> To;
maybe_convert(N, {_From, N}) -> throw({error, already_exists});
maybe_convert(N, {_, _})     -> N.

maybe_key({Key, Vals}, B) ->
    Keys   = [ram_copies, disc_copies, disc_only_copies],
    Member = lists:member(Key, Keys),
    maybe_key({Key, Vals}, B, Member).

maybe_key({Key, Vals}, B, true) ->
    L = [ maybe_convert(X, B) || X <- Vals ],
    {Key, L};
maybe_key(A, _, false) -> A.

Now, all the while working in ‘/home/jimbo/tmp’ we compile the new module using the humble erlc command. We do that as follows:

erlc convert.erl

Next, let’s put convert to work and get our database dump in a state where our local machine can work with it. In that same directory, do as follows:

erl

1> l(convert).
2> convert:new().
3> init:stop().

Done. We now have a copy of our database ready to be loaded into the development version of our OTP project. If you ls -lhart the directory, you can see the new file convert created.

Pulling it in

Go ahead and change directories to where your OTP project lives.

It’s time to fire up our project locally, but first we make sure any old Mnesia development data is blown away so not to interfere. Assuming that the location of these old Mnesia database files are in the project’s root directory, you can tar those up and place them aside, or simply rm -fr ./Mnesia.* if you won’t be missing any development data.

Now go ahead and start up your OTP release locally, and attach to it in your favorite manner.

Recall that we now have a missing database schema, and need to create one afresh. We will want to do that right away, followed by a Mnesia routine that populates our local database with our production data. We do both as follows:

1> foo_lib:init(db). % for example
2> A = "/home/jimbo/tmp/mnesia.dev.restorable".
3> mnesia:restore(A, []).

And there you have it: your local Mnesia database now holds all the goodies from production. If this is something you need to do often, then the lion’s share of the above ought to be automated. But there’s no shame in bookmarking this post & coming back here when you need to do it all again.