Sync’ up! … without getting drained

sep 16

Building an ikura app

N.B. This post is deprecated. For archival purposes, it remains here, but generally, it ought to be disregarded by readers.

ikura is a tool for creating light-weight CRON tasks for your scripts & web applications. One of our secret sauces is a task can finish and depending on the outcome, can spawn a new generation of tasks. It’s this feature that makes ikura powerful, and we thought we would showcase how our ikura customers can make this happen.

We thought it would be fun to make a little web utility that tweets Twitter users who have posted something along the lines of ‘hey all, just broke my arm. Super bummed out.’ The holiday season is fast approaching, and the North-East seems to have its way with skiers, commuters, and fun-loving bar-hoppers. Wouldn’t it be nice if a ’bot sent you a little love soon after you were down-n-out from an injury?

We are going to call our application ‘bonegram’ & it will utilize the standard ikura services. We will write the core bonegram application in Erlang/OTP as it’s a great language for handling network-y things like calling Twitter’s API & interfacing with ikura.

The core bonegram application needs to do several things:

  1. poll Twitter’s API and pull-in tweets that contain keywords we care about
  2. perform some lightweight data-mining of the above tweets
  3. tweet messages from bonegram’s Twitter account to those we care about
  4. interface with ikura, which does all the automatic co-ordination

We will be using the latest version of rebar, a tool which can be found here. In addition, we will build the Erlang/OTP application with version 17.1. To keep this blog-post terse, we are not going into the details of installing OTP, nor how to obtain the few shell tools that we assume most developers have handy already.

Let’s get started.

Setting things up

To make an OTP application, rebar offers a quick way to bootstrap things. But first, we need to create a working directory, and get a version of rebar.

mkdir bonegram
cd !$
wget https://github.com/rebar/rebar/wiki/rebar
chmod +x ./rebar
./rebar create-app appid=bonegram

This will give us our ‘src’ directory; you’ll notice rebar has given us a few files.

Getting dependencies

Ikura talks to its customers via TCP/IP, which means our bonegram app will have to offer a couple of HTTP endpoints. This sounds like a job for Cowboy! Cowboy is a set of libraries which we can leverage for this situation. Using rebar & a new file called ‘rebar.config’ — which will reside in the root of our project directory — we can set this all up as follows:

{require_otp_vsn, "17"}.

{deps, [
  {cowboy, ".*", 
    {git, "git://github.com/nato/cowboy.git", {branch, "stable"}}}
]}.

Our ‘rebar.config’ now has the configuration we need to fetch this dependency. We do this by performing the following:

./rebar get-deps

You will notice the new ‘deps’ directory after this is run.

Wiring Cowboy

If we cat our ‘src/bonegram_app.erl’ file, we should notice that it is waiting for us to include some meaningful Cowboy code. This new code will simply get Cowboy up and running when our bonegram app starts, and will route the endpoints to the few places we need it. After our edits, the ‘src/bonegram_app.erl’ now looks as follows:

-module(bonegram_app).
-behaviour(application).

%% behavior callbacks
-export([start/2, stop/1]).

-define(CLIENT_ACCEPTORS, 10).

%%
%% behavior callbacks
%%

start(_StartType, _StartArgs) ->
    Port      = port(),
    Routes    = routes(),
    Dispatch  = cowboy_router:compile(Routes),
    TransOpts = [{port, Port}],
    ProtoOpts = [{env, [{dispatch, Dispatch}]}],
    {ok, _}   = cowboy:start_http(
      main_http_listener, ?CLIENT_ACCEPTORS, 
        TransOpts, ProtoOpts),
    bonegram_sup:start_link().

stop(_State) ->
    ok.

%%
%% support routines
%%

port() ->
    {ok, Port} = application:get_env(http_port),
    Port.

routes() ->
    [{'_', [test_route()]}].

test_route() -> 
    {<<"/test">>, bonegram_handler, [{type, test}]}.

And of course, the module that will handle the http requests needs to be created, too. Above, we called it bongram_handler, so in ‘src,’ we write a quick-n-dirty placeholder module called ‘bonegram_handler.erl’ which looks as follows:

-module(bonegram_handler).

%% cowboy callbacks
-export([init/2]).
-export([content_types_provided/2]).

%% user-def callbacks
-export([handle_json/2]).

-record(state, {type}).

%% 
%% cowboy callbacks
%% 

init(Req, [{type, T}] = _Opts) ->
    State = #state{type=T},
    {cowboy_rest, Req, State}.

content_types_provided(Req, State) ->
    provided(Req, State).

%%
%% user-def callbacks
%%

handle_json(Req, State) ->
    Json = "{\"status\": \"todo\"}\n",
    {Json, Req, State}.

%%
%% support routines
%%

provided(Req, State) ->
    {[json_type()], Req, State}.

json_type() ->
    {{<<"application">>, <<"json">>, []}, handle_json}.

Two things: we are going to need a small configuration file for our Twitter API auth-key & auth-secret. We are not quite ready for that yet, but since we need something else in there for our app to work, go ahead and create in the root directory of the project, the file ‘dev.config’ with the following:

[
  {bonegram, [
    {http_port, 8004}
  ]}
].

Lastly, our application should crash if Cowboy is not started up front. We add it to the ‘src/bonegram.app.src’ file as follows:

{application, bonegram,
  [
    {description, ""},
    {vsn, "1"},
    {registered, []},
    {applications, [
      kernel,
      stdlib,
      cowboy
    ]},
    {mod, { bonegram_app, []}},
    {env, []}
 ]}.

(Continued in part II.)