RPC between Go and Ruby using Protocol Buffers

I’ve previously written some thoughts on service oriented architectures and since then I’ve wanted to explore beyond the currently accepted standard of sending and receiving JSON data over web APIs.

Protocol Buffers

With JSON there’s an overhead of transmitting plain-text over the wire coupled with parsing that text in your application once received. For many cases this is probably acceptable but for a large-scale distributed system this can represent a not insignificant cost in terms of bandwidth and computing resources.

In response to this situation Google came up with Protocol Buffers which basically give you a mechanism to define ahead of time the data structures your services will be exchanging and to convert these structures into a more efficiently transmitted and deserialised binary format.

The supplied compiler generates the code to handle this for you and supports the majority of popular languages which is essential for an ecosystem comprised of heterogeneous technologies.

gRPC

The concept of being able to invoke a piece of code on another machine is far from a new one. Many languages provide a way of doing so yet the fashion in the web world seems to be to create REST-like services on top of HTTP, even if the end result could almost be described as a JSON RPC service.

With gRPC you can describe your services as you can the data they interchange and the generated code handles all the interaction for you, allowing you to concentrate on the actual function of the service and not the logistics of how it is delivered.

As an added bonus it uses HTTP/2 under the hood so you automatically get the benefit of things like header compression and request multiplexing.

Same but different

To demonstrate gRPC in action I’ve simply forked and updated the code from the previous post. There’s still a simple Go service being accessed by a Ruby client though now my application code doesn’t have to be concerned about HTTP and JSON. Accepting the format of the generated code seems to be the price to pay for this but I think it’s been a worthwhile exercise.

Check it out on Github.

A simple service oriented architecture using Ruby and Go

Some common pain points seem to keep cropping up time and time again as applications grow in size and complexity:

  • tests get longer to run
  • deployments become a much more involved process
  • making a small fix to one area of code can often break something in another
  • it can be much harder to reason about the system as a whole

These large, monolithic applications are affectionately known as monorails in the Rails world and the common wisdom is to start splitting up the codebase into services: to extract areas of functionality out into self-contained, independent applications which work in unison to deliver the original capability.

Here’s how Songkick did it and how SoundCloud did similar recently.

Close, but no cigar

On a previous contract I worked on a platform which claimed 20 million users and the only way it was possible to handle that sort of capacity was to use services, lots of services.

The platform was centred around a legacy ColdFusion application hooked up to a MySQL database and the other applications had a variety of ways of accessing the central data:

  • using ActiveResource or libraries derived from it to consume a RESTful API provided by the ColdFusion app
  • connecting directly to the database and defining their own models and schemas
  • using one of the two semi-aborted attempts at packaging up a collection of ActiveRecord models into a shared library
  • mixing and matching a selection of the above in different areas of their codebases

Unfortunately there was little consistency and the organisational culture encouraged quick fixes and neglect of old code. Both new development and maintenance work provided numerous challenges as you can well imagine.

The ideal

As I progressed through the term of the contract and had to deal with the peculiarities of the technical ecosystem at hand I discussed with the other engineers what the ideal situation would look like, everything else being equal. The only sane solution seemed to be to develop both an API and the client used to access it.

Since moving on from that contract I’ve wanted to implement a proof of concept of that idea and I’m glad I’ve now codified the basics of it.

A worked example

The API is written in Go and allows manipulation of resources in a standard RESTful manner. No surprises there other than everything is stored in-memory and will so be lost when execution stops.

The client library is in Ruby and basically just wraps a HTTP client with an ActiveRecord-like interface. The final piece is a demo script which uses the library to add and then manipulate a handful of items of data. All very straightforward but I believe it illustrates the concept.

This shameless display of hipster polyglotism can all be found on GitHub. Fill your boots!

Contracting in London

Since I last wrote about my experience working outside of Northern Ireland I’ve returned to the UK, turned down prospects of contract work in Belfast and have set up shop in London.

Serendipity

A good while back Rob and myself had discussed the possibility of London. He’d been working remotely with a company based here and due to changes in his circumstances leaving N. Ireland was a valid option.

Lines of communication between the two of us dropped off for a while as a result of travel, work and such like and when I finally got caught up with him I discovered he’d already bitten the bullet and was moving over. Further conversation uncovered that the house he was going to be sharing in Ealing had a box room going spare which I was welcome to make use of for a month. Wheels were set in motion.

The Search For Work

With my mind made up about London I started keeping my eye out for contract work. Apart from the usual job sites I focussed my efforts on the LRUG mailing list as many companies will directly post job ads there and going direct with the client is very often the preference with contracting.

A week or so before I was due to fly back to Ireland positions with two companies were posted to the list, I responded and eventually interviews via Skype were arranged. These both went well and next steps were organised.

>30 hours travelling had me back in my home county of Fermanagh. I rested, caught up with my family and returned to Belfast long enough to pack a carry-on case and head to London for the joys of face-to-face interviews.

Song and Dance

The first interview was on a Thursday, the next on the Friday and the first offer was in my inbox on the Monday. Not bad at all and very indicative of the state of supply and demand for technical talent here in London specifically and across the industry in general.

Unfortunately the first offer turned out to be unable to match the rate I was on during my last contract. Adding to the displeasure was me having effectively done a morning of unbilled work as part of the process along with having to suffer the song and dance of a competency based interview. Such is life, the world doesn’t owe anyone a living.

The other offer when it arrived didn’t involve a drop in rate and looked to have the opportunity to get my hands dirty with a bunch of interesting technology. From a business point of view this seemed to be the one to take so I went for it.

The Road Ahead

As has happened before in my career, the job I was sold and the job I ended up getting didn’t quite match up. I did my three months, put myself forward for a handful of contracts, did a couple of technical tests and one face-to-face interview and I’m now with another client and things are better.

Before coming to London I only knew a little about the place. One area was more or less interchangable with any other area. I soon discovered that the majority of the contract Ruby work going is focussed in Central and East London and being out West in Ealing for a lot longer than originally planned has meant a great deal of commuting and a general feeling of being disconnected from the hustle and bustle of the technology world here.

The solution to this has been to move closer to the trendy part of town and within a few weeks we’ll have transplanted everyone across to the other side of town and into a converted warehouse near Whitechapel.

I still have my flat in Belfast, have started investigating parts of the world with even lower costs of living and have only the vaguest idea of what the future might hold but can’t see myself working anywhere other than in London over the next few years.

No choice but to continue to making it up as I go along then :D

Two years of working away from home

I’m currently sitting in Melbourne enjoying the Australian summer and a well deserved break from work. I can barely remember the northern hemisphere winter I left behind a few weeks ago.

I’ve only briefly mentioned working away from home before but January past marked two years since I last worked in Belfast. The pace has been hectic at times and I’ve spent more nights in hotels than I care to recall but from my first contract in Dublin to my last in the south of England I’ve found a consistant theme: being treated with more respect, working on more interesting problems and for higher pay. Not a bad combination!

RubyConf Australia

Coinciding with my trip to Melbourne was the first ever RubyConf Australia where I was able to shake hands and speak with some of the known names in the Ruby world. I’ve also been able to attend a Ruby Australia meetup and a Travis CI coffee morning so there’s been plenty of opportunities to geek it up along with soaking up the sun.

Adventure

In recent times I’ve done more and more travelling with the last 12 months seeing me in Dublin, London, Amsterdam, Barcelona and now Melbourne. My taste for adventure has grown, I’ve a hunger to be where the action is and a part of the world where people are arguing over a flag is just not where it’s happening.

There’s no contract market for Ruby in Belfast though so in order to continue contracting, working with Ruby and living in Belfast the first flight out of town on a Monday morning seems unavoidable.

Where to next?

There’s no two ways about it, living out of a suitcase sucks. The alternatives seem to be to leave Belfast, start up my own product company or take on a remote working permanent position. Needless to say there’s drawbacks to all of these and most of the time it’s seems best to just suck up the drawbacks of working away from home.

Considering the strength of the contract market in London I’ve often thought I’d end up there for a stint and after a recent conversation with Tim I’m also quite interested about contracting in Berlin.

The day I booked my flight to Oz was the final day of my last contract and I’d chosen not to put any effort into lining up the next piece of work. Within a fortnight I’ll be back in the UK and dealing with jetlag. Will something turn up by then? If if does where will it be, what will it be doing and how much will it pay? The usual uncertainties associated with contracting.

Whatever happens, the next few weeks should be interesting!

Using Vagrant and Chef to setup a Ruby 1.9 development environment including RVM and Bundler

It has become common practice these days to use tools like RVM and Bundler to manage a project’s dependencies. When these pacakages are in place, getting up to speed with another project is a breeze.

The Pain

But how about installing these tools themselves? How about other dependant pieces of software such as databases and the like? What if they have to be compiled from source? What if they have to be installed with a package manager? Which package manager do you then use: Fink, MacPorts, Homebrew?

Not always so easy!

It’s a UNIX system! I know this!

I love UNIX. I’ve worked in the investment banks, telecoms companies and startups of the world building and supporting software on FreeBSD, Solaris, RHEL, Debian and more. For just over a half decade now I’ve been using a Mac to do this. I was reluctant to begin with and I could care less for the fanbois but I now strongly believe that a Mac running OS X is the best UNIX workstation I can get my hands on.

Despite all I like about it, OS X can be pretty hostile towards developers, just ask anyone who has setup a Ruby dev environment on a fresh install of Lion recently or who uses Xcode regularly.

Enter Vagrant and Chef

Vagrant can be used to provide an isolated environment running inside a virtual machine. When combined with a provisioning tool like Chef it’s possible to concisely specify the necessary environment for an application and then have it available in a dependable manner on a on-demand basis.

This combination is to systems what RVM and Bundler are to Ruby interpreters and libraries.

I’d been hearing good things about Vagrant and Chef for some time but what prompted me to delve deeper was the upcoming addition of a junior dev and an intern at Converser. Having all dependencies scripted, including operating systems, programming languages, document and key/value datastores seemed like a good way to get the new starts up and running with the minimum of time and headaches.

An Example

I have an example of what I’ve learned on GitHub. It’s a simple Sinatra app but demonstrates all the moving parts.

Standalone installers for Vagrant, Chef and Git are to be found here, here and here which removes the need for any form of package manager on the host OS X system.

Once everything’s installed and the repo cloned, the following commands will start up the example app within a fresh virtual machine:

vagrant up
vagrant ssh
cd /vagrant
bundle exec rackup

Browse to http://0.0.0.0:8080 to view the output.

For the curious, the versions or Ruby and Bundler can be checked thus:

$ vagrant up
$ vagrant ssh
$ cd /vagrant
$ ruby -v
ruby 1.9.2p320 (2012-04-20 revision 35421) [x86_64-linux]
$ bundle -v
Bundler version 1.1.4

The virtual machine can be powered down and disposed of with this:

vagrant destroy

Should Things Be This Convoluted?

Perhaps the world would be a better place if all this was simpler. Perhaps returning to a Linux workstation would remove some of these headaches. Maybe I’ve just grown accustomed to the pain.

For the time being OS X appears to have the correct balance of desktop software and UNIX-y stuff for my needs. Until something appears that surpasses the form-factor, power and utility of my current setup I’ll continue to pay the Apple-tax.

An introduction to Sinatra

Tonight I gave an introductory talk about Sinatra at the second meetup of BelfastRuby.

The Converser platform I’ve been building this last while uses a lot of Sinatra so when I was asked to give a talk about using Ruby to develop web apps without Rails it wasn’t hard to think of a subject.

This was my first technical talk and apart from a touch of nerves and forgetting some of the jokes I had in mind I think things went well. Maybe I’ll end up giving another one before too long :)

The slides are available at belfast-ruby-sinatra.herokuapp.com and make use of showoff which itself uses Sinatra. Meta.

The source, including the code samples, is on GitHub.

Update: some photos from the evening are now available on Flickr.

Tracking my coffee consumption with Redis, Sinatra and Cocoa

I’ve often joked about putting together an app to track my coffee consumption, such is my reputation for consuming the black goodness. Like a lot of my other personal projects, the idea had a prolonged gestation period and was finally born through a welcome spark of motivation.

Crafting fine web APIs

Over the past 6 months the bulk of what I’ve been doing day to day with Vigill has involved building web APIs for consumption by mobile clients. This has involved lots of Sinatra, Redis and MongoDB.

In this time I’ve also put together a couple more Cocoa apps.

Thick client boogie

An unexpected resurgance in enthusiam for developing desktop apps combined with a fluency in cranking out webservices put me in a good position to put together a simple API and client.

I’m not a big user of mobile apps but I do spend the bulk of my waking hours sitting in front of a Mac so producing a client for OS X was the logical choice.

For the API I considered using something a bit more esoteric than my standard toolkit, but no matter what combination of technologies I investigated not much seems to come close to the power and flexibilty I find with Ruby and it’s frameworks for performing the bulk of tasks required by the web applications of today.

Areas of note

A valid API key must be sent in the request headers when sending a POST to the Sinatra app. Validity of the key is determined by checking set membership in Redis. A 4xx status code is returned if the key is missing or invalid.

The daily count is boosted with an atomic increment of a hash field.

The client is not much more than a GUI wrapper around some HTTP requests sent using AFNetworking. A previous native iOS client I developed used NSURLRequest directly and I found AFNetworking much simpler to use.

Get the code

The source for the API and the OS X client is available on GitHub as usual and some further technical details are available in the READMEs.

You can keep up to date with my coffee consumption at coffee-tracker.herokuapp.com.

Enjoy.

A minimal Rack app to lookup your external IP address

I think I first heard the term “Rack-app” when I started using Heroku to host a couple of projects. I didn’t know an awful lot about Rack but I did know that it helped a great deal when it came to getting my Rails and Sinatra apps off my workstation and onto the web.

Tonight I thought I’d have another quick look at how it works, what it takes to build a Rack-app and what information about the request is available to the handler.

The result was the following, an object which responds to call and returns a HTTP response code, headers and a body:

ip = lambda do |env| 
  [200, {"Content-Type" => "text/plain"}, [env["REMOTE_ADDR"]]]
end
 
run ip

I have this running on ip.stevenwilkin.com and it provides the same useful function as whatismyip.com but without the ads :)

The code is available on GitHub as always.

Backing up Delicious bookmarks to CouchDB

Last year I started researching how CouchDB could be used for the BodyGuard web-app QA platform. Unfortunately I had to put my tinkerings to one side when some extra freelance work came in; contracting during the day and freelancing in the evenings quickly put an end to any spare time and my fitness efforts took a hit but I was only self-employed for a short while and it seemed the thing to do.

Since the urgency of the freelance work died down I’ve been looking for a small project to help me get back up to speed with Couch again and when the rumours of Delicious being shutdown surfaced I saw the opportunity.

The code is quite simple but it served to refresh my memory and has an actual practical aspect which is a bonus :) It’s available on GitHub as usual and the instructions below should get you up and running.

Prerequisites

  • a pre-Yahoo Delicious account (they use HTTP Basic Auth for authentication)
  • CouchDB running on port 5984 on your local machine, I recommend the binary distributions from CouchOne
  • nokogiri and couchrest gems must be installed:
    sudo gem install nokogiri couchrest

Clone the code from GitHub

git clone https://github.com/stevenwilkin/delicious-couch
cd delicious-couch

Run the import script to fetch your saved bookmarks

USER=username PASS=password ./import.rb

Substitute your Delicious username and password on the command line

Create the design document

./views.rb

Querying the data

Now that the database had been populated and the views created, the contents can be queried

View all urls

./list_urls.rb

View all tags along with their count

./count_tags.rb

View all urls with a specific tag

./urls_by_tag.rb git

Views all bookmarks tagged with “git”

This set of scripts is only scraping the surface of what is possible with CouchDB but it demonstrates the immense power of MapReduce along with the basics of interacting with CouchDB using Ruby. Clone the code and enjoy!

Installing RMagick on Debian Lenny

Just a quick reminder to myself on how I installed RMagick on a Debian 5.0.4 “lenny” VPS. All commands to be run as root.

Build ImageMagick from source

curl -O ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz
tar xvzf ImageMagick.tar.gz -C /usr/src/
cd /usr/src/ImageMagick-6.6.6-6/
./configure
make
make install

Install the RMagick Gem

gem install rmagick

Bingo!

As an interesting aside, ImageMagick handles a lot of image formats via delegate libraries. I previously installed RMagick on a CentOS box and had to separately install TrueType fonts which were necessary for the project in question. These had to be installed before ImageMagick and were accessible through yum and the xorg-x11-fonts-truetype package.