Blogging on Ansible

A good friend of mine recently voiced the opinion that the day will soon be upon us when the traditional SysAdmin will no longer be relevant and all operations activities will be some variety of DevOps.

Now, I’ve been managing my own infrastructure for some years but every package has been manually installed with every configuration file modified by hand, resulting in snowflake servers which are nigh-on impossible to recreate exactly and have occasionally caused me a lack of sleep. I’ve known for a long time that I could be doing better.

Configuration Mismanagement

In recent years all the clients I’ve contracted with have used either Chef or Puppet to automate the management of their infrastructure.

The funny thing has been that the last couple have both used Puppet in conjunction with some bespoke, cobbled together system to upload the modules, manifests etc to the target machines and run puppet from there, bypassing the usual master/agent setup. Seeing this once was unusual but seeing it twice in a row struck me as very odd indeed and made me think that there was perhaps something overcomplicated with the “usual” approach.

Ansible Simplicity

With Ansible there is no client/server model: all you need is a repository of playbooks, SSH access to an inventory of machines and the rest is as simple as running ansible-playbook. Very straightforward.

Something I’d wanted to do for some time is migrate my own modest infrastructure away from a single overloaded, handcrafted VPS and to get everything under configuration management in the process.

I’d seen a few examples of Ansible and it was one of the buzzwords being bandied about at the time so it seemed like good opportunity to get my hands dirty and approach my own environment as I would a client’s albeit with cheap OpenVZ containers instead of EC2 instances.

Migrate All The Things

One of the things to be migrated was this blog. It had been running on a woefully out-of-date version of WordPress and using the same theme for the past half decade or more so getting both of those updated along with the infrastructure aspect was good.

Whilst I was shaking things up I’ve also taken the opportunity to move the blog from it’s own domain, sickbiscuit.com, to a subdomain of my personal site, blog.stevenwilkin.com, in an effort to simplify my web presence.

In terms of servers this is now running on one I’m now using to host PHP apps, there’s another that handles redirects from the old domain along with other static sites and there’s a third that stores the nightly backups produced. All three have been provisioned with Ansible.

I found using YAML for the playbooks to be a bit unusual to begin with but on the whole Ansible has been easy to get up and running with and so far seems simpler than the alternatives.

Next Steps

Before I can decommission the old VPS completely I still have to migrate my mail server along with a few other apps which will require a bit of effort. When that’s out of the way I may for the sake of completeness setup monitoring with Nagios and I’ve a strange hankering to start running my own DNS servers. I know.

Time will tell what exactly I’ll do but with Ansible provisioning new machines will be a breeze.

Serving up a static site using Go

Like a lot of people in technology a portion of my time is spent wondering what will be the next programming language, framework or whatever to take off and become popular. Of greater personal importance is the matter of which one of them will get added to my toolboox next and press-ganged into productive use.

For years now Ruby has been my goto for a lot things and I’ve been doing more and more OS X and iOS development but it had been some time since I last picked up something new and I had a hankering for some shinyness.

I started tinkering with Node a few years back but it didn’t have enough appeal to invest much of myself in it and Scala looked a likely contender for a while but I didn’t get any further than solving a few Project Euler problems with it, mostly while spending my evenings in hotel rooms in Dublin.

Enter Go

Earlier this year I spent a month in Melbourne and thanks to having some time away from paid work I could see that my preference seemed to be leaning towards developing web services and then consuming those services with native client apps. I wanted to keep my skills sharp so started tinkering with a personal project and for the API piece the relatively new Go seemed ideal for the job at hand.

Once I got started what struck me was that I was producing a single, standalone binary which didn’t need any external dependencies in order to be run on the target architecture. No need for a particular version of a language or a slew of packages to be installed, just a command to be executed. It’s almost like I’d forgotten that that was how things used to be once upon a time after these years of using scripting languages and it was a liberating experience.

For that project I wanted to do everything test-first but unfortunately the depth of my knowledge of Go in general and web application development with it more specifically are not what I want them to be so when I hit a stumbling the project was abandoned. Perhaps I’ll come back to it in the future.

A Need Emerges

So I still wanted to get some Go out into the wild and when I decided to put together a new website for my limited company I thought I could use the situation as a learning experience.

Serving up a static site can be done any number of ways and I certainly didn’t need to go re-inventing the wheel but with a bit of tinkering I had put together a simple app that could look at a requested path and if it corresponded to a file under public/ then send the file’s contents to the client.

What pleased me with the solution was that it was achieved with only what’s provided out-of-box with Go and without too much code. Simple and to the point and getting it running on Heroku was trivial with the Go buildpack.

The End Result

The site, as simple as it may be, can be found at nulltheory.com and the code is on GitHub. Bon appetit!

Keeping an eye on UK inflation with Google App Engine

For a long time now whenever I’m on the search for something interesting to work on that could potentially be of value to myself and others my attention is brought to topics like machine learning, data mining and natural language processing.

These are all areas within computer science which are not particularly easy and have a higher barrier of entry compared to building a typical web application for instance. Yet, if implemented correctly they can provide great insight and fame and fortune are bound to be near.

Joking aside. I studied artificial intelligence at university and my undergraduate dissertation was focussed on expert systems so I’ve a bit of exposure to these topics but nothing decidedly practical nor within the past decade.

Interesting Books

Some searching on Amazon turned up a couple of books which looked to be talking about the type of tasks I was interested in performing.

One thing these books have in common is that they use code examples in Python.

For nearly half a decade now a lot of my interest has been with Ruby and related technologies and so I’ve largely avoided Python considering the similarity of the space the two languages sit in. A look through the most popular AI books revealed none using examples written in Ruby however.

The other day I came across a copy of another book using Python so I thought I may as well get a bit of hands on experience with the language and take it from there.

Something Practical

In the past I’ve tried to teach myself many languages from books but it has never been until I’ve needed to use the language for something practical that the know-how stuck.

Thankfully I didn’t have to wait long until inspiration struck. As a private investor in the current economic climate, a topic that has been in my mind has been the rate of inflation. Every time I want to check how quickly my assets are being eroded by inflation I have to unleash Google and hope I land on the correct page and from there find the current rate amongst all the other information. This seemed like the perfect fit for yet another single serving web app.

My starting point was to write a Python script to scrape the Office for National Statistics for the current value for the Retail Price Index.

Next up was creating a basic web-app with App Engine. I’d tinkering with App Engine a bit previously so it was relatively straight-forward to follow the tutorials and piece together a single page app. Slightly more complicated was reading up on the Datastore and Memcache APIs but there were plenty of examples to follow.

Conclusion

The end result is ukinflation.appspot.com. It’s basic but deals with the job at hand. It’s also been a good introduction to various components of the App Engine environment: Datastore, Memcache, Cron, templates and the built-in webapp framework.

I’ve used in all of these types of technologies in other forms over the years and it’s amazing that you get so much functionality out of the box and as long as you embrace the constraints, scaling out comes for free.

I’ve still to make my mind up about Python though, maybe I’m just too accustomed to Ruby. Still, I’m glad to have another tool in the toolbox and to not be restricted to any one set of tech.

Share the Love

As always the code is available on GitHub, fork away!

A minimal Rack app to lookup your external IP address

I think I first heard the term “Rack-app” when I started using Heroku to host a couple of projects. I didn’t know an awful lot about Rack but I did know that it helped a great deal when it came to getting my Rails and Sinatra apps off my workstation and onto the web.

Tonight I thought I’d have another quick look at how it works, what it takes to build a Rack-app and what information about the request is available to the handler.

The result was the following, an object which responds to call and returns a HTTP response code, headers and a body:

ip = lambda do |env| 
  [200, {"Content-Type" => "text/plain"}, [env["REMOTE_ADDR"]]]
end
 
run ip

I have this running on ip.stevenwilkin.com and it provides the same useful function as whatismyip.com but without the ads :)

The code is available on GitHub as always.

Minder – a simple web-app monitoring tool written in Ruby

Background

A few months ago I was tasked with migrating and maintaining a bunch of legacy Rails apps and a couple of them were misbehaving. Requests were hanging which was tying-up the front-end Apache child processes and resulting in all the web-apps on the server becoming unresponsive

At the time I didn’t know what was causing the hanging requests, it wasn’t happening in a predictable manner and on the surface I had all the apps dependencies in place.

Seeing as this was a job I was doing on the side I had very limited time to investigate the underlying issue and decided I would have to make do with just knowing when the apps went down so I could ssh in and manually kill off the offending processes.

My initial thought was to use Nagios for monitoring but it seemed excessively heavy-weight for the task at hand so I seized the opportunity to quickly develop a tool perfectly tailored to my needs.

The resulting script, minder, checks a list of domains and if the root of each can’t be read within 10 seconds a notication is sent via XMPP. Nothing more, nothing less. Simple.

Minder proved effective and I was bombarded with instant messages whenever any of the apps went under. Thankfully I found to bit more time to investigate more thoroughly and tracked the issue down to a missing TrueType font used by ImageMagick.

The first iteration of minder was very simple but I thought I’d put a bit of extra effort in to make it easier to customise and share it with the internet-at-large, hopefully someone will get some use out of it.

Prerequisties

  • Two XMPP accounts:

    • one to send notifications from, and
    • a second to receive notifications

    Google Talk or any regular Jabber/XMPP accounts will suffice

  • Blaine‘s xmpp4r-simple gem must be installed:

    sudo gem install xmpp4r-simple

Usage

  • copy minder.yaml.sample to minder.yaml
  • edit minder.yaml to specify the username and password of the sending XMPP account
  • specify your own personal XMPP account (xmpp_to)
  • modify the list of domains to monitor
  • run the script
    $ ./minder.rb
  • Optionally run the script from a Cron job, the following will run every hour:

    0 * * * * /path/to/minder.rb > /dev/null 2>&1

Get the code

Clone the repo from GitHub. Fire any queries to @stevebiscuit.

An experimental lifestream app

Some time ago DW and myself had an enthusiastic conversation about logging various aspects of our electronic lives: emails sent and received, RSS feeds read, incoming and outgoing phone calls, the list goes on.

At that time I’d already started tinkering with apps to track my efforts in the gym and changes in my bodyweight, I’ve used notebooks for this for years and thought it’d be interesting to have this information in electronic form.

Sharpening skills

While I was on the bench waiting for my current contract to start I was sharpening my skills through developing some small projects with the technologies I’d be using. One of those technologies was SQLite which I knew of but didn’t have much hands on experience with.

I started by playing with the Ruby interface to SQLite, using it to insert and retrieve data from a simple database. I’ve since grown to love the simplicity and flexibility of SQLite and will no doubt use it again in the future.

I needed a practical application and thought back to my intentions to track my online activities. Now was as good a time as any to to tick this off the list of potential projects.

When I’d glued together some Ruby to parse the RSS feed of this blog I knew the basics were in place so I started on a simple Sinatra app to act as a web interface.

Stumbling block

I scoured the web for design inspiration and got a good start to the visual aspect of this project but couldn’t make my mind up over a few subtle items. Next thing my agent called: the purchase order finally came through and my contract was to start the next day.

The demands of adjusting to a new work environment meant I felt little motivation to work on anything on my own time and I became more interested in funneling my excess income into the stockmarket. The project joined those other half-finished unfortunates.

A fresh jolt of motivation

After a couple of months on the contract I started feeling a renewed motivation to get something of my own out there. I couldn’t bare the thought of starting another project to not finish it so I set out to to whip this latest app into shape and release it as soon as possible and no later.

I dug through the uncommited changes in my working copy and made some arbitrary decisions which resulted in a design I felt was “good enough.” I modified my RSS parsing script to feed into SQLite and extended it to process my Delicious bookmarks and loved tracks on Last.fm. I created pagination links, wrote a README and sorted out the hosting and cron jobs. Bingo.

Check it out

My original intention was to track much more of my online life and there’s some details I’d like to change but I love releasing code and I’m pleased with this first iteration.

The app is hosted at life.stevenwilkin.com and the code is available on GitHub, I hope you like it.

Home time revisited – redeveloping a CakePHP application with Sinatra

I’ve just released the latest iteration of hometi.me, a little nano-app I’ve mentioned before.

Deliberate Practice

It’s only a trivial app but I’ve a bit of free time before the start of a contract so I thought it would be good practice to redevelop it, a task I’d been ignoring for a long time.

Goodbye CakePHP, Hello Sinatra

The original app was put together quickly with CakePHP, a PHP-based framework I’ve used since my first paid programming gig. Since then though my interest and proficiency with Ruby and it’s frameworks has come more to the forefront. By choosing Sinatra, the app was given access to the rich Ruby ecosystem, including the joyful HAML and SASS.

The interesting aspect of this release is that all the logic is now implemented on the client-side using Javascript. Originally the countdown was calculated on the server-side and passed to the client with an AJAX call, but the server is more-or-less just hosting the markup, stylesheets and Javascript files required to get the app up and running in the browser.

Heroku

As this app doesn’t make use of a database or Cron jobs etc I thought I’d take the opportunity to move it from my own infrastructure and onto the awesomeness of Heroku whose free offering is a lot more performant than my little VPS. Heroku is a service I’ve been playing with for a while now and it makes deploying a Ruby web app into production a breeze, assuming your project can work within it’s limitations.

All that’s left now is for the DNS changes to propagate. I’d like to show a bit more love to this app in the future, the controls to specify your own home time could be more intuitive for instance, but for now I’m content with the improvements I’ve made to it and it’s new home.

Is it passed your home time yet?

node-shoutbox – a simple node.js demo

Last month, at the inaugural Super Happy Dev Castle, my buddy Dave Rice told me about a project he was working on involving node.js, an event-based I/O system for Javascript.

Javascript outside of the browser?

This concept seemed almost ridiculous to me at that time but I thought it was worth a look and cobbled together a quick “Hello World.” I was greatly amused by creating a command-line app with Javascript but did nothing more about it until last week. I’m currently starting out as an independant contractor & have a bit of spare bandwidth so I thought I’d put together something a bit meatier in order to learn more about this system & to also sharpen up my Javascript skills.

Dave had shown me node-comment and I also studied node_chat. From there I had just enough reference material to understand the mechanics of long-polling and to start piecing together a simple real-time commenting system.

What is node?

In and of itself, node is not a web development stack. It provides an I/O platform of which HTTP handling is a part. People are building blogging and wiki engines, FTP servers and command-line tools with this thing, anything I/O based is possible and thankfully micro-frameworks like (fab) can ease the pain of creating web apps.

Node is still very young software, which I have felt the pain of. To serve up static content I wanted to use paperboy as it can be used as middleware within (fab), however mere days after the last update to the project the API it was relying upon in node changed fundamentally. Despite being new to node development I was able to patch paperboy to work with node v0.1.31 and get on with my actual project. Another sticking point was the inability of (fab) to handle POSTed form variables, this functionality is planned for it’s next release though.

Fun

Ultimately, my app is of little practical use: it doesn’t make use of persistance or moderation and node’s suitability for the production environment may itself be questionable at this time, but it was quite a fun experience. From something I more-or-less despised a decade ago I’ve come to develop a real affection for coding Javascript over the past few years.

When I had the basics of the app laid-down I eagerly awaiting the moment when I could have it open in two browser windows and enter a comment into one and see it appear in the second. That moment came and it brought a smile to my face. I suggest you give it a go :)

Get the code

As a wise man once said, code speaks louder than words, so please be my guest and clone my demo on GitHub.