Redux Blog

Technologies behind our Dashboard and Website

(June 15, 2011, Posted in Platform Development )

I've had quite a few requests to go into detail about the technologies we use to power our website and our automated dashboard, so today I'd like to go into detail about that.

Website

Our website is a custom CMS that I built on top of the Sinatra framework. I chose Sinatra for a couple reason: 1) I am a ruby developer, so it's very easy for me to build it exactly how i want it to work, and 2) it's an incredibly lightweight and easy to manage framework.

Most of the pages are static erb templates, while our blog and answers are stored in MongoDB. If you're not familiar with mongo, I highly recommend you look into it; it's a very fast and scalable document-based database.

Composing our blog posts and answers is done using markdown with the redcarpet ruby gem.

Searching on both the blog and answers section is handled by taking all of words in the body & title of the post and create an array of keywords.

Example:

title = title.gsub(/<\/?[^>]*>/, "").gsub(/\W+/, ' ').downcase
content = content.gsub(/<\/?[^>]*>/, "").gsub(/W+/, ' ').downcase
keywords = (title + ' ' + content).split.uniq

This may seem a little 'messy', but it actually works quite nicely. I do some other logic behind the scenes to take out words like 'the' and 'a' to cut out some fat.

Dashboard

Our dashboard is made up of a collection of technologies and languages. The main application is a Rails 3 app running on nginx and Passenger.

When designing the main architecture, we chose to build the app to leverage queuing as much as possible so the user experience would always be fast.

Queue

For queuing, were using Resque which uses the Redis database, and this is monitored with god. To learn more about this, here is a great blog post that the Resque developers put out: https://github.com/blog/542-introducing-resque.

From within the application, all orders, device changes (resize, restart, shutdown, etc), reverse dns, and email are queued up. This is mainly why our dashboard is so quick. It also allows us to handle errors very gracefully, if a worker fails to complete it's job for whatever reason, we can deal with that transparently behind the scenes; the customer doesn't need to be involved in that process.

Provisioning

Provisioning of the VM is handled by a perl driver I wrote to speak with our virtualization platform. Even though I'm by no means a perl developer, I chose perl for it's speed and socket connections. The perl driver spits out the raw data for the ruby-based worker to process the response. In internal testing, we found that this method was substantially faster over letting ruby make the socket requests.

Device Metrics

All of our device metrics are done by a separate set of workers polling all of the devices every few minutes, and then storing that data into MongoDB.

Were using the Highcharts graphing engine to generate our graphs.

If you have any questions, please feel free to contact me at kris at networkredux.com