We Love Lost is open-source!

May 15th, 2010 by Richard Felix

A few days ago I had the idea to make We Love Lost, a website that celebrates the end of the TV show Lost, and I enlisted the help of an old friend/partner-in-crime, Chris Coyier to help with the front end design/development.

Since I’m in the MediaTemple (ve) beta and I had a couple of servers from them that I wasn’t doing anything with yet, I decided to go ahead and build a Ruby on Rails stack on top of it, including MySQL 5, Apache and Passenger. It was incredibly easy to get things going. It took me around 2 hours of setup/tweaking to get things the way I wanted on the box… then I started writing the app, which ended up taking me less time to write than it took me to configure the (ve). Also, it’s fast. I’ve used other VPS setups recently, most notably Slicehost, and even things like installing packages just fly in comparison. It’s also holding up great under the load (it loads a new tweet for everyone who has the site open, every second).

What I wanted out of the website was a Rails app that would display all tweets hashtagged with #welovelost, at random, while caching them to the database for speed/archival/contest drawing purposes. I wanted for the back-end and front-end to work independently of each other (the back-end would gather new tweets every so often, and the front-end would pick a random one from the database for display). After talking with Chris, we also decided that we wanted for the random tweets to be available via JSON returned from a particular URL, so that the website wouldn’t even have to be reloaded for new tweets to be displayed. I set out to build that, and it turned out to be pretty easy because decoupling of the front-end and back-ends of a service, background tasks and such is the sort of principle that we use on Dispatch to process incoming email/requests and on Are My Sites Up to check websites.

Here’s the Rails project and the SQL database. Download

It uses the following gems, which saved me a lot of time:
lockfile
twitter

The app consists of one controller, with 2 actions: index and show, and a back-end script that runs periodically to grab new tweets. The “index” action is the front page of the website, and in that action we grab the total number of messages in the database so we can display that on the front end… the rest of that page is all front-end dev that Chris whipped up. The “show” action is the action that serves up the JSON for a random tweet from the database, so Chris’s front-end code can call that action to its heart’s content and continually update the tweets without a refresh. Rails makes it super easy to return your results in JSON, but by default, if you do something like this

format.json { render :json => @message }

you’ll get all fields in the database for that model returned, and some I didn’t want to be public… So I did this, which gives me only the fields that I want:

format.json { render :text => @message.to_json(:only => [:screen_name, :text, :profile_image, :link]) }

Now, on the backend, in lib/get_tweets.rb, I wrote a script that gets the last tweet saved to the database, and then uses the Twitter gem to search twitter for all tweets containing “welovelost”, getting the last 100 tweets, and only grabbing tweets that are newer than the last tweet saved to the database. It then saves their username, the tweet text, tweet approval, the URL of their profile image, their twitter id, and a link to the actual tweet on Twitter, by combining the Twitter URL, their screen name, and the tweet id.

ENV["RAILS_ENV"] ||= "development" #change to production on the server
 
require "/Users/richard/projects/welovelost/config/environment.rb" #this line is used whem running locally
#require "/home/welovelost/public_html/welovelost/config/environment.rb" #this line is used on the server
 
require 'net/http'
require 'rubygems'
require 'lockfile'
require 'twitter'
 
begin
  Lockfile.new('get_tweets.lock', :retries => 0) do
    @last_id = Message.find(:first, :order => "id desc")
    @h = Twitter::Search.new("welovelost").per_page(100).since(@last_id.twitter_id).each do |r|
      @exists = Message.find(:first, :conditions => ['twitter_id = ?', r.id])
      if @exists == nil
        @message = Message.new
        @message.screen_name = r.from_user
        @message.text = r.text
        @message.approved = "y"
        @message.profile_image = r.profile_image_url
        @message.twitter_id = r.id
        @message.link = "http://twitter.com/#{@message.screen_name}/status/#{@message.twitter_id}"
        @message.save
      end
    end
 
    puts "Finished running Tweet Fetcher in #{RAILS_ENV} mode"
  end
rescue Lockfile::MaxTriesLockError => e
  puts "Another Tweet Fetcher is already running. Exiting."
end

I wrap the whole thing in a lockfile call. Lockfile is a very useful gem when you’re calling a script periodically via cron and want to make sure that only one instance is running at a time… I have this script being called via cron every 2 minutes, so if one run of the tweet fetcher takes longer than 2 minutes to complete, instead of starting another instance up, it’ll just let the first one finish running.

Here’s my one line cron task that runs the tweet fetcher every 2 minutes:

*/2 * * * * cd /home/welovelost/public_html/welovelost; RAILS_ENV=production script/runner /home/welovelost/public_html/welovelost/lib/get_tweets.rb  >> /home/welovelost/public_html/welovelost/log/tweet_fetcher.log

And that, my friends is how We Love Lost was written.

Join our private mailing list

Get exclusive updates and insights from Sense Labs right in your inbox. (No more than twice a week)

2 comments

  1. This makes a lot of sense. Looks like the next semester break would be spent either with RoR or Django. Both languages seams so logical for a guy who only have PHP experience.

    I have been thinking about “upgrading” my GS to VE since the price difference is so minimal. The more I read about the VE the more convinced I get.

  2. Nice work.. I love the app!

Leave a Comment