Recycling Group Finder app for the iPhone

I’ve launched my first iPhone app, the Recycling Group Finder for iPhone. It complements the Recycling Group Finder web app making it even easier to find your closest Freecycle or Freegle group by using the iPhone’s in-built GPS. Check out the information page and give it a go, it’s free.

First of all I made my action respond to the plist format:

Next I created a builder file to format the data:

Then register the MIME type at the bottom of environment.rb:

Mime::Type.register "text/plist", :plist

And that’s it! Well mostly. The XML file generated can be made significantly smaller by converting it into the binary plist format, run this on the command line in terminal after downloading the generated XML plist.

cat things_xml.plist | plutil -convert binary1 - -o things.plist

The resultant binary plist is almost half the size of the XML one, much better for inclusion in an iPhone app:

pleb:~ will$ ls -l things*
-rw-r--r-- 1 will will 1247300 20 Jan 18:50 things.plist
-rw-r--r--@ 1 will will 2110437 20 Jan 18:50 things_xml.plist

Of course it would be much better to generate the binary format directly, and the plist-official gem looks like it can handle that and I mean to investigate, but I wrote the XML version before finding the gem, and it works for me!

This month John Leach of Brightbox will be talking on UNIX: Rediscovering the wheel.

“Those who don’t understand UNIX are condemned to reinvent it, poorly.”

We in the Ruby Community seem to have a habit of re-inventing things. Sometimes this is for good reason, but in some cases we don’t know we’re even doing it! We’re wasting valuable time that could be spent learning Erlang!

UNIX-like operating systems have been around for decades and lots of problems have come and gone in that time. I’m going to talk about some of the tools available that can be used to solve common Ruby and Rails deployment and development problems.

Brightbox will also be sponsoring the event so there will be pizzas available after the meeting in Odder across the road from the meeting.

Schedule

6:30pm :: Welcome & Pre-session bar visit.
7:00pm :: UNIX: Rediscovering the wheel by John Leach of Brightbox
7:45pm :: Pizzas at the Odder bar across the road sponsored by Brightbox

If you want more information email nwrug@willj.net, call Will on 07939 547 962 or tweet @will_j.

Sign Up

If you would like to attend this event please sign up here as the BBC need a list of attendees before the event, and I need to arrange the correct amount of food.

Location

This meeting is being held at one of our regular venues, the BBC Manchester main building on Oxford Road in central Manchester (Directions). If you get lost call Will on 07939 547 962.

OMG wow. Our website is amazing. It’s so good no-one will want to leave it. Ever. Let’s help users enjoy our website forever by making all external links open in new windows so then they close the other websites our site will still be open. Our users will thank us until the end of time for making it easier to stay on our site, and anyway Marketing said we had to do it and they know the internet better than anyone!

Sad Panda

Oh dear. Most people don’t know this, but making external website links open in a new window makes pandas sad. Look, here’s a sad panda made sad because it used a website that opened external links in new windows.

Doesn't this panda look sad?

You did that, with your new window link opening. (Panda by sholt).

Why Pandas cry

You need to consider that your website is going to be just one part of a users browsing session. The user will probably already have open tabs in their current browser window and the tab they have your website in will probably have history before your site. When you force a new window to open for a user you are interrupting their browsing flow. When this happens the user has a jarring user experience because of your website. Well done your website.

There are already controls in browsers to let users open links in new windows or tabs, in Safari they are the the first two options in the right-click context menu, or a Cmd+click:

Browser controls already exist giving the user control over where links open

When you force the user to open links from your website in a new window you are taking away control the user already has.

It’s a PITA and I have to work around it

Here’s what I personally do when your website opens a link in a new window:

1. Your website forces new window to open when I click on a link.
2. New window opens, I close it immediately.
3. On your website again I Cmd+click the link or right click and select ‘open in new tab’.
4. I close the tab your website was in and re-position the new tab with the new website in where the tab for your website used to be.
5. I mentally remove one karma point from your website in my internal website excellence tracker.

Look at the amount of messing around your website made me do. And now, because of this messing around, your website is no-longer accessible via my browser back button. You’ve succeeded in making your website even less accessible, the exact opposite of that you were trying to achieve.

Luckily I’m mentally tough much like Chuck Norris and so can take this two, maybe three times before cracking, but Pandas aren’t as tough as me. If this happened to a panda the panda would just cry. Sad.

$ echo ”flush_all” | nc localhost 11211

However I almost never use the actual shell version of this, mostly I do the equivalent in Ruby by opening up a socket and communicating through that. Here’s a simple example:

socket = TCPSocket.new( '127.0.0.1', 11211 )
socket.write( "flush_all\r\n" )
result = socket.recv(2)
puts "Success!" if result == 'OK'
socket.close

I’ll be starting my eldest daughter off on the first Star Wars film (Episode 4) soon too, it just seems like the natural order. When I had calmed down and stopped frothing at the mouth I explained my worries to him and he asked for references.

Every conversation I have ever had on the subject of the correct order to watch the Star Wars movies in has resulted in an agreement that 4, 5, 6, 1, 2, 3 is the correct order, but I never got it in writing so I decided it was time to get evidence, from Twitter!

The first response was direct to me in IRC:

17:01 Caius: 4,5,6,1,2,3
17:01 Caius: *IF* you let them watch 1,2,3

That’s fairly standard, it’s often debated wether bothering with Episodes 1, 2 and 3 is worth it, though I think I will show 4, 5 and 6 to my daughters eventually. The rest of the responses came quickly on Twitter:

Well that was the sort of response I was expecting, except the one person who suggested Episodes 4, 5 then 6 followed by the Christmas Special. @mibly, you’re sick! Here is the response as a pie-chart to better illustrate the responses:

Not a single person out of my highly representative sample group voted for Episode 1, 2 then 3 first. I kind of agree with @prettierpixels that the in-jokes will be missed if Episodes 1, 2 and 3 are watched first, but I just have a sort of deep-down intuitive feeling that Star Wars will be spoiled if Episodes 4, 5 then 6 aren’t watched first.

Got an opinion? Post it in the comments!

The performance problem

I’ve been very happy with the responsiveness of the Recycling Group Finder, so just out of interest, just to see what it would tell me, I installed the NewRelic RPM plugin and activated the free Bronze account available to EngineYard customers. The results were pretty satisfying as my average response time for the most popular page was 163ms maximum with the second most popular page at 90ms. Those are good response times and fall well within the 37signals response time rule:

Our general rule of thumb is that most pages should render their HTML on the server in less than 200ms and almost all in less than 500ms.

Suspicious looking

One of the great things about data visualisations is it can make it really easy to spot patterns. Take this New Relic graph for example:

Recycling Group Finder - before optimisation

The yellow on the graph represents time spent in the database, the blue is time spent in Ruby, ie. rendering, controllers etc. Memcached accesses are on there too but they’re so fast they hardly appear. This graph looked suspicious to me, I’d normally expect database time to be a much smaller proportion of the overall request time. So it looks like there may be some optimisation that can be done, but in order to optimise I first need to know what to optimise.

The hunt

Google for “rules of optimisation“. Most rules are something like this:

1. Don’t optimise yet
2. If you need to optimise, profile first.

I’m never going to be able to optimise my code unless I know what to optimise. If I trawl through looking for places that might be slow and trying to make them faster the chances are I’m going to spend hours changing code for no benefit. I might even make it slower. I need to know exactly where the bottleneck is, I need to profile my code.

There are a bunch of ways of finding out where your code is slow and I’ve personally used ruby-prof before with good results. However I know that the issue here is in the database, and I know that Rack::Bug will show me SQL queries that have run for an action, and importantly how long they took, so that’s what I’m going to try first. I install the plugin, configure it and load it up. The issue is immediately obvious:

Recycling Group Finder - Rack::Bug SQL queries

Almost all of the SQL that is executed is under 0.5ms per query, there are a few queries at ~4ms but he one query that really stands out is the third one down. At 44.75ms it is more than half of the overall SQL time. Bingo! Now I know what is slow I need to know why it is slow. Time to break out the query analyser.

Fixing it

I needed to dig deeper into that SQL statement to see what it was doing, so I opened up a postgres shell and ran an explain analyse on the query:

The issue seems pretty clear. There is a Sequential scan on groups:

Seq Scan on groups (cost=0.00..626.75 rows=4885 width=363) (actual time=0.038..26.495 rows=5126 loops=1)

A Sequential scan on a large table is going to sink performance. I can see that the sequential scan is definitely the issue in this case as the cost and time taken are significant proportions of the overall query time. I need to eliminate it. Here’s the code that generates that query:

@groups = Group.find(:all, :include => :group_page, :origin => [@location.lat, @location.lng], :limit => 30, :conditions => ["defunct = false AND lat is not null and lng is not null and full_address is not null and full_address != '' and country_code = ?", @location.country_code], :order => 'distance ASC, num_members DESC')

I wrote this code ages ago and re-reading it now I can see that although I am limiting the returned results to 30 rows the query will have to hit every row in the table to determine which rows are in the returned 30 as there are no conditions to the query. Whoops. Looking over the Geokit docs I see there’s a :within condition so I added a :within => 100 to the find. Testing the resultant query in the postgres shell using explain analyse again and the query has dropped to 10ms. Not bad but it’s still using a sequential scan. Adding an index on the conditions speeds up the query further to ~1.2ms:

Not bad when it started out at nearly 45ms. Here is the result reflected in the New Relic graph:

Recycling Group Finder - after optimisation

I deployed the new code approximately in the middle of the graph, it should be pretty obvious where.

Conclusion

Before you can optimise your Ruby on Rails app (or your app in any other framework/language for that matter) you need to know know where to optimise. Tools like Rack::Bug and NewRelic allow you to do this effectively and easily allowing you to direct your attention only on those parts of your app that need the attention.

On the Recycling Group Finder I cut response times drastically in about half an hour. Without knowing exactly where to make the change I would have been left guessing and may never have made the optimisation I did.

It was just rather short.

I was kind of hoping for it to be about twice as long. It cost more than renting a film for the same amount of gameplay time. What happened to games taking 40 hours to complete? *mutter* *grumble* it were all fields etc. It was the only thing that let down an otherwise excellent game, and a fairer price would have been £15 to £20 in my opinion, especially considering the tiny distribution cost as I downloaded the game using Steam.

Maybe I’ll buy it off eBay next time.