Guardian SXSW Hack Review

What I built and why

I really wanted to scratch my own itch at the hack weekend with SXSW in mind. There is going to be over a thousand bands playing at the music festival, and many of those will be trying to break through and make it. This means, there’s a fair chance I won’t have heard of many of the bands. Also, the sheer number of bands playing means it’ll be difficult for me to do any quick research about those bands to find out who I should go and see play.

I’ve used Last.FM for nearly four years and have something like approaching 20,000 scrobbled tracks in their dataset, so they have a good impression of the type of music and artists I like listening too and I wanted to tap into that data. Bearing in mind what I just said about not knowing any of the bands playing, I needed a different way to look at the data. Using Matt Andrew’s band listing API, I used the Last.FM API to find around 20 similar artists for each band playing at the festival, leaving me with a dataset of about 20,000 artists that are like those playing at the festival. My thinking here was there would be a good chance of bigger bands would be amongst this dataset and I might have more of a chance of finding a match.

Now, using the top artists from the Last.FM API, I could do an intersection of the artists I like and artists similar to those playing at SXSW. I could then do a look up to see what bands are similar to bands I like, helping me discover new music at the festival without investing too much time doing any research on all of the artists. Yes, it gave me a bit of a headache too.

I only really got this working by lunch time on the second day, and although I had intended to build a simple HTML layer on the data I’d built, I just didn’t have the energy. I went for a coffee, a chat and a nice sit down instead. I don’t think it helped tremendously with the presentation not having any visuals, but I think a few people saw the potential of the application and the data behind it.

What’s next

The hack day is done, but I want to push this project on a little more. The code certainly needs a lot of love and I wasn’t using a complete dataset over the weekend, so I need to import that to get more comprehensive results. I’d also like to make this more generic for any festival, be it SXSW or Glastonbury.

Lessons, observations and stories

The hack weekend was only my second hack event, and the first one I went too was only for the first day of a two day event, so I suppose this was my first full hack event. I’ve been to plenty of small meet-ups, larger Bar Camp events and mammoth conferences, but a hack day definitely has a different and more intimate vibe too it. The number of attendees seemed pretty optimal and it was certainly a good mix of developers, designers and journalists although this only really shone through in the final presentations.

Although having an idea of what I was intending to build at the event, and even doing some thinking about how the application might work (but no coding before the pistol on Saturday morning, as that’s cheating!) it still took me a while to get started. I probably would have coded my idea up in Node.JS as it’s quickly becoming my language of choice in the ‘get something done fast’ category, but as I was building something that I hoped would eventually be brought into the Guardian for festival coverage I thought I should build the application in the fast becoming language of choice at the Guardian, scala.

Scala, although fantastically awesome on one hand, still runs on the JVM and still requires Java like setup with web.xml files and the like. This isn’t a problem in itself, but it means some non-trivial time is spent just setting up a project the way you like it.

This leads nicely into lesson one:
If intending to use a language that requires some investment in setup, look for a way of reducing this, perhaps by having a library of “hello world” applications pre-configured for repeat use.

Keeping with Scala, as a relative newbie to the language I know I was doing certain things in an inefficient way and that was compounded by the timescales of an event like this. When working on a project and you suffer a setback, either you don’t know how to do something or something you thought would work just doesn’t, you generally have time to investigate, ask around and try a few things out. However, that is a real luxury when trying to build things in hours and not weeks.

Lesson Two:
If you’re going to use a hack day to experiment with a new technology, expect frustrations and delays. Even a half hour delay can feel catastrophic.

Just to finish the Scala points, and this may be my lack of knowledge of the language, but I wish the JSON and HTTP support were a little better. Compared to the XML support, which is an excellent xpath like implementation, the JSON support felt clunky. I actually had to change the data I was getting (I was reading from a file, so I could do this) to remove some parts which I couldn’t get the code to parse. As for the HTTP support, I had to bring in the Jetty HTTP client (which didn’t seem to recognise ‘utf-8’ as a character encoding), then bring in the Apache Commons HTTP client to request data from the Last.FM API. One post on Stack Overflow I was reading while I was looking for answers to a Scala problem suggesting having a personal library for wrapping functions you wish were supported better.

Lesson Three:
Knowing how to do common things really well, and fast is essential. In my case, using web based APIs using JSON and rendering JSON in turn.

One technology I did use which I’m now using on a day to day basis is MongoDB and this is where knowing something about a technology really came into it’s own. Getting stuff into and out of the database was as easy as it should be. I used the effective but perhaps slightly noisy Casbah Scala driver to talk to MongoDB. I was also using the Last.FM API to get information about bands and I realised that the Last.FM API was probably one of the first public web APIs I used thanks to a Paul Downey workshop when I was a fresh faced graduate. In a good way the API doesn’t look to have changed, which should be great as anything I built in that workshop might have a chance of working today. However, the web has moved on a bit from RPC and XML and I’d like to see Last.FM offering more of a RESTful JSON based API. That might just have influenced my decision to use Node.JS instead of Scala due to the support of JSON over XML.

As this was a music project, the sensible thing was using Music Brainz ids for the bands, and although this worked in many instances some of the bands playing SXSW don’t yet have Music Brainz ids and perhaps more surprisingly the Last.FM API doesn’t seem to provide Music Brainz ids for all of the bands in it’s API, even top 100 chart bands can be without one. The algorithm I built depended on this data, and although I could do a best effort and call Music Brainz directly, it would have been nice if this was covered off by the larger provider, Last.FM.

Update: I've already found out Last.FM does actually offer JSON in it's API, that would have been handy to know, and sorta proves the lesson of know what you're doing before starting.

Cache me if you can

From the producers who brought you 'Tengo and Cache', we present 'Cache me if you can'.

The concept is all about HTML delivery into iOS devices alongside a mechanism for updating the application without going through the Apple app store. Loading HTML from a local file is not a new idea, as a few companies and projects have sprung up around this type of mobile app development have emerged, most notably Phonegap and more recently Apparatio.

The original iOS app I had built, Tengo and Cache, created a way that took this idea in a slightly different direction. The projects above deliver HTML within an application and would rely on Javascript to update any content within the application, using local storage to persist data. What Tengo and Cache offered was a way to download new HTML documents in their entirety, storing the files in writable areas of the file system on a device. This worked by providing a manifest file, based on the HTML5 manifest, on the domain which was the iOS app was trying to download files from.

During a hack day for the Guardian, I extended this work to add a further downstream cache. Although every effort is made to cache resources before loading, some files may be requested which are not included in the manifest. Overriding the NSURLCache class, the application can intercept any calls which attempt to go out to the web. This cache then retrieves the file, stores it, and then serves this instead of continuing down the pipeline. The next time this file is requested, the cache serves the file on disk instead of hitting the web at all.

Now that I had an iOS application that should pre-cache and intercept, I needed some content to install onto a device. I had wanted to use a Wordpress blog, or RSS feed, but felt that this content could prove difficult. Without knowing what content would be delivered, I couldn't build a manifest file that would capture the entire content. Also, I thought it might prove to be bad user experience if something that would work on a web page when online might not work offline. Even something as ubiquitous as search might look to have failed miserably.

I decided to use something I knew I had control over, the Guardian Content API. Using a small NodeJS application, I could retrieve a query and present that as HTML, alongside an appropriate manifest file. Then all that was required was a small property change in the iOS app and a native application was ready to be launched.

Cache me if you can

View more presentations from Robbie Clutton.

Recipe: Guardian Fried Chicken

This evening I made an attempt at the Guardian Fried Chicken, and although Tim Hayward gives the recipe in the associated video, I wanted to share how I had gotten on.

Although the recipe is simple, it does take some time.

• Soak the chicken in milk overnight and cover with cling film


• Poach the chicken in the milk it was soaked in by bringing to the boil and then simmering for 20 minutes


• Meanwhile combine the spices and blend, then add to plain flour


• When the chicken is cooked, leave to cool in the milk


• Coat the chicken in the spiced flour


• Return to the milk and dip again in the spiced flour


• Place into hot (but not scalding) oil and cook for 5-6 minutes until nicely brown


• Pat down with kitchen paper

I also added some BBQ beans (just normal beans with brown sauce and lots of pepper) and some coleslaw for good measure.

I also had to make some compromises. I don't have a fryer, so I used a pan. Without a temperature guide, I used a trick my Mum used to do which is to drop a small piece of bread into the oil to get a feel for how hot the oil is.

So generally it all went well, although I had the oil too hot, so the coating became too crispy too early and when cut, the chicken looked slightly underdone. I put it in the oven for 10 minutes to finish it off, so while the meat was cooked it also dried out the coating. However, as I was cooking the pieces one at a time, I turned the heat down a lot so that the 5-6 minutes fried cooked the meat all the way through.

The one thing that was missing from the recipe was MSG for the spices. It might have enhanced the flavour more, it was already pretty good. It still didn't have that KFC feel to it though. I think I'll be trying it again though.

Check out the set for more photos.

Latest Scores Makes The Guardian

I was randomly following up on one of my new followers for a latest score tweeter when I found this link to the Guardian: http://www.guardian.co.uk/technology/2008/may/08/socialnetworking.twitter

Towards the bottom under miscellaneous tools is a link off to http://www.code.google.com/p/latestscorestwitter

AWESOME!

My New Years Resolutions

I know it's a little late, but better late then never eh?

My new years resolution is simple really: be more selfish. In the nicest possible way of course. Over the last few months I've found myself being online a lot, in a sort of attached-to-the-hip sort of way.

It really came to me on Saturday when I was sitting on the sofa, enjoying a cup of tea and reading the Guardian. I paused and thought to myself, "this is nice, it's been too long since I just sat and thoroughly read the paper". I went on to think about when I used to go away for the weekend hiking, or mountain biking, or how I used to actually go to the gym regularly, play football and generally be a lot fitter then I am now.

I kept found myself wanting to check my email, Twitter and my RSS reader to see what was going on and thinking back, it seems nothing that I didn't miss that much after all.

Here's an interesting twittersation for you, which left me thinking my new years resolution is to spend less time online, and spend more time doing stuff I used to like doing. (This coming from a man who just wrote four blog posts in an hour!)

To do that, I may have to cull my RSS reader and unfollow people on Twitter, so sorry about that in advance. Please add me to your Google address book though if you use Google Reader, the shared items is a great filter of information.

http://twitter.com/robb1e/statuses/572002322
http://twitter.com/FND/statuses/572157652
http://twitter.com/jayfresh/statuses/572158492
http://twitter.com/FND/statuses/572238092
http://twitter.com/robb1e/statuses/572686122

The customer is always right, and now they've only got themselves to blame

Are you fed up of buying the same old crap from the same old places (for anyone who's been to my flat, please ignore the fact it looks like an Ikea showroom, just for now)? It seems that more power is being given to the people, more power, more choice, more voice!

Though I was taken my Barry Schwartz view on this (an excellent presentation I'm sure you'll agree), it seems now that the small guy is coming through, and they are giving people what they want. Something different.

Threadless.com is a great example here. A community driven clothes selling retail web site. Users can upload their designs and other users can vote for those. Winning* designs are printed by threadless for a limited run. Once those t-shirts sell, users can request another print run though at threadless's discretion. What an awesome idea.

Flicking through Saturday's Guardian, I come across another innovative, if similar idea. Essentially the same, except for household objects. People can upload designs of products and others can suggest changes and make comments. Once that product has a certain amount of votes (currently 1000), the business agrees to take on the design (unless it conflicts with copyright, health and safety laws etc) and create a batch to sell to retail channels. Now, this guy has just set up a channel agreement with Muji, a UK high street distributor. Instead of the threadless payback (kudos, plus some one off cash and gift vouchers - plus potentially more for re-prints and winning further competitions), the designer is given royalties on profits. How awesome is that. Both interesting models.

So now, not only can you buy something that's different, on a limited run, you can now potentially get your own ideas made for yourself and into the homes of others.

These sites let you contribute, but personalisation sites are doing well too, least we forget moo.com =)

*winning defined by threadless.com, not exactly sure what that is as I've not entered a competition

Social Networks and Inner Dialogs

"With Britain home to four million blogs, the inner monologue is in peril. But when everything is made public, something is lost" - Marina Hyde

Amazing! That comment and the rest of the article made me question why I even write blog posts in the first place. Yet, I'm still here, typing away. The part about the inner dialog makes me think about one of JDs daydreams from Scrubs (yes, I love that show). It all got me thinking, there certainly is a lot of information on the superhighway, and it's becoming more and more important how we filter that as more and more people have a voice and use it. I've certainly seen some complete gibberish on some of the social networks out there, along with the many blogs.

Does your inner dialog feel repressed in todays world? I certainly find myself talking/thinking to myself as much as any other time. Except now I think about how I can turn it into a blog post lol.