github.com/spf13
spf13.com
@spf13
OpenSky eCommerce platform
products in mongo using custom fields for different product types (i.e. actors in movie, tracklist for album etc)
- ordered items need to be fixed at time of purchase, table inheritance bad for this
## 3 things for e-commerce
- optimistic concurrency (update if current, then try again if document is not current)
- assumes environment with low data contention
- works well for Amazon with long tail product catalogue
- works bad for ebay, groupe, anything with flash-sales (high data contention)
# Commerce is ACID in real-life
purchasing something from a store deals with this without concurrency as each product can only be held by one customer
MongoDB e-Commerce
- each item (not sku) has it's own document
- contains
-- reference to sku
-- state
-- other meta-data (timestamp, order ref etc)
cart in card action difficult, but in Mongo changing state on item makes it unavailable to other customers (e.g. if state is 'in-cart')
## Blending
Doctrine (OS - ORM/ODM) modelled on Hibernate
in SQL
-- product inventory, sellable inventory, orders
- inventory is transient in Mongo collections
- inventory kept in sync with listeners
for financial transactions we want security and comfort of RDBMS
## Playing Nice
products are stored in a document
orders are entities stored in relational DB
store reference not relationships across two databases
jwage.com/2010/08/25/blending-the-doctrine-orm-and-mongdb-odm/
Monday, September 19, 2011
Notes from "Schema design at scale"
# Eliot Horowitz
## Embedding sub-documents vs separate collection
i.e. blog post and comments
- embedded
-- Something like a million sub-documents is going to be unwieldy
-- need to move whole document (expensive) if gets large
- Not embedded
-- Million 'comments' in separate documents means lots of reads
- Hybrid
-- one document for core meta-data
-- separate document for n comments with array separated by buckets (i.e. 100 in each bucket)
-- reduces potential seeks (if 100 in each, reduces from say 500 to 5)
## Indexes
- Right-balanced index access on the B-Tree
-- only have to keep small portion in RAM
-- time based, object id, auto-increment
- Keep data sequential in index (covered index)
-- create an index with just the fields you need so you can retrieve the data straight from the index
-- index is bigger
-- good for reads like this
## Shard Key
- determines how data is partitioned
- hard to change
- most important performance decision
- broken into chunks by range
- want to distribute evenly but be right balanced, like month() + md5(something)
- if sharding for logs, need to think
-- why do you want to scale (for write or reading)?
-- 'want to see the last 1000 messages for my app across the system'
-- take advantage of parallelising commands across shares (index by machine, read by app-name)
- no right answer
## Lessons
Range query vs regex (that uses ^ - essentially 'starts-with') is about same performance
If you have a genuinely unique id, use that instead of the ObjectId
## Embedding sub-documents vs separate collection
i.e. blog post and comments
- embedded
-- Something like a million sub-documents is going to be unwieldy
-- need to move whole document (expensive) if gets large
- Not embedded
-- Million 'comments' in separate documents means lots of reads
- Hybrid
-- one document for core meta-data
-- separate document for n comments with array separated by buckets (i.e. 100 in each bucket)
-- reduces potential seeks (if 100 in each, reduces from say 500 to 5)
## Indexes
- Right-balanced index access on the B-Tree
-- only have to keep small portion in RAM
-- time based, object id, auto-increment
- Keep data sequential in index (covered index)
-- create an index with just the fields you need so you can retrieve the data straight from the index
-- index is bigger
-- good for reads like this
## Shard Key
- determines how data is partitioned
- hard to change
- most important performance decision
- broken into chunks by range
- want to distribute evenly but be right balanced, like month() + md5(something)
- if sharding for logs, need to think
-- why do you want to scale (for write or reading)?
-- 'want to see the last 1000 messages for my app across the system'
-- take advantage of parallelising commands across shares (index by machine, read by app-name)
- no right answer
## Lessons
Range query vs regex (that uses ^ - essentially 'starts-with') is about same performance
If you have a genuinely unique id, use that instead of the ObjectId
Notes from "Scaling MongoDB for Real-Time Analytics"
* Scaling MongoDB for Real-Time Analytics, a Shortcut around the mistakes I've made - Theo Hultberg Chief Architect, Burt
40gb, 50 million documents per day
use of mongo
- virtual memory (throwaway)
- short time storage (throwaway)
- full time storage
why
- sharding makes write scale
- secondary indexes
using jRuby
mongo 1.8.1
updates near 50% write lock incrementing single number
pushing near 80% write lock
mongo 2
updates near 15% write lock
push near 50% write lock
iterations
- 1
-- one document per session
-- update as new data comes along
-- 1000% write lock!
-- lesson: everything is about working around the global write lock
- 2
-- multiple documents using same id prefix
-- not as much lock, but still not great performance, couldn't remove data at same pace as inserting
-- lesson: everything is about working around the global write lock
-- being more thought about designing primary key (same prefix for a group)
- 3
-- wrote a new collection every hour
-- lots of complicated code, bugs
-- enables fragmented database files on disk
-- lesson: make sure you can remove old data
- 4
-- sharding
-- higher write performance
-- lots of problems (in 1.8) and ops time spent debugging
-- lesson: everything is about working around the global write lock, sharding is a way around this (4 shards means 4 global write lock)
-- lesson: sharding not a silver bullet (it's buggy, avoid it if you can)
-- lesson: it will fail, design for failure (infrastructure)
- 5
-- move things to separate clusters for different usage (high writes then drop; increment a document)
-- lesson: one database with one usage pattern per cluster
-- lesson: monitor everything (found replica that was 12 hours behind, useless!)
- 6
-- upgraded to monster servers (high memory quad extra large with AWS) (downgraded to extra large machine - 6 machines with mongod 12gb ram, 3 machines for mongo-config, 3 machines for arbiters across three availability zones writing 2000 documents per second, reading 4000 documents per secondc)
-- lesson: call the experts when you're out of ideas
- 7
-- partitioning again and pre-chunking (need to know your data and range of your keys)
-- partition by database, new db each day
-- decrease size of documents (less in RAM)
-- no more problems removing data
-- lesson: smaller objects means smaller documents
-- lesson: think about your primary key
-- lesson: everything is about working around the global write lock
Q&A
- would you recommend?
-- best for high read load, high write load is not a solved problem yet
- EC2
-- you have replicas, why also have EBS?
-- use ephemeral disk, it comes included and is predictable performance (with caveats like spreading across availability zones to avoid data centre loss)
-- use RAID 10 if using EBS
- monitoring
-- mongostat
-- plugins that are available
-- mms from 10gen
-- server density (here at Mongo con)
- map reduce
-- moved away to be more real-time (but was using hadoop)
40gb, 50 million documents per day
use of mongo
- virtual memory (throwaway)
- short time storage (throwaway)
- full time storage
why
- sharding makes write scale
- secondary indexes
using jRuby
mongo 1.8.1
updates near 50% write lock incrementing single number
pushing near 80% write lock
mongo 2
updates near 15% write lock
push near 50% write lock
iterations
- 1
-- one document per session
-- update as new data comes along
-- 1000% write lock!
-- lesson: everything is about working around the global write lock
- 2
-- multiple documents using same id prefix
-- not as much lock, but still not great performance, couldn't remove data at same pace as inserting
-- lesson: everything is about working around the global write lock
-- being more thought about designing primary key (same prefix for a group)
- 3
-- wrote a new collection every hour
-- lots of complicated code, bugs
-- enables fragmented database files on disk
-- lesson: make sure you can remove old data
- 4
-- sharding
-- higher write performance
-- lots of problems (in 1.8) and ops time spent debugging
-- lesson: everything is about working around the global write lock, sharding is a way around this (4 shards means 4 global write lock)
-- lesson: sharding not a silver bullet (it's buggy, avoid it if you can)
-- lesson: it will fail, design for failure (infrastructure)
- 5
-- move things to separate clusters for different usage (high writes then drop; increment a document)
-- lesson: one database with one usage pattern per cluster
-- lesson: monitor everything (found replica that was 12 hours behind, useless!)
- 6
-- upgraded to monster servers (high memory quad extra large with AWS) (downgraded to extra large machine - 6 machines with mongod 12gb ram, 3 machines for mongo-config, 3 machines for arbiters across three availability zones writing 2000 documents per second, reading 4000 documents per secondc)
-- lesson: call the experts when you're out of ideas
- 7
-- partitioning again and pre-chunking (need to know your data and range of your keys)
-- partition by database, new db each day
-- decrease size of documents (less in RAM)
-- no more problems removing data
-- lesson: smaller objects means smaller documents
-- lesson: think about your primary key
-- lesson: everything is about working around the global write lock
Q&A
- would you recommend?
-- best for high read load, high write load is not a solved problem yet
- EC2
-- you have replicas, why also have EBS?
-- use ephemeral disk, it comes included and is predictable performance (with caveats like spreading across availability zones to avoid data centre loss)
-- use RAID 10 if using EBS
- monitoring
-- mongostat
-- plugins that are available
-- mms from 10gen
-- server density (here at Mongo con)
- map reduce
-- moved away to be more real-time (but was using hadoop)
Sunday, June 05, 2011
My first scala app
Well I say first, but that’s not quite the whole story. I’ve worked on existing Scala code bases but recently got the opportunity to start a new project and I got to see how the experiences on those projects and conversations with colleges influenced design decisions. I wanted to note down some of the tricks I’d picked up.
We start with the tools and first up is SBT, version 0.7.x. We depend on the IDEA plugin to generate IDEA project files. We also use the Scaliform plugin which formats code every compile within SBT and the Sources plugin which when run downloads the source Jars without having to stipulate the ‘withSources’ directive on the dependencies.
The IDEA and Sources plugin are run as processors and are not checked into the SBT project file, whereas the Scalariform plugin is. The distinction is that a developer might choose to use a different IDE or not download the sources, but we should always try and conform to the same coding practices and being lazy, having a plugin do it for you is one less thing to worry about.
One of the interesting developments early on was the choice of the dependency injection frameworks. Would it be Spring, Guice or something custom? Actually, we decided not to have any and just use plain old servlets through our routing framework, Scalatra. I think this has actually lead us to write better code, for instance we have taken advantage of the companion objects for classes instead of relaying on repository classes that would need to be injected.
We started with the routing framework, Scalatra which is similar to Sinatra and other simple routing frameworks. It provides a clean way of exposing your application to the web and combined with the following, enables all of the methods to be only a few very focused lines of code. We used Scalate and SSP for the template rendering which can feel a little verbose with the variable declarations at the top, but when you can pre-compile the templates you can understand why those are required.
All new projects are using MongoDB and we’ve been using the Scala driver, Casbah, for a while but this gave us an opportunity to try out Salat the case class wrapper. This is probably as close as I’ve seen ORM like features in Scala while maintaining the MongoDB query sugar. As mentioned, using companion objects instead of repository classes allowed us to do tidy things like:
And use them like:
This meant we were using a repository like model, but hiding it behind the case class and the companion object. It makes for very neat code further upstream.
The app has to call out to other web APIs and for this we use dispatch which wraps the Apache HttpClient project with a bow making it really simple to get resources and convert them to string, XML or JSON. We were already familiar with lift-json so we would get the response as a string and use case classes to parse the JSON into. Once again, XML is almost a joy to work with in Scala compared to other languages, but lift-json still wins in simplicity so we use JSON formatted APIs where we can.
What’s next
I’ve got my eye on Lift and Akka. I’ve had a play with Akka, but I feel I’d like to understand it a little more before using it in a project. Other than that, I think I’m generally getting my head around Scala and functional programming all the more and I’m rather enjoying it. I still feel like I’m just scratching the surface, but it is my first app remember?
Tools
We start with the tools and first up is SBT, version 0.7.x. We depend on the IDEA plugin to generate IDEA project files. We also use the Scaliform plugin which formats code every compile within SBT and the Sources plugin which when run downloads the source Jars without having to stipulate the ‘withSources’ directive on the dependencies.
The IDEA and Sources plugin are run as processors and are not checked into the SBT project file, whereas the Scalariform plugin is. The distinction is that a developer might choose to use a different IDE or not download the sources, but we should always try and conform to the same coding practices and being lazy, having a plugin do it for you is one less thing to worry about.
Inversion of control / Dependency injection
One of the interesting developments early on was the choice of the dependency injection frameworks. Would it be Spring, Guice or something custom? Actually, we decided not to have any and just use plain old servlets through our routing framework, Scalatra. I think this has actually lead us to write better code, for instance we have taken advantage of the companion objects for classes instead of relaying on repository classes that would need to be injected.
Routing
We started with the routing framework, Scalatra which is similar to Sinatra and other simple routing frameworks. It provides a clean way of exposing your application to the web and combined with the following, enables all of the methods to be only a few very focused lines of code. We used Scalate and SSP for the template rendering which can feel a little verbose with the variable declarations at the top, but when you can pre-compile the templates you can understand why those are required.
Databasing
All new projects are using MongoDB and we’ve been using the Scala driver, Casbah, for a while but this gave us an opportunity to try out Salat the case class wrapper. This is probably as close as I’ve seen ORM like features in Scala while maintaining the MongoDB query sugar. As mentioned, using companion objects instead of repository classes allowed us to do tidy things like:
case class Artist(_id: ObjectId = new ObjectId, name: String) {
def save() = Artist.saveArtist(this)
}
object Artist extends SalatDAO[Artist, ObjectId](...mongo config...) {
def saveArtist(artist: Artist) = update(Map(“_id” -> artist._id, artist
}
And use them like:
val artist = Artist(“My new favourite band”)
artist.save()
This meant we were using a repository like model, but hiding it behind the case class and the companion object. It makes for very neat code further upstream.
Webifying
The app has to call out to other web APIs and for this we use dispatch which wraps the Apache HttpClient project with a bow making it really simple to get resources and convert them to string, XML or JSON. We were already familiar with lift-json so we would get the response as a string and use case classes to parse the JSON into. Once again, XML is almost a joy to work with in Scala compared to other languages, but lift-json still wins in simplicity so we use JSON formatted APIs where we can.
What’s next
I’ve got my eye on Lift and Akka. I’ve had a play with Akka, but I feel I’d like to understand it a little more before using it in a project. Other than that, I think I’m generally getting my head around Scala and functional programming all the more and I’m rather enjoying it. I still feel like I’m just scratching the surface, but it is my first app remember?
Sunday, April 10, 2011
Building Curseleague.com
I really wanted an excuse to play with NodeJS once again and after a conversation with blagdaross in the office I had my idea. I had also been looking for an excuse to stretch the legs of Joyents no.de hosting service for NodeJS. I had all of the pieces needed:
Once again I turned to the trusty toolset of ExpressJS for routing and EJS for template rendering. I wanted to build straight out of Twitter lists, so first thing was to hit Twitter to get the members of a list. As this is an open call, no oAuth handshaking has to take place, which is nice. However, getting the list of members returns a whole bunch of metadata about each member on the list. I didn't need any of this information, and I couldn't find all of the members without pagination. This is where I started to have some fun with writing asynchronous code for very much a synchronous web request.
Once I had the list of members of a list, I could do a lookup on Cursebird for the points they have given those members. I have to say the Cursebird API is just fantastic! Talk about the API being the website, just stick .json on the end of any page you visit and get the data you need. No API key nonsense for open data is also a great move. Well done to Cursebird on their efforts. After sorting the list according to points, all it takes is rendering.
I recognised that crunching all of this data with potentially a few hundred server side calls to Twitter and Cursebird would be slow, so I introduced a really dumb key-value pair cache and use that cache at every level, from caching a single request to Twitter or Cursebird, to caching a whole dataset for a webpage.
Onto hosting, and no.de has this really interesting deployment technique where you do a git push into a repository you're given on the virtual server you have. From there a post-push hook is fired which deploys your app and runs a named file (server.js). The no.de real time analytics are really impressive, although it'd be great to get historical information out of that too.
The one on-going grievance I'm dealing with is locally I deploy on port 3000 but in production I deploy on port 80, and currently I'm changing this on commits as a work. I had a way of doing this through a node module I'd built a while ago to get properties from a json file, although up until now I'd been using nDistro to manage my dependencies, but with no.de I'd have to do this a little differently. I could push the properties project into the NPM repositories but I'm also thinking of building a node module that can read nDistro files and add dependencies at run-time. It's food for thought.
Finally a big thank you to JT for working up the styling.
Cheers,
Robbie
Monday, February 14, 2011
Guardian SXSW Hack Review
What I built and why
I really wanted to scratch my own itch at the hack weekend with SXSW in mind. There is going to be over a thousand bands playing at the music festival, and many of those will be trying to break through and make it. This means, there’s a fair chance I won’t have heard of many of the bands. Also, the sheer number of bands playing means it’ll be difficult for me to do any quick research about those bands to find out who I should go and see play.
I’ve used Last.FM for nearly four years and have something like approaching 20,000 scrobbled tracks in their dataset, so they have a good impression of the type of music and artists I like listening too and I wanted to tap into that data. Bearing in mind what I just said about not knowing any of the bands playing, I needed a different way to look at the data. Using Matt Andrew’s band listing API, I used the Last.FM API to find around 20 similar artists for each band playing at the festival, leaving me with a dataset of about 20,000 artists that are like those playing at the festival. My thinking here was there would be a good chance of bigger bands would be amongst this dataset and I might have more of a chance of finding a match.
Now, using the top artists from the Last.FM API, I could do an intersection of the artists I like and artists similar to those playing at SXSW. I could then do a look up to see what bands are similar to bands I like, helping me discover new music at the festival without investing too much time doing any research on all of the artists. Yes, it gave me a bit of a headache too.
I only really got this working by lunch time on the second day, and although I had intended to build a simple HTML layer on the data I’d built, I just didn’t have the energy. I went for a coffee, a chat and a nice sit down instead. I don’t think it helped tremendously with the presentation not having any visuals, but I think a few people saw the potential of the application and the data behind it.
What’s next
The hack day is done, but I want to push this project on a little more. The code certainly needs a lot of love and I wasn’t using a complete dataset over the weekend, so I need to import that to get more comprehensive results. I’d also like to make this more generic for any festival, be it SXSW or Glastonbury.
Lessons, observations and stories
The hack weekend was only my second hack event, and the first one I went too was only for the first day of a two day event, so I suppose this was my first full hack event. I’ve been to plenty of small meet-ups, larger Bar Camp events and mammoth conferences, but a hack day definitely has a different and more intimate vibe too it. The number of attendees seemed pretty optimal and it was certainly a good mix of developers, designers and journalists although this only really shone through in the final presentations.
Although having an idea of what I was intending to build at the event, and even doing some thinking about how the application might work (but no coding before the pistol on Saturday morning, as that’s cheating!) it still took me a while to get started. I probably would have coded my idea up in Node.JS as it’s quickly becoming my language of choice in the ‘get something done fast’ category, but as I was building something that I hoped would eventually be brought into the Guardian for festival coverage I thought I should build the application in the fast becoming language of choice at the Guardian, scala.
Scala, although fantastically awesome on one hand, still runs on the JVM and still requires Java like setup with web.xml files and the like. This isn’t a problem in itself, but it means some non-trivial time is spent just setting up a project the way you like it.
This leads nicely into lesson one:
If intending to use a language that requires some investment in setup, look for a way of reducing this, perhaps by having a library of “hello world” applications pre-configured for repeat use.
Keeping with Scala, as a relative newbie to the language I know I was doing certain things in an inefficient way and that was compounded by the timescales of an event like this. When working on a project and you suffer a setback, either you don’t know how to do something or something you thought would work just doesn’t, you generally have time to investigate, ask around and try a few things out. However, that is a real luxury when trying to build things in hours and not weeks.
Lesson Two:
If you’re going to use a hack day to experiment with a new technology, expect frustrations and delays. Even a half hour delay can feel catastrophic.
Just to finish the Scala points, and this may be my lack of knowledge of the language, but I wish the JSON and HTTP support were a little better. Compared to the XML support, which is an excellent xpath like implementation, the JSON support felt clunky. I actually had to change the data I was getting (I was reading from a file, so I could do this) to remove some parts which I couldn’t get the code to parse. As for the HTTP support, I had to bring in the Jetty HTTP client (which didn’t seem to recognise ‘utf-8’ as a character encoding), then bring in the Apache Commons HTTP client to request data from the Last.FM API. One post on Stack Overflow I was reading while I was looking for answers to a Scala problem suggesting having a personal library for wrapping functions you wish were supported better.
Lesson Three:
Knowing how to do common things really well, and fast is essential. In my case, using web based APIs using JSON and rendering JSON in turn.
One technology I did use which I’m now using on a day to day basis is MongoDB and this is where knowing something about a technology really came into it’s own. Getting stuff into and out of the database was as easy as it should be. I used the effective but perhaps slightly noisy Casbah Scala driver to talk to MongoDB. I was also using the Last.FM API to get information about bands and I realised that the Last.FM API was probably one of the first public web APIs I used thanks to a Paul Downey workshop when I was a fresh faced graduate. In a good way the API doesn’t look to have changed, which should be great as anything I built in that workshop might have a chance of working today. However, the web has moved on a bit from RPC and XML and I’d like to see Last.FM offering more of a RESTful JSON based API. That might just have influenced my decision to use Node.JS instead of Scala due to the support of JSON over XML.
As this was a music project, the sensible thing was using Music Brainz ids for the bands, and although this worked in many instances some of the bands playing SXSW don’t yet have Music Brainz ids and perhaps more surprisingly the Last.FM API doesn’t seem to provide Music Brainz ids for all of the bands in it’s API, even top 100 chart bands can be without one. The algorithm I built depended on this data, and although I could do a best effort and call Music Brainz directly, it would have been nice if this was covered off by the larger provider, Last.FM.
Update: I've already found out Last.FM does actually offer JSON in it's API, that would have been handy to know, and sorta proves the lesson of know what you're doing before starting.
Monday, December 20, 2010
Cache me if you can
From the producers who brought you 'Tengo and Cache', we present 'Cache me if you can'.
The concept is all about HTML delivery into iOS devices alongside a mechanism for updating the application without going through the Apple app store. Loading HTML from a local file is not a new idea, as a few companies and projects have sprung up around this type of mobile app development have emerged, most notably Phonegap and more recently Apparatio.
The original iOS app I had built, Tengo and Cache, created a way that took this idea in a slightly different direction. The projects above deliver HTML within an application and would rely on Javascript to update any content within the application, using local storage to persist data. What Tengo and Cache offered was a way to download new HTML documents in their entirety, storing the files in writable areas of the file system on a device. This worked by providing a manifest file, based on the HTML5 manifest, on the domain which was the iOS app was trying to download files from.
During a hack day for the Guardian, I extended this work to add a further downstream cache. Although every effort is made to cache resources before loading, some files may be requested which are not included in the manifest. Overriding the NSURLCache class, the application can intercept any calls which attempt to go out to the web. This cache then retrieves the file, stores it, and then serves this instead of continuing down the pipeline. The next time this file is requested, the cache serves the file on disk instead of hitting the web at all.
Now that I had an iOS application that should pre-cache and intercept, I needed some content to install onto a device. I had wanted to use a Wordpress blog, or RSS feed, but felt that this content could prove difficult. Without knowing what content would be delivered, I couldn't build a manifest file that would capture the entire content. Also, I thought it might prove to be bad user experience if something that would work on a web page when online might not work offline. Even something as ubiquitous as search might look to have failed miserably.
I decided to use something I knew I had control over, the Guardian Content API. Using a small NodeJS application, I could retrieve a query and present that as HTML, alongside an appropriate manifest file. Then all that was required was a small property change in the iOS app and a native application was ready to be launched.
The concept is all about HTML delivery into iOS devices alongside a mechanism for updating the application without going through the Apple app store. Loading HTML from a local file is not a new idea, as a few companies and projects have sprung up around this type of mobile app development have emerged, most notably Phonegap and more recently Apparatio.
The original iOS app I had built, Tengo and Cache, created a way that took this idea in a slightly different direction. The projects above deliver HTML within an application and would rely on Javascript to update any content within the application, using local storage to persist data. What Tengo and Cache offered was a way to download new HTML documents in their entirety, storing the files in writable areas of the file system on a device. This worked by providing a manifest file, based on the HTML5 manifest, on the domain which was the iOS app was trying to download files from.
During a hack day for the Guardian, I extended this work to add a further downstream cache. Although every effort is made to cache resources before loading, some files may be requested which are not included in the manifest. Overriding the NSURLCache class, the application can intercept any calls which attempt to go out to the web. This cache then retrieves the file, stores it, and then serves this instead of continuing down the pipeline. The next time this file is requested, the cache serves the file on disk instead of hitting the web at all.
Now that I had an iOS application that should pre-cache and intercept, I needed some content to install onto a device. I had wanted to use a Wordpress blog, or RSS feed, but felt that this content could prove difficult. Without knowing what content would be delivered, I couldn't build a manifest file that would capture the entire content. Also, I thought it might prove to be bad user experience if something that would work on a web page when online might not work offline. Even something as ubiquitous as search might look to have failed miserably.
I decided to use something I knew I had control over, the Guardian Content API. Using a small NodeJS application, I could retrieve a query and present that as HTML, alongside an appropriate manifest file. Then all that was required was a small property change in the iOS app and a native application was ready to be launched.
Cache me if you can
View more presentations from Robbie Clutton.
Tuesday, November 23, 2010
Samsung Galaxy Tab - Review
The first thing you notice when you start using a Samsung Galaxy Tablet is that it clearly thinks it’s a phone. Most people I show the device to also think it’s a phone as they do their best Dom Joly impersonation. I had only intended to use the device over wifi but I’m constantly reminded that I haven’t put a SIM card in, and that the phone can only make emergency calls. I noticed at one point that a significant portion of battery life was being put to use on phone related activities. At that point, I put the device into flight mode and then enabled wifi to try and make the battery last longer. Trying not to use the device as a phone means many of the Samsung applications simply don’t work as they require a SIM card, although why they need the SIM card I couldn’t tell you.
One of the things I was looking forward to doing with the device when I got it was to use it around the office as I’m using a desktop and I wanted a way to take notes, look things up and do demos. My first snag there was that the electronic keyboard is just too small for any note taking at length and when I held the device in portrait mode I was typing by thumbs alone. I’ve found that the screen on the Galaxy is just about right to web pages, and due to the size, I can hold the device in one hand, not unlike the Kindle. Many websites are redirecting me to their mobile version, which although can look very nice on the size of the device, it sort of deflects the purpose of having a tablet as opposed to a phone.
Some of the plain oddities are exposed when trying to use Google Docs on the phone. Having sent you to the mobile version, you can edit documents and spreadsheets, but not presentations. When trying to change to desktop mode you get a bizarre error about the browser not supporting web word processing. There was me thinking that all that was needed was HTML rendering, Javascript processing and an Internet connection.
I have felt I’m getting benefit when you consider just about how ‘cloud-enabled’ the device is. I’m yet to plug the device in to my computer, apart from trying to deploy an application I’d built onto it. For music I’ve been using Spotify, Last.FM or iPlayer for Radio 6. Tools like Google Voice have let me find, download and play podcasts without needing desktop software like iTunes with varying success. The speakers on the device are just about good enough to carry around the house with you, or plug the headphone jack into stereos in different rooms. I’ve used DropBox to drop video files onto the device and they just play without having to configure anything or install any codecs.
I’m generally enjoying being able to think of this as more of a computer than a locked down device though, by seeing the running applications and being able to navigate the file system. However there seem to be simple things I just can’t find, like changing the auto-lock timeout or being able to wake the device from being locked other than pushing a button at the top of the device, when my hands tend to be at the other end of the device.
I’m happy that Apple has some competition in the tablet market, but I think the experience needs to improve a lot, through both the physical device and the software that powers it. I’d suggest a good start would be to get the device to stop thinking it’s a phone. The Amazon Kindle has a SIM card for 3G access, but it’s knows it isn’t a phone. It may be a small point, but I prefer my devices without the split personality.
One of the things I was looking forward to doing with the device when I got it was to use it around the office as I’m using a desktop and I wanted a way to take notes, look things up and do demos. My first snag there was that the electronic keyboard is just too small for any note taking at length and when I held the device in portrait mode I was typing by thumbs alone. I’ve found that the screen on the Galaxy is just about right to web pages, and due to the size, I can hold the device in one hand, not unlike the Kindle. Many websites are redirecting me to their mobile version, which although can look very nice on the size of the device, it sort of deflects the purpose of having a tablet as opposed to a phone.
Some of the plain oddities are exposed when trying to use Google Docs on the phone. Having sent you to the mobile version, you can edit documents and spreadsheets, but not presentations. When trying to change to desktop mode you get a bizarre error about the browser not supporting web word processing. There was me thinking that all that was needed was HTML rendering, Javascript processing and an Internet connection.
I have felt I’m getting benefit when you consider just about how ‘cloud-enabled’ the device is. I’m yet to plug the device in to my computer, apart from trying to deploy an application I’d built onto it. For music I’ve been using Spotify, Last.FM or iPlayer for Radio 6. Tools like Google Voice have let me find, download and play podcasts without needing desktop software like iTunes with varying success. The speakers on the device are just about good enough to carry around the house with you, or plug the headphone jack into stereos in different rooms. I’ve used DropBox to drop video files onto the device and they just play without having to configure anything or install any codecs.
I’m generally enjoying being able to think of this as more of a computer than a locked down device though, by seeing the running applications and being able to navigate the file system. However there seem to be simple things I just can’t find, like changing the auto-lock timeout or being able to wake the device from being locked other than pushing a button at the top of the device, when my hands tend to be at the other end of the device.
I’m happy that Apple has some competition in the tablet market, but I think the experience needs to improve a lot, through both the physical device and the software that powers it. I’d suggest a good start would be to get the device to stop thinking it’s a phone. The Amazon Kindle has a SIM card for 3G access, but it’s knows it isn’t a phone. It may be a small point, but I prefer my devices without the split personality.
Saturday, September 11, 2010
nodejs, ndistro and git submodules
On the (very early) tapas platform at theteam I got a little stuck when mixing nDistro and git submodules and wanted to explain what I'd done to get around those issues. On the tapas-models module, MongoDB is used for data storage and tapas-models uses Mongoose for MongoDB integration, but nDistro doesn't download Mongoose's downstream dependencies
This is because nDistro downloads the tarball of the project from Github and that tarball doesn't include dependencies. It might be nice for Github to do this, and I'll search the Github Support site to see if I can find something about that. Anyhoo, the tarball doesn't contain it's git bindings, so I can't go in there and update the git submodules.
The unfortunate thing is I've had to expose myself to some of the Mongoose inner workings to get the dependencies, but once I know them, it's a light touch for the next part. Using nDistro as normal, I include that dependency, including the revision number in my .ndistro file. As nDistro is executed in a bash environment, shell scripting can be used alongside the module declarations. So, I use a Linux move command to put the dependency where Mongoose expects it and everyone's happy.
It's a simple solution to what was a nagging issue and keeps me on my happy path toward NNNN
This is because nDistro downloads the tarball of the project from Github and that tarball doesn't include dependencies. It might be nice for Github to do this, and I'll search the Github Support site to see if I can find something about that. Anyhoo, the tarball doesn't contain it's git bindings, so I can't go in there and update the git submodules.
The unfortunate thing is I've had to expose myself to some of the Mongoose inner workings to get the dependencies, but once I know them, it's a light touch for the next part. Using nDistro as normal, I include that dependency, including the revision number in my .ndistro file. As nDistro is executed in a bash environment, shell scripting can be used alongside the module declarations. So, I use a Linux move command to put the dependency where Mongoose expects it and everyone's happy.
It's a simple solution to what was a nagging issue and keeps me on my happy path toward NNNN
Thursday, August 19, 2010
Recipe: Guardian Fried Chicken
This evening I made an attempt at the Guardian Fried Chicken, and although Tim Hayward gives the recipe in the associated video, I wanted to share how I had gotten on.
Although the recipe is simple, it does take some time.
I also added some BBQ beans (just normal beans with brown sauce and lots of pepper) and some coleslaw for good measure.
I also had to make some compromises. I don't have a fryer, so I used a pan. Without a temperature guide, I used a trick my Mum used to do which is to drop a small piece of bread into the oil to get a feel for how hot the oil is.
So generally it all went well, although I had the oil too hot, so the coating became too crispy too early and when cut, the chicken looked slightly underdone. I put it in the oven for 10 minutes to finish it off, so while the meat was cooked it also dried out the coating. However, as I was cooking the pieces one at a time, I turned the heat down a lot so that the 5-6 minutes fried cooked the meat all the way through.
The one thing that was missing from the recipe was MSG for the spices. It might have enhanced the flavour more, it was already pretty good. It still didn't have that KFC feel to it though. I think I'll be trying it again though.
Check out the set for more photos.
Although the recipe is simple, it does take some time.
- Soak the chicken in milk overnight and cover with cling film
- Poach the chicken in the milk it was soaked in by bringing to the boil and then simmering for 20 minutes
- Meanwhile combine the spices and blend, then add to plain flour
- When the chicken is cooked, leave to cool in the milk
- Coat the chicken in the spiced flour
- Return to the milk and dip again in the spiced flour
- Place into hot (but not scalding) oil and cook for 5-6 minutes until nicely brown
- Pat down with kitchen paper
I also added some BBQ beans (just normal beans with brown sauce and lots of pepper) and some coleslaw for good measure.
I also had to make some compromises. I don't have a fryer, so I used a pan. Without a temperature guide, I used a trick my Mum used to do which is to drop a small piece of bread into the oil to get a feel for how hot the oil is.
So generally it all went well, although I had the oil too hot, so the coating became too crispy too early and when cut, the chicken looked slightly underdone. I put it in the oven for 10 minutes to finish it off, so while the meat was cooked it also dried out the coating. However, as I was cooking the pieces one at a time, I turned the heat down a lot so that the 5-6 minutes fried cooked the meat all the way through.
The one thing that was missing from the recipe was MSG for the spices. It might have enhanced the flavour more, it was already pretty good. It still didn't have that KFC feel to it though. I think I'll be trying it again though.
Check out the set for more photos.
Thursday, August 05, 2010
Tengo and Cache
This project was born from a desire to push on from some work of creating iPhone applications using only web technologies that I had done some time ago. That work can be seen http://github.com/robb1e/iWeb. This first project was a way of deploying a HTML/CSS/Javascript application onto an iOS device and have it run as a native application.
The issue with that was the only way to change HTML, CSS, Javascript, images or other data was to depend on a live internet connection or do an application update. I was looking for a way to make it easier to cache those resources with a specific eye on offline experience.
After some thought, the lightbulb switched on and it became an immediate choice to use the HTML5 Cache Manifest file as a basis of discovering what resources should cached. This project aims to do just that.
Between the application bootstrapping and a webpage being rendered, the cache manifest is read and each file is downloaded and stored in the iOS application sandbox directory. After all files have been saved, the UIWebView will render an index.html file in that sandbox.
Check it out: http://github.com/robb1e/Tengo-and-Cache
The issue with that was the only way to change HTML, CSS, Javascript, images or other data was to depend on a live internet connection or do an application update. I was looking for a way to make it easier to cache those resources with a specific eye on offline experience.
After some thought, the lightbulb switched on and it became an immediate choice to use the HTML5 Cache Manifest file as a basis of discovering what resources should cached. This project aims to do just that.
Between the application bootstrapping and a webpage being rendered, the cache manifest is read and each file is downloaded and stored in the iOS application sandbox directory. After all files have been saved, the UIWebView will render an index.html file in that sandbox.
Check it out: http://github.com/robb1e/Tengo-and-Cache
Thursday, July 08, 2010
Json and Grails
Recently I was trying to return JSON data from a Grails app and I was having some issues. Grails has some built in conversion utilities to enable code like
This is all well and good, but I have a few fundamental issues with this. First off, attributes like fully-qualified class name and the database primary key are returned. I don't really want to expose internal information to client side applications. It also doesn't allow any renaming of the attributes. Say you have a String called firstName using camel case in your domain object, it returns that to the client. What if you want the API to return first_name?
The answer was a lot simpler then my searching could muster, and that is to use the template system, Sitemesh in this case, to render exactly what you want in JSON as you would if you were rendering HTML. OK, it's a bit of a headache, but you have complete control over the content and style. Combine that with a one liner in the controller to set the content type, and you're flying.
render myObject as JSON
This is all well and good, but I have a few fundamental issues with this. First off, attributes like fully-qualified class name and the database primary key are returned. I don't really want to expose internal information to client side applications. It also doesn't allow any renaming of the attributes. Say you have a String called firstName using camel case in your domain object, it returns that to the client. What if you want the API to return first_name?
The answer was a lot simpler then my searching could muster, and that is to use the template system, Sitemesh in this case, to render exactly what you want in JSON as you would if you were rendering HTML. OK, it's a bit of a headache, but you have complete control over the content and style. Combine that with a one liner in the controller to set the content type, and you're flying.
response.contentType = "text/javascript"
Friday, June 25, 2010
Spider and Validate
One of the things we've been trying to achieve is automation is every sense. As my old mate Otu once said "automate until it hurts!" Well, hopefully this will make it a little less painful. We wanted to point to a single page, have the application spider the entire site finding pages which return anything apart from 200 OK, and validate all those who do return 200 OK.
By combining two PHP projects, PHPCrawl (http://sourceforge.net/projects/phpcrawl/) and the frontend-test-suite (http://github.com/NeilCrosby/frontend-test-suite) it's been possible to do just that. Throw a little Ant build script in there too and deploy into a continuous integration container like Hudson, and you start to get a feel for the state of websites in development, or even in production as part of a monitoring tool.
To make use of this, edit the build.xml file to change the SITE_URL to be the endpoint which you want to test and run 'ant validate'. This will do an Ant copy with filtering to create a file called test.php. Ant then runs this and then captures the output from this. It collects which pages may have returned a 500, a 404 and 200 OK pages. It then passes an array of 200 OK pages into the frontend-test-suite which uses PHPUnit to report on HTML validation.
There's still plenty of work to do, mainly introducing the ability to supply your own W3C validator endpoint and introducing CSS validation. It'll come though, so keep an eye out!
The result of this is over at Github - http://github.com/robb1e/Validator
As a side note, to get frontend-test-suite as a submodule on Github I followed instructions from FND, thanks!
By combining two PHP projects, PHPCrawl (http://sourceforge.net/projects/phpcrawl/) and the frontend-test-suite (http://github.com/NeilCrosby/frontend-test-suite) it's been possible to do just that. Throw a little Ant build script in there too and deploy into a continuous integration container like Hudson, and you start to get a feel for the state of websites in development, or even in production as part of a monitoring tool.
To make use of this, edit the build.xml file to change the SITE_URL to be the endpoint which you want to test and run 'ant validate'. This will do an Ant copy with filtering to create a file called test.php. Ant then runs this and then captures the output from this. It collects which pages may have returned a 500, a 404 and 200 OK pages. It then passes an array of 200 OK pages into the frontend-test-suite which uses PHPUnit to report on HTML validation.
There's still plenty of work to do, mainly introducing the ability to supply your own W3C validator endpoint and introducing CSS validation. It'll come though, so keep an eye out!
The result of this is over at Github - http://github.com/robb1e/Validator
As a side note, to get frontend-test-suite as a submodule on Github I followed instructions from FND, thanks!
Tuesday, June 22, 2010
Grails multiple web sites, same web application
We had a need recently to have one web application serving many web sites, where the same data schema was in place with data shared across multiple web sites, but we wanted to show different styling and different data depending on the URI. In the end it turned out to be fairly simple. With our naming conventions, it's possible to hit one of several URIs for the same site, such as:
* project.testing.com
* project.staging.com
* project.com
So I had created a domain lookup service that when passed the request URI could work out what site the request for targeting.
I did a lookup for that target name against an object stored in the database relating to that targetUri and passed that back, making it available for queries for the correct data to show, and passing that down into the views to render the relevant CSS and Javascript files.
* project.testing.com
* project.staging.com
* project.com
So I had created a domain lookup service that when passed the request URI could work out what site the request for targeting.
def targetUri = request.getServerName()
I did a lookup for that target name against an object stored in the database relating to that targetUri and passed that back, making it available for queries for the correct data to show, and passing that down into the views to render the relevant CSS and Javascript files.
Thursday, June 17, 2010
Grails, Ant, Hudson and automated deployments
One of the things I like about Grails is the embedded server, it makes deployment and getting things up and running nice and simple. I've tried building a WAR and deploying into Jetty and Tomcat, but for some reason the system resources seem to spike when deployed in this fashion compared to the built in 'run-app' functions of the Grails command line tools.
With this in mind, I wanted to use the Grails command line to do automated deployment based off our Hudson continuous integration container via Ant build scripts.
First I created an init.d start/stop script for the server. At theTeam we can be working on multiple projects at the same time, so we need to set some conventions about port usage so we can manage our applications easier, so this is the first step towards just that.
This script is simple enough, on start it checks to see if that port is used, if so it bombs out, if not it nohup's the Grails command, telling it which port to use, and also just for ease of management, adding the name of the site. This means running commands like ps -ef and pipping to grep shows what's running. Stopping searches for a process running with that port and kills it.
That's the first step. Next is the Ant script to deploy the latest code after tests and static analysis. In this, the target folder has the latest code, we stop any server that's running, do a Grails clean and then run the server. The script then waits to see if it can open an socket to the port that the server is running on, and fails it it cannot. This usually means the server has failed to startup. The variables are gathered from a properties file and should be fairly obvious.
Now, you might have noticed a few oddities in the script like setting JAVA_HOME and BUILD_ID. First, when running in Hudson, any processes are run using the JRE and JDK that is deployed with Hudson, and I wanted to ensure it was the system Java I had installed. The BUILD_ID is a little trick that tells Hudson not to terminate the process after the build has finished, which it had a habit of doing in earlier versions.
When starting Hudson, a flag should also be set to tell it not to kill child processes. My start command is thus:
Adding this all together means it's possible to get automated deployments with Grails added into your builds.
With this in mind, I wanted to use the Grails command line to do automated deployment based off our Hudson continuous integration container via Ant build scripts.
First I created an init.d start/stop script for the server. At theTeam we can be working on multiple projects at the same time, so we need to set some conventions about port usage so we can manage our applications easier, so this is the first step towards just that.
This script is simple enough, on start it checks to see if that port is used, if so it bombs out, if not it nohup's the Grails command, telling it which port to use, and also just for ease of management, adding the name of the site. This means running commands like ps -ef and pipping to grep shows what's running. Stopping searches for a process running with that port and kills it.
That's the first step. Next is the Ant script to deploy the latest code after tests and static analysis. In this, the target folder has the latest code, we stop any server that's running, do a Grails clean and then run the server. The script then waits to see if it can open an socket to the port that the server is running on, and fails it it cannot. This usually means the server has failed to startup. The variables are gathered from a properties file and should be fairly obvious.
Now, you might have noticed a few oddities in the script like setting JAVA_HOME and BUILD_ID. First, when running in Hudson, any processes are run using the JRE and JDK that is deployed with Hudson, and I wanted to ensure it was the system Java I had installed. The BUILD_ID is a little trick that tells Hudson not to terminate the process after the build has finished, which it had a habit of doing in earlier versions.
When starting Hudson, a flag should also be set to tell it not to kill child processes. My start command is thus:
Adding this all together means it's possible to get automated deployments with Grails added into your builds.
AWS S3 Integration with Grails
I’ve been playing with the S3 plugin for a Grails project and wanted to share how I set it up, as the documentation on the site doesn’t cover an example.
First, in my model I added a byte array field and made it transient. Transient basically means it doesn’t add the bytes to the database, but just keeps the data in memory. This means you can take advantage of all of the automatically generated controllers and view classes which are created from the model without having to really deal with the contents.
class ImageHolder{
byte[] image
String imageUri
static transients = ['image']
}
In the controller we use the request object to get the file and save it to disk, removing any spaces that may have crept in. Then we start using the plugin, and create a S3Asset from the file and put it into the asset service class. The assetService must also be declared at the top, and it’s injected via Spring from the plugin.
def s3AssetService
....
def imageholder = new ImageHolder(params)
....
def mhsr = request.getFile('image')
def fileName = mhsr.getOriginalFilename().replace(' ','_')
def ext = fileName.substring(fileName.lastIndexOf(".")+1);
def file = new File(fileName)
if (!mhsr?.empty && mhsr.size < 1024*200){
mhsr.transferTo(file)
def asset = new S3Asset(file)
asset.mimeType = ext
s3AssetService.put(asset)
imageholder.imageUri = asset.url()
imageholder.save()
}
What this does, is now uses the async service to put the file into S3, so your request returns very quickly. You can see that the image URI is now persisted into the domain object, but the content isn’t.
One final thing is to configure AWS in config.groovy. I set the flag to not add a prefix to the bucket name, as I’m using reverse DNS/package style bucket name to ensure uniqueness.
aws {
domain="s3.amazonaws.com"
accessKey="YOURACCESSKEY"
secretKey="YOURSECRETKEY"
bucketName="your.bucket.name"
prefixBucketWithKey=false
timeout=3000
}
First, in my model I added a byte array field and made it transient. Transient basically means it doesn’t add the bytes to the database, but just keeps the data in memory. This means you can take advantage of all of the automatically generated controllers and view classes which are created from the model without having to really deal with the contents.
class ImageHolder{
byte[] image
String imageUri
static transients = ['image']
}
In the controller we use the request object to get the file and save it to disk, removing any spaces that may have crept in. Then we start using the plugin, and create a S3Asset from the file and put it into the asset service class. The assetService must also be declared at the top, and it’s injected via Spring from the plugin.
def s3AssetService
....
def imageholder = new ImageHolder(params)
....
def mhsr = request.getFile('image')
def fileName = mhsr.getOriginalFilename().replace(' ','_')
def ext = fileName.substring(fileName.lastIndexOf(".")+1);
def file = new File(fileName)
if (!mhsr?.empty && mhsr.size < 1024*200){
mhsr.transferTo(file)
def asset = new S3Asset(file)
asset.mimeType = ext
s3AssetService.put(asset)
imageholder.imageUri = asset.url()
imageholder.save()
}
What this does, is now uses the async service to put the file into S3, so your request returns very quickly. You can see that the image URI is now persisted into the domain object, but the content isn’t.
One final thing is to configure AWS in config.groovy. I set the flag to not add a prefix to the bucket name, as I’m using reverse DNS/package style bucket name to ensure uniqueness.
aws {
domain="s3.amazonaws.com"
accessKey="YOURACCESSKEY"
secretKey="YOURSECRETKEY"
bucketName="your.bucket.name"
prefixBucketWithKey=false
timeout=3000
}
Wednesday, June 16, 2010
Apple, standards and plugins
I hate to sound dramatic but with Apple sounding off about web standards over plugins with their latest products I was wondering just how much Apple was dragging he industry forward, or if it was doing the opposite.
Some years ago, developing web applications meant having to know the details about the browsers your audience were using. We're still suffering from this today with discussion over different box models and questions about just when are Microsoft going to release a standards compliant browser. One example of this is anyone who uses Exchange for email. They might know the pain of using the web client given how rich it is in Internet Explorer and how average it is in any other browser.
The advent of cross browser JavaScript libraries helped reduce many of these pains and it's great to see a lot of work going into the new HTML5 specs that are taking lessons direct from these libraries. However, whilst singing the praises of a few websites that are using the new HTML5 audio and video elements, Apple seem to be creating their own events for touch that I can't see anywhere in the specs. It seems Mozilla are also playing the same game also and I'm worried that we're returning to a world where we have to tailor lots of code to target specific browsers.
You only have to go to Nike.com or log into Gmail to see iPad websites that have orientation and swipe support. It's all very nice but how much extra work is going to have to take place to get a website to work in different browsers and different devices? What happens when a new device comes out with different events, or even as with the new iPhone and it's different screen resolutions, how will that change the work already in the wild?
This raises further questions for me, especially after reports of JavaScript execution speed on the iPad , even though some frameworks claim this isn't so much of a problem, such as Spoutcore Touch
and others go further saying that no framework should be used at all to ensure performance
Although Apple have created fairly complete documentation about building web content for their devices, it does beg the question why there should be differences if we are supposed to live in a standards world? How long will it be before you hit a webpage and have it say "The website is best viewed with iPad", or any other device or browser for that matter?
Should we ignore the devices ad go for standards like the BBC iPlayer for big screens like PS3 and iPad , use libraries to augment sites like how apple.com/iphone allows the user to swipe the image carousel, tailor something specific for each device such as Nike.com or should we just be building native applications?
How should we progress? Should we let our analytics decide? Should we take the standards route or device specific route? This will certainly be one to keep an eye on.
Some years ago, developing web applications meant having to know the details about the browsers your audience were using. We're still suffering from this today with discussion over different box models and questions about just when are Microsoft going to release a standards compliant browser. One example of this is anyone who uses Exchange for email. They might know the pain of using the web client given how rich it is in Internet Explorer and how average it is in any other browser.
The advent of cross browser JavaScript libraries helped reduce many of these pains and it's great to see a lot of work going into the new HTML5 specs that are taking lessons direct from these libraries. However, whilst singing the praises of a few websites that are using the new HTML5 audio and video elements, Apple seem to be creating their own events for touch that I can't see anywhere in the specs. It seems Mozilla are also playing the same game also and I'm worried that we're returning to a world where we have to tailor lots of code to target specific browsers.
You only have to go to Nike.com or log into Gmail to see iPad websites that have orientation and swipe support. It's all very nice but how much extra work is going to have to take place to get a website to work in different browsers and different devices? What happens when a new device comes out with different events, or even as with the new iPhone and it's different screen resolutions, how will that change the work already in the wild?
This raises further questions for me, especially after reports of JavaScript execution speed on the iPad , even though some frameworks claim this isn't so much of a problem, such as Spoutcore Touch
and others go further saying that no framework should be used at all to ensure performance
Although Apple have created fairly complete documentation about building web content for their devices, it does beg the question why there should be differences if we are supposed to live in a standards world? How long will it be before you hit a webpage and have it say "The website is best viewed with iPad", or any other device or browser for that matter?
Should we ignore the devices ad go for standards like the BBC iPlayer for big screens like PS3 and iPad , use libraries to augment sites like how apple.com/iphone allows the user to swipe the image carousel, tailor something specific for each device such as Nike.com or should we just be building native applications?
How should we progress? Should we let our analytics decide? Should we take the standards route or device specific route? This will certainly be one to keep an eye on.
Saturday, September 05, 2009
iPhone development using standard web technologies
Fresh after giving my talk from BarCampBright4, I wanted to get my notes down from my session on iPhone development using standard web technologies. This all started when I wanted to do an iPhone application for my Latest Scores Twitter app. So I bought a book and made my way through it slowly, trying to decode Objective C before coming across the part I was really after, the UIWebView class.
What I ended up arriving at was the ability to load and render a HTML file with it's own resources imported such as CSS, Javascript and images. The real magic in Objective C is this:
What this is doing is creating a URL with a resource on the path called 'index' with the file extension 'html'. Build a request from this URL and pass the request into the instance of UIWebView.
From that point, as long as all of the resources that the index page requires are bundled into the project, you can load the page and have that actually be the application.
So that's all nice if you have a certain type of application which doesn't require any device interaction. For example, if you had a blog/news site and wanted to make it into an application, you could just get the content and display. You can also change a few settings in a file called info.plist to set the icon image and application name.
That template project is available on GitHub, http://github.com/robb1e/iWeb
However, if you want device interaction you'll need a richer framework, and that's where something like PhoneGap.com comes in. With the abstracted Javascript file that works for iPhone, Blackberry and Android, functions are available to get access to the contact list, audio playback and other device features.
To play an embedded audio resource for example, you can write the following Javascript, provided you've included the PhoneGap.js file.
With both of these projects there is one caveat with Xcode in that it wants to compile Javascript files. You have to make sure you set those files as resources to copy, not to compile. This can be done in the target list on the left hand side.
To extend the idea of using standard web technologies whilst not building web sites, this next example uses Adobe's AIR run time and the HTML, CSS, Javascript and image from the first example. This is also available on GitHub: http://github.com/robb1e/AirWeb
By creating the application.xml file, you can tell Air to launch a file as the initial content and that will be rendered using it's WebKit engine.
To run the desktop app, install the Adobe AIR SDK then run the command:
This launches the application in debug mode. To create an Air file, you need to compile using ADT.
This has shown that using the same technologies and languages you'd use to build a web page can be used for both mobile devices with and without device interaction and client applications that are installed on a computer.
There are other ways of achieving the same result using Apple's web namespace tags in the markup with HTML5 tricks like offline storage. There's also other libraries like jQTouch, so check out what else is around too.
The "write once, run anywhere" idea may just be coming into reality .
What I ended up arriving at was the ability to load and render a HTML file with it's own resources imported such as CSS, Javascript and images. The real magic in Objective C is this:
NSURL *url = [NSURL fileURLWithPath:[ [ NSBundle mainBundle ] pathForResource: @"index" ofType:@"html" ]];
NSURLRequest *request = [NSURLRequest requestWithURL:url];
[webView loadRequest:request];
What this is doing is creating a URL with a resource on the path called 'index' with the file extension 'html'. Build a request from this URL and pass the request into the instance of UIWebView.
From that point, as long as all of the resources that the index page requires are bundled into the project, you can load the page and have that actually be the application.
So that's all nice if you have a certain type of application which doesn't require any device interaction. For example, if you had a blog/news site and wanted to make it into an application, you could just get the content and display. You can also change a few settings in a file called info.plist to set the icon image and application name.
That template project is available on GitHub, http://github.com/robb1e/iWeb
However, if you want device interaction you'll need a richer framework, and that's where something like PhoneGap.com comes in. With the abstracted Javascript file that works for iPhone, Blackberry and Android, functions are available to get access to the contact list, audio playback and other device features.
To play an embedded audio resource for example, you can write the following Javascript, provided you've included the PhoneGap.js file.
new Media('beep.wav').play();
With both of these projects there is one caveat with Xcode in that it wants to compile Javascript files. You have to make sure you set those files as resources to copy, not to compile. This can be done in the target list on the left hand side.
To extend the idea of using standard web technologies whilst not building web sites, this next example uses Adobe's AIR run time and the HTML, CSS, Javascript and image from the first example. This is also available on GitHub: http://github.com/robb1e/AirWeb
By creating the application.xml file, you can tell Air to launch a file as the initial content and that will be rendered using it's WebKit engine.
<content>www/index.html</content>
To run the desktop app, install the Adobe AIR SDK then run the command:
adl application.xml
This launches the application in debug mode. To create an Air file, you need to compile using ADT.
This has shown that using the same technologies and languages you'd use to build a web page can be used for both mobile devices with and without device interaction and client applications that are installed on a computer.
There are other ways of achieving the same result using Apple's web namespace tags in the markup with HTML5 tricks like offline storage. There's also other libraries like jQTouch, so check out what else is around too.
The "write once, run anywhere" idea may just be coming into reality .
Sunday, May 17, 2009
Metadata Cloud Gaming
So, what the hell am I talking about? Let me give you a concrete example, and a personal wish list of mine for many a long year.
When I was growing up, one of the most capturing games was Championship Manager. This allowed the player to experience running a club, from transfers, tactics, training and more. However, the thing it always missed was being able to actually play the game. In terms of actually playing the game, when I was younger, the FIFA/EA Sports combo was hard to beat, although now I prefer the Konami Pro Evolution Soccer range.
I've always though that it'd be amazing to merge these two together, and I think now with Web2.0/Mashups/Cloud Computing and Web APIs this really should be possible. I could easily see a situation where as a gamer I could mash two of these together. For the tactical, and running of the game, I could use Championship Manager, and as a simulator for the actually games, I could use my game of choice. In this instance it really comes down to player stats. If I could load the game from Championship Manager, with the teams, players and 'stats' to boot, I could play out the game in my simulator and then upload the result along with accommodating stats into the cloud where my 'managerial' game could pick them up from, and vice versa.
Really, this is ripe for the picking.
When I was growing up, one of the most capturing games was Championship Manager. This allowed the player to experience running a club, from transfers, tactics, training and more. However, the thing it always missed was being able to actually play the game. In terms of actually playing the game, when I was younger, the FIFA/EA Sports combo was hard to beat, although now I prefer the Konami Pro Evolution Soccer range.
I've always though that it'd be amazing to merge these two together, and I think now with Web2.0/Mashups/Cloud Computing and Web APIs this really should be possible. I could easily see a situation where as a gamer I could mash two of these together. For the tactical, and running of the game, I could use Championship Manager, and as a simulator for the actually games, I could use my game of choice. In this instance it really comes down to player stats. If I could load the game from Championship Manager, with the teams, players and 'stats' to boot, I could play out the game in my simulator and then upload the result along with accommodating stats into the cloud where my 'managerial' game could pick them up from, and vice versa.
Really, this is ripe for the picking.
Sunday, April 26, 2009
The Great Twitter Camera Migration
I've just started a fun little side project called the 'Great Twitter Camera Migration', check it out: http://cameramigration.blogspot.com
Subscribe to:
Posts (Atom)