Some Guiding Principles for Supporting Production Apps

I just finished a two week rotation supporting our production web services and wanted to jot down some thoughts. This coincided with being sent a link to a Scott Hanselman blog post about guiding principles of software development which really enjoyed reading. I thought I'd extend a few of those now I've come out of my support rotation an enlightened developer (I hope).

Although Scott's blog is for Windows/.NET development, there's lots of goodies in there which can be applied to any language or ported to any language. I'd like to narrow in on the support based principles: tracing (logging) and error handling.

Scott mentions that for critical/error/warn logging, your audience is not a developer. Well in this case, I am, however, I may not know the application, so I need as much information as I can get. This leads me onto my first extension; transaction IDs.

Now, there's two kinds of transaction IDs really, especially when dealing with a service based application. One is every invocation of your service should be given a unique ID and that should be passed around the code (and logged against) until the path returns to the client. Depending on your application, this could get really noisy, but a multi-threaded application could benefit from such information. Second is a resource/task based ID, if a service does something (let's call it A), but the execution path returns to the client before A is complete, the ID for A should be returned to the client and this ID should live through the execution of A and stored or achieved accordingly. In terms of logging, when ever you come across this ID, use it. This generally follows for all IDs, if you have an ID, use it when logging.

With error handling, it's accepted that catching a generic exception is bad, although permissible if at the boundary of your application. If you must catch 'Exception' or 'RuntimeException', log it and use at least a error condition. It's a generic exception for a reason and that reason is you don't know what happened.

AOP is a great technique for logging, don't be frightened (and this goes with all logging) to use it everywhere.

Moving onto a topic glanced upon, but not thoroughly explored; configuration. If building a server product, config is your friend. When things go bad and you need to re-wire your application, not having to do a code change is a massive benefit. If the config is dynamically read, that's even better as it won't require a restart to the server.

Lastly, think about your archive mechanism and think about what data you'll need to preserve and for how long. Ideally, from that data, execution paths should be interpretable, at least at a high level.

Oh, and please be kind to your support team, make it easy for them to gather information about your application. If an application is one of many, consider implementing a common health/management (and provisioning if you can) interface.

I think putting together a list of principles like this is a great idea, each engineering group should think about the ways they work and how to drive commonality across products. Teams shouldn't be scared of thinking about support throughout the development process either, although they've been hearing the same thing about security for years ;)

Scott Hansleman podcast with the Poppendieck's

A great podcast which deserves it's own post to comment on it. Mary Poppendieck really came out with some cracking soundbites including:

* product teams should be driven by profit and loss, not by on-time, on-budget, in-scope, customer satisfaction and quality.
* engineering and business - it's not them and us, it should be all of us
* no excuse for IT people to be separate (organisational) from the business if IT
* If IT it routine (email etc), outsourcing is fine, how innovate can your email be? If building core competencies, outsourcing is generally a bad idea
* leverage workers intelligence to deliver, management should be leaders, not micro managers

loads of good stuff, check it out!

A Day at OpenTech

I spent my Saturday at the OpenTech conference in London and it was a day well spent. I do have to learn to go and listen to new things though, although talks from Tom Morris (who I later met for the first time), Simon Willison and Gavin Bell were interesting, those were subjects I already knew about or were aware about to a fair degree. What did strike me however were some of the last sessions by Tom Loosemore and Chris Jackson.

Between them they told stories which included building a peer-to-peer TV network for persisted content, and a project to become 'the DNS for media'. I've always had an interest in home entertainment, and these talks made my eyes light up. I'm really pleased to see that Tom has posted his presentation.

Tom's 'impossibox' was capable of storing 1TBs worth of media data, roughly a months worth. If that were to be networked in a P2P fashion, you could have months of recorded TV, redundantly stored in every house on your street. I think this plays into the networks hands here as well. This isn't real time data and can be sync'd, or you can download a program or series at the networks leisure without breaking net neutrality. Throw in, if there were 100 such networked boxes with a mile's radius, chances are you could take care of the P2P sharing at the local exchange so you wouldn't even need to push large amounts of data all over the place.

Although this future is a 'slightly bonkers' version, it sparks the imagination. Just think of the social networking benefits of being able to capture this data, see friends recommendations, read comments, video mashups, trailers, directors (or other users) commentaries. So much potential.

Great talk.

Data, Data all around

I attended 2gether08 today for the afternoon and walked into a very interesting discussion on making data public. Now following the Free Our Data campaign from the Guardian I've had a keen interest on the subject. Many points were discussed from innovating on this newly available data, to cost savings for doing so through to privacy issues surrounding private data (such as medical records) and data which people seem to give away freely through social sites such as Facebook and finally data which people give away to serve a better experience (think supermarket club card, or Google search).

One of the interesting points raised was about who the public tend to put their trust in, and maybe not so surprisingly, it's not the government. There is a trend that people are giving or putting more data into corporate entities, be they your favoured supermarket, accountant or bank. An interesting point was raised by Paul Downey that although he does online banking, he doesn't have the same trust in Yahoo! to mashup his banking data through Pipes or some other such technology, this is mainly due to banks be regulated authorities.

The conversation moved on and touched upon escrow like accounts for personal data and this got me thinking about identity. As we move towards tools like OpenID and OAuth, the thought occurred to me, would you log into your bank with your OpenID? If your OpenID provider isn't regulated, or even in the same country as you, would you trust it enough with potential access to your online bank? Let me turn that question around a bit now, if you're OpenID provider was a regulated authority, perhaps the bank in question or even some government department - would that change your answer?

The final question left in my mind was, that with a few people thinking about decentralised escrow like data accounts, could you trust people to look after their own data? I know that when I carry my passport with me, I get slightly paranoid. I also know that after losing 2 or 3 hard discs over the last 10 years, I'm not sure I'd want that responsibility. What I do know is that I hate repeating myself, and if government bodies can share data easier between themselves, and if there's a way I can authorise companies to share my data in a way that makes my life easier to lead, I might just buy into that.

I suppose with the level of data that may become available, the responsibility may shift somewhat with who owns, maintains and secures that data.