Usability and testing (A/B and MV)

Hi James,

Usability testing, analytics and ABT/MVT are all powerful tools for gauging the effectiveness of a website.

Usability testing I find is most useful during the design stage of a project. At this stage you can get lots of useful insight into the sorts of things your users may be looking for from your site, and it really helps with things like natural language (how users describe navigation elements, etc), usability allows you to do much more than test a specific design - use it for things such as card sorting, navigational analysis, etc. This solves a lot of issues very early in the design process, and prevents rushing off down dead-ends.

Usability is also very useful for testing designs just before you go live with them, once you've got things like functional prototypes or similar.

Once the site is live, your analytics gives you a wealth of real-life information about how your site is actually being used. Be aware that analytics can give you mountains of very detailed data, so you need to set some clear metrics about what you THINK is important.

This 'THINKING' about what is important is, obviously, important. You need to form some ideas about what you think your site's problems are. Analytics will be able to tell you if these things are causing issues to users.

Once you've identified the areas you think may be causing problems, then you can use ABT/MVT to define some experiments to try and fix or improve them. ABT/MVT lets you test changes to the site design or messaging on a case by case basis.

Remember that a complete testing programme means that you need to park what you 'know' and park your 'opinions' - let your decisions be made based on the data and the outcomes of experiment.

Obviously your experience will help you focus on the areas that are likely to yield results, but it's important to use a testing programme to deliver incremental improvements and to allocate resources to producing multiple variations over time.

I find that usability testing is usually done earlier in the process, as analytcis and ABT/MVT rely on actual live traffic. That said, usability testing on your live site will also yield results.

And with regard to conflicts (ie, in usability 6 people hate a particular feature, but on the live site MVT says that feature delivers a conversion uplift) My advice would be to err on the site of the MVT results - they're based on larger pools of users, and real-life results. The effect of the lab can skew your users responses, and the users you test with may not be 100% representative of your actual audience, no matter how carefully you vet them.

All of these approaches are necessary, though, for any major website development. And with testing tools and analytics now very affordable, it's possible to continually improve your website's performance.

Good luck with it,

Stephen Foxworthy

On 12:08:32 26 August 2008 Jamesonline wrote:

Hi - I'm curious as to how usability and testing work together. Essentially you can test one issue using two approaches - one humanistic and the other automated but I don't really know how to use them together.

Should one take precedance over the other - what happens if you get conflicting results. Or should you choose just one approach, or use them at different stages of development?

Would be interested in your thoughts and experiences.

James

Hi James - Good question as both have their place in a site development.  We have used both with clients, and usability testing is a core service, but where possible we like to work with the A/B testing as well.  I say where possible because it depends on a few things such as

• the client's eagerness for quanitative data for basing decisions - some companies are much more numbers driven
• the time to set up the options for the A/B testing
• budget of course
• Ability to capture and interpret the data, and whether they have the in-house skills to do so

There are good tools to help run the live testing, such as Google optimiser (free) or ones like Maximiser which we recommend.

The main thing to be aware of is that any A/B or MVT testing will tell you what is happening and which page options are working best eg for converting traffic, but it will never tell you WHY the ones are better (or if there are even better ones than the one that tests show is best).  One of the great advantage of simple usability testing is you can discuss with the user what they are doing and explore some of journeys taken or not taken in a way that gives you a much richer picture of the user experience and what could be done to make the content, design , navigation etc better.  Then there are additional insights if eye tracking is applied, great for cross sales placements, calls to action options etc.  Any automated testing would never give that. 

Hope that helps and good luck with the project

Chris Rourke

www.uservision.co.uk