A quick, and very un-scholarly review of the literature reveals a few papers you may be interested in. Contrary to my own assumptions, there is some encouraging (but not conclusive) evidence for RTA. It seems we still lack a robust body of knowledge on the subject.

One problem is the studies are all of varying quality and not directly comparable. However unlike my opinions, these are findings of real studies that have undergone peer review.

Academia is a bit of a dirty word in UX these days, but this is one of those times when what you need is good, robust evidence to help you make up your mind (Note I haven’t included any of Jared Spool’s arguments, because although I happen to instinctively agree with him, he hasn’t published his findings as far as I know).

For your interest, here are some of the more relevant studies I found (not intended to be exhaustive):
http://bit.ly/9AiBkn
http://bit.ly/bPmgYy
http://bit.ly/9Cf2ox
http://bit.ly/dp41mj

Modern eye tracking systems do not ‘de-calibrate’ if the user looks away, that’s a myth. We watch users constantly looking at the keyboard during research sessions and then looking back at the screen. And I’m not sure what these ‘artifacts’ would be?

I’m quickly coming to the conclusion that using the term ‘RTA’ to describe the retrospective session is misleading. Tradtitional RTA may rely on the user rationalising their thoughts – just as you could argue CTA does, but eye tracking is completely different because you are having a granular conversation about how a user viewed the page by playing back their eye fixations. The user isn’t rationalising – they are recalling their behaviour. And before somebody mentions the ‘missing the ketchup in the fridge’ argument – with good facilitation, a user in a retrospetive session would tell you that they didn’t see something, even if they were looking at it.

In terms of RTA, various critics say that rationalisation and confabulation is an issue. Various advocates deny this.

I agree with Guy. Eye tracking is very useful. And you give a good summary of the analysis problems that inexperienced or poorly trained people have. However, the slideshow does fall short in describing how to use the tool in the right way, for the right reasons and in the right context.

Improvements don’t just come about by using RTA. It is great but there are plenty of other things that can be done to really leverage the power of eye tracking.

A few of us are working on a part 2.

JB

I agree with your note on using the RTA for accurate task time measurement. And for the benefit of other readers who may not know, it may also be used as an alternate to the CTA when participants are not comfortable with thinking aloud, which is what I have used it for in the past for considerably *shorter* test sessions.

Guy, I want to let you know that both the CTA and RTA (what you refer to as retrospective protocol) are variations of think aloud. CTA/ Concurrent Think Aloud is what you are referring to as think aloud.

Both have a number of pros and cons which I went over with the Tobii and SMI guys including a few other folks at the India HCI/ IDID 2010 conference which I participated at as a tutor a few months ago.

Thanks for the slides.