Last week we ran some research in the lab. We were testing comprehension of some new concepts in the communication space. Usually this would be straightforward – create screens/flows for key tasks and see if people understand what is going on and what to do next. The issue though is that sociability is full of subtleties. I may do one action for one friend, but something different for another. And when looking at a user interface, I might think one thing if I have one friend in mind, but something different if I have another friend in mind. For example, if I have a close friend in mind I may be concerned about privacy of our conversation, but maybe less so for a friend of a friend. So to try and uncover some of that subtlety, we did an exercise to make the tasks as “real” as possible. We created a fake “real-life” social network…
…and asked research participants to substitute real people in their life for these fake people. So they took the print-out of this diagram and scribbled names of real people in their life on top of the fake people. Then when they encountered a mocked up design with the fake names, they could substitute the real people. Likewise, when they were asked to do a task, they substituted the real people and we learned a lot more about the context of that task in their real life. For example with fake names we may get this feedback on a particular part of U.I.:
“…I’d type the message here and then click ‘send’…”
But with real people substituted in, we got this instead:
“…well, I could type in here and click ‘send’ but I wouldn’t do that because Joe never checks his email, so I’d phone him to make sure…”
We were worried that the cognitive load of substituting people would be too high, but it wasn’t – it worked out really well and we learned a lot about the broader context of what we were proposing. One thing we did have to do was constantly remind people to reference their sheet of real names. People slip into “hypothetical mode” very easily.
I think this tactic of “substitution” could work well in many lab research studies. Try it out!
I like how you call them Fake, when they’re real
It’s a cool idea. We had to test an e-mail client out a long time ago, and I ran into a simliar problem. All our “dummy data” was useless as it didn’t reflect what people really used their e-mail for. In the end we paid a little extra to each user and had them set up a forward for inbound emails to our new inbox, where we ran the test.
Much better than a simple “How would you search THIS mail box” script.
Forwarding mail to a test account – nice alternative if you’re at that stage of development. We were testing flat click-through images, and looking at multiple communication tools. Definitely something I’ll think about in the future though. You’d need to be careful how setting that up changes peoples behaviour, and whether that matters e.g. they never knew there was a “settings” page until you explained to them how to forward email. You may have primed their behaviour for the research.
Yep, I’m almost confident I did prime them somewhat – but it was the only way I could work out how to get users to perform realistic activities on a realistic inbox.
Our original plan was to simply take their e-mail details and set it up ourselves, but ethically that didn’t seem quite right
. It would be akin to saying “Enter your facebook details here” and then scraping the data with a script. Realistic? Yes. Ethically Sound? Hmm
Anyways, I look forward to guessing what the output of all of this is, in months to come.