-
There’s an intrinsic tension in design:
-
On one hand, you’ll want to be able to get feedback from people as soon as you possibly can.
-
On the other, in order to be able to get that feedback you need to make something that works.
-
You’ve learned how to rapidly prototype things with paper prototyping.
-
And today what we’re going to talk about is how you can test rapid prototypes with users.
-
So, let’s imagine for a moment —
-
What if it was possible to create an interactive application without writing much or any code
-
and get feedback on that interactive application quickly?
-
I think that’s a pretty powerful dream, and, like Dorothy learned in « The Wizard of Oz »,
-
dreams can be really powerful, and, sometimes, they can even come true.
-
And there’s something else that Dorothy learned that we’re going to use in the lecture:
-
If you’ve seen the movie « The Wizard of Oz », you’ll know that
-
when Dorothy and her compatriots arrived in the Emerald City,
-
they saw a big giant wizard that was really scary and freaked them out.
-
But eventually they learned that that wizard was just a little man behind a curtain,
-
and the ability of that little man behind the curtain to put on a big realistic show
-
that, to the visitors to the Emerald City, seemed completely real and immersive,
-
is exactly what we’re going to accomplish with this video’s prototyping technique.
-
So, the idea behind Wizard-of-Oz Prototyping
-
is that we’re going to simulate interactive behaviour and machine functionality
-
by having a human operator manipulate the levers behind the scenes.
-
If you want to get a sense for just how this realistic this can be,
-
watch our friend from « Seinfeld » call a “Movie Phone application”
-
to try and find out when and where movies are playing.
-
(Hello, and welcome to Movie Phone.
-
If you know the name of the movie you’d like to see, press 1.
-
[beep] C’mon, c’mon…
-
Using the touch-tone keypad, please enter the first three letters of the movie title, now!
-
[beep, beep]
-
You’ve selected « Agent Zero ». If that’s correct, press 1.
-
What?!
-
Ugh…)
-
So George thinks that he’s calling this automated Movie Phone application,
-
but it turns out that, behind the scenes, it’s just Kramer who’s playing a wizard
-
and behaving as if he were the machine.
-
To be honest, he’s not very good — I think that you’d all be able to do better!
-
Wizard of Oz techniques got their start with speech user interfaces.
-
The term “Wizard of Oz” was coined by Jeff Kelly in his PhD dissertation around 1980.
-
And Jeff’s setup at John Hopkins
-
actually bore more of a resemblance to the Wizard of Oz than you might first imagine;
-
so, in addition to some one-way mirrors and things like that, there literally was a curtain
-
separating Jeff, as the wizard, from the participants that were using the speech user interfaces.
-
And these Wizard-of-Oz prototypes were really important for understanding
-
what would be an effective speech user interface,
-
because, at the time, the recognition algorithms weren’t very good,
-
but we knew they were going to get better.
-
And Jeff asked the very important questions of:
-
“If we get the recognition to the point where it was effective, what will the user interaction be?
-
And what kind of strategies might be [the] most effective?”
-
So the Wizard of Oz enabled Jeff to time-travel
-
into a future where speech recognition technology worked better,
-
to try and understand what the user experience issues of that domain would be.
-
In general, a Wizard-of-Oz prototype comprises a couple of pieces:
-
First, you’ve got some kind of user interface that you’re delivering to the user
-
which feels-ish like a user interface — even if it’s sketches or custom-spoken words.
-
And, at the same time, there’s not any or much code that goes behind it —
-
the code and interactivity [that] is being created is a mirage from a human.
-
Sometimes there’s a user interface that the wizard has,
-
which they cobbled together to make life better for them —
-
and in fact back in 2000 colleagues at Berkeley and I built a system called Suede,
-
which helped automate Wizard-of-Oz testing and speech user interfaces.
-
Wizard-of-Oz prototypes make sense when it’s faster and cheapter and easier than building the real thing,
-
and that’s going to be especially true any time you have recognition-based interfaces,
-
or personalization that’s custom to a user.
-
This tradeoff of whether it’s cheaper and faster and easier is also going to depend on your abilities:
-
There will be some things that you know how to do quickly and easily and then you might just implement;
-
and there’s other things that may be more difficult or more time-consuming for you,
-
and those you’ll choose to Wizard-of-Oz.
-
Wizard-of-Oz prototypes, like any other kind of prototype, can be higher or lower fidelity.
-
And there’s some important tradeoffs here —
-
The higher fidelity the prototype, the more it feels like the real thing to users —
-
In fact, sometimes you can build an interface that people don’t even know is being powered by a wizard,
-
as in the famous story of the mechanical Turk, who is inside a chess machine.
-
However, making a high-fidelity user interface is going to take a lot of time,
-
and users may be more reluctant to critique a user interface that’s really high-fidelity,
-
and so, early on, I recommend quick-and-dirty paper prototypes being the most powerful thing
-
to use for Wizard-of-Oz tests.
-
Because users will feel like they can give you any kind of feedback;
-
it’s clear you haven’t put a lot of time into it yet,
-
and it’s fast for you to make, and so you actually haven’t a lot time into it yet.
-
A few years ago, my friend Sep Kamvar created a company called Aardvark,
-
which is a social search engine.
-
It was really good at helping you use your social network to find answers from other people.
-
So the way it works is you ask a question via instant message, via Google chat;
-
and it’s going to be automatically routed to a friend of yours, or a friend of a friend,
-
who has the expertise to answer that question.
-
One of the hard parts about building a service like this is: How do you do the routing?
-
Eventually they figured out an algorithm to do this; but to bootstrap that —
-
both to figure out what algorithm to build,
-
and to get a user experience going before they had built anything —
-
they had a Wizard-of-Oz prototype.
-
So the way that worked was that Aardvark employees would get the question,
-
and they would manually select the person that was going to get routed to.
-
Damon Horowitz, the co-founder of Aardvark, put it this way in a conference:
-
he said that, “If people like it in this super-crappy form, then it’s worth building,
-
because they’ll like it even more when we build the real thing!”
-
So, if you want to be like Kramer or Aardvark, how can you make a wizard-powered prototype?
-
Here’s a sketch of what you’ll need to do in five easy steps:
-
First, figure out what scenarios you’re going to support.
-
It’s a lot of easier to build a Wizard-of-Oz prototype for a limited set of functionality.
-
I like Wizard-of=Oz prototypes
-
because it forces you to figure out what should happen in response to human behaviour.
-
Then, put together some kind of user interface skeleton —
-
the things that the end users are going to see.
-
Third, if you need to, develop the “hooks” for the wizard [to] input if you’re going to have a remote wizard
-
or if they’re going to control some software functionality from behind the scenes.
-
Fourth, and as part of this,
-
you’ll need to figure out exactly what kind of input the wizard is allowed to offer:
-
Are they going to be selecting from a menu of options? offering free response? speaking text?
-
Will you give them a decision tree that controls their behaviour?
-
or you’re going let them be more free-form?
-
If you have a paper prototype, their role is going to pretty manual —
-
adding widgets and moving sliders, and making the interface go.
-
When you’re doing this, it’s important to remember that,
-
ultimately, the functionality that you’re wizarding now is ultimately going to be replaced with a computer,
-
and so at some point down the line, you will have to build some software to make it go.
-
And that’s important, because it can be easy to fake stuff that’s not possible to ever happen.
-
And, last, as with any prototype that you’re going to test with people,
-
practice it first with a friend or a colleague to get the hang of both doing the wizard —
-
it’s a very different kind of interaction than you may be used to —
-
and also, figuring out what scenarios are most effective, what instructions you’ll need to give people.
-
Get out the easy bugs in the user interface before you bring in real users.
-
And now you’re ready to run the user interface: Like I said, you’ll want to practice with a friend at first.
-
Once you’ve got the really obvious kinks out, and when you’ve got your patter down and the tasks ready,
-
then it’s time to recruit some people to come in and try out your prototype.
-
You can even go to places like train stations, or airports, or city street corners, or coffee shops
-
as a way of going to the people who might be using your system.
-
In a Wizard-of-Oz prototype, there’s two roles:
-
you’ve got a facilitator who’s going to talk to the user,
-
and you’ve got a wizard who’s going to manipulate the prototype.
-
If you have the luxury of a team, it’s helpful if there are two different people,
-
in part because your mind is going to be full of all the things that you’ll have to do,
-
and if you separate these roles, each person can concentrate more, do a better job,
-
and you have two sets of eyes looking at what the person’s doing, so you can learn more.
-
But can do this yourself if you need to — these can be one person doing two roles.
-
If you’re trying to convince somebody that this is actually is a real system,
-
then you’ll want your wizard to be hidden or remote somehow, like the Aardvark example.
-
And think about how you’d like to get feedback from users —
-
this is something that we’ll touch on several times more throughout this course.
-
For now, think about whether you’d like to get think-aloud feedback —
-
which is, when the person’s using the prototype,
-
have them speak aloud what it is that’s going through their mind, what decisions they’re making and why,
-
what they’re confused by when they run up against things that they can’t figure out what to do.
-
This think-aloud protocol is great for getting stuff that otherwise you might not know or learn.
-
However, as you can imagine, thinking aloud can change how people interact with the prototype.
-
So, if think-aloud is too distracting, you can ask people at the end of the study
-
what was going through their mind and what they were thinking about —
-
that’s called a retrospective approach.
-
If you like, you can even show them the video of them using it.
-
You can then say things like, “When you got stuck here, what was the issue?”
-
Third, if you have specific things that you’d like to make sure that your interface does well,
-
you can have categories of problems or heuristics that you asked your users to pay attention to.
-
And heuristic evaluation is something that we’re going to talk more about later in this course also.
-
Once you’re all done, make sure to thank people for their time.
-
You may want to give them a gift certificate, or ice-cream, or some other kind of thank-you.
-
And, if you haven’t yet told them that there’s actually a wizard behind the scenes operating the user interface,
-
the end of the experiment is probably a really good time to that —
-
You’ll want to make sure to be honest with people.
-
You can use wizards throughout the entire development of your software,
-
not just at the very beginning.
-
And so, if you have the fully-functional software here, and over time you move towards there,
-
you can use wizards to fill in the gap of whatever is not built at that time.
-
So, the wizard may do all of the interactions at the very beginning,
-
and then, as you get closer and closer to a final project,
-
you can have it just fill in only the parts that aren’t done yet.
-
As you can see, there’s a lot to like about Wizard-of-Oz prototypes:
-
They can be really fast to make, which makes them cheaper,
-
and it also means you get to turn the crank on iteration more.
-
Because they are quick, you can rapidly create multiple alternatives.
-
And you decide that you want the “implementation” to work differently,
-
you just give your wizard different instructions; there’s no code to rewrite.
-
Because it is in a sense interactive,
-
it is more real and you get better feedback than just from paper prototyping,
-
although using paper prototypes as the substrate for a wizard
-
is an excellent strategy early on in the design process.
-
It’s a great way to get feedback about bugs and problems with your user interface design.
-
Every single user interface that I’ve ever built, or that I’ve seen built,
-
or that students of mine have ever built, has had bugs early on.
-
It’s too hard to get right the first time.
-
You’re going to iterate and fail fast, so that you can succeed sooner.
-
And a Wizard of Oz is a great tool to be able to do that.
-
Because you’re getting feedback from users early,
-
it’s a great way to place people at the centre of the development process.
-
It’s also really wonderful for forward-looking applications
-
as we start to think about how sensors and cameras and other futuristic technologies
-
might enable other new kinds of user interfaces.
-
You can imagine these now by building Wizard-of-Oz prototypes.
-
And lastly, strange as it may sound, you actually learn a lot
-
about what your application logic of your interface ought to be, and what makes sense
-
by playing the wizard and embodying that role in the system.
-
But there are some disadvantages too:
-
It can be easy to gloss over errors in technology that will ultimately come back to bite you —
-
like speech recognition systems for example.
-
If, at the end of the day, you’re going to have bugs in your speech system,
-
you’ll want to make sure that your Wizard-of-Oz prototype encodes
-
what happens when the speech reco[gnition]s screws up.
-
Don’t assume it’s going to work perfectly all the time.
-
’Cause if you’re not careful, you can end up simulating a technology that doesn’t exist and may never.
-
People, for better or for worse, are inconsistent,
-
and, to get them to play the wizard role, can require training, so that’s a ramp-up time.
-
And because you have to have a physical human there to run it,
-
your runtime efficiency is slower,
-
and it can be more exhausting for the people that are playing the wizard role.
-
Wizards are great for some kind[s] of functionality, like Kramer did with the speech user interface.
-
But other stuff, it’s harder to figure out what you can do with a wizard interface,
-
and it may not be quite as effective — it may be klunkier.
-
Or it might be simply be inappropriate —
-
there’re some situations where having a wizard around may not be ok.
-
All in all, I think wizards are an incredibly powerful technique,
-
and a really fun one to stretch your creativity.
-
What kinds of new user interfaces can you imagine by creating a Wizard-of-Oz prototype?
-
I’d like to thank Stephen Dow in creating this lecture —
-
I drew on a lot of the materials that’s he’s used in writing and speaking about Wizard-of-Oz prototypes.
-
And if you’re interested in learning more about wizards, you can see these resources here.
