0:00:01.362,0:00:04.346
There’s an intrinsic tension in design:

0:00:04.346,0:00:09.557
On one hand, you’ll want to be able to get feedback from people as soon as you possibly can.

0:00:09.557,0:00:15.620
On the other, in order to be able to get that feedback you need to make something that works.

0:00:15.620,0:00:19.371
You’ve learned how to rapidly prototype things with paper prototyping.

0:00:19.371,0:00:23.856
And today what we’re going to talk about is how you can test rapid prototypes with users.

0:00:24.856,0:00:26.667
So, let’s imagine for a moment —

0:00:26.667,0:00:31.747
What if it was possible to create an interactive application without writing much or any code

0:00:31.747,0:00:35.943
and get feedback on that interactive application quickly?

0:00:35.943,0:00:40.757
I think that’s a pretty powerful dream, and, like Dorothy learned in « The Wizard of Oz »,

0:00:40.757,0:00:45.704
dreams can be really powerful, and, sometimes, they can even come true.

0:00:45.704,0:00:49.269
And there’s something else that Dorothy learned that we’re going to use in the lecture:

0:00:49.269,0:00:51.464
If you’ve seen the movie « The Wizard of Oz », you’ll know that

0:00:51.464,0:00:54.700
when Dorothy and her compatriots arrived in the Emerald City,

0:00:54.700,0:00:58.741
they saw a big giant wizard that was really scary and freaked them out.

0:00:59.633,0:01:03.492
But eventually they learned that that wizard was just a little man behind a curtain,

0:01:04.077,0:01:08.440
and the ability of that little man behind the curtain to put on a big realistic show

0:01:08.440,0:01:13.356
that, to the visitors to the Emerald City, seemed completely real and immersive,

0:01:13.356,0:01:17.527
is exactly what we’re going to accomplish with this video’s prototyping technique.

0:01:18.819,0:01:20.579
So, the idea behind Wizard-of-Oz Prototyping

0:01:20.579,0:01:24.373
is that we’re going to simulate interactive behaviour and machine functionality

0:01:24.373,0:01:28.909
by having a human operator manipulate the levers behind the scenes.

0:01:29.571,0:01:32.596
If you want to get a sense for just how this realistic this can be,

0:01:32.596,0:01:36.381
watch our friend from « Seinfeld » call a “Movie Phone application”

0:01:36.396,0:01:39.978
to try and find out when and where movies are playing.

0:01:40.747,0:01:44.537
(Hello, and welcome to Movie Phone.

0:01:44.537,0:01:48.655
If you know the name of the movie you’d like to see, press 1.

0:01:48.655,0:01:50.487
[beep] C’mon, c’mon…

0:01:50.487,0:01:56.156
Using the touch-tone keypad, please enter the first three letters of the movie title, now!

0:01:56.156,0:01:58.179
[beep, beep]

0:01:58.179,0:02:05.796
You’ve selected « Agent Zero ». If that’s correct, press 1.

0:02:05.796,0:02:07.753
What?!

0:02:07.753,0:02:09.843
Ugh…)

0:02:09.843,0:02:13.468
So George thinks that he’s calling this automated Movie Phone application,

0:02:13.468,0:02:17.310
but it turns out that, behind the scenes, it’s just Kramer who’s playing a wizard

0:02:17.310,0:02:20.070
and behaving as if he were the machine.

0:02:20.070,0:02:23.876
To be honest, he’s not very good — I think that you’d all be able to do better!

0:02:24.630,0:02:28.049
Wizard of Oz techniques got their start with speech user interfaces.

0:02:28.049,0:02:34.353
The term “Wizard of Oz” was coined by Jeff Kelly in his PhD dissertation around 1980.

0:02:34.353,0:02:36.447
And Jeff’s setup at John Hopkins

0:02:36.447,0:02:40.101
actually bore more of a resemblance to the Wizard of Oz than you might first imagine;

0:02:40.101,0:02:45.778
so, in addition to some one-way mirrors and things like that, there literally was a curtain

0:02:45.778,0:02:50.871
separating Jeff, as the wizard, from the participants that were using the speech user interfaces.

0:02:50.871,0:02:55.070
And these Wizard-of-Oz prototypes were really important for understanding

0:02:55.070,0:02:57.471
what would be an effective speech user interface,

0:02:57.471,0:03:01.365
because, at the time, the recognition algorithms weren’t very good,

0:03:01.365,0:03:03.253
but we knew they were going to get better.

0:03:03.253,0:03:05.739
And Jeff asked the very important questions of:

0:03:05.739,0:03:10.763
“If we get the recognition to the point where it was effective, what will the user interaction be?

0:03:10.763,0:03:13.612
And what kind of strategies might be [the] most effective?”

0:03:13.627,0:03:17.086
So the Wizard of Oz enabled Jeff to time-travel

0:03:17.086,0:03:20.625
into a future where speech recognition technology worked better,

0:03:20.625,0:03:24.329
to try and understand what the user experience issues of that domain would be.

0:03:25.862,0:03:29.380
In general, a Wizard-of-Oz prototype comprises a couple of pieces:

0:03:29.380,0:03:33.362
First, you’ve got some kind of user interface that you’re delivering to the user

0:03:33.362,0:03:41.790
which feels-ish like a user interface — even if it’s sketches or custom-spoken words.

0:03:41.790,0:03:47.036
And, at the same time, there’s not any or much code that goes behind it —

0:03:47.036,0:03:53.856
the code and interactivity [that] is being created is a mirage from a human.

0:03:53.856,0:03:57.561
Sometimes there’s a user interface that the wizard has,

0:03:57.561,0:04:00.656
which they cobbled together to make life better for them —

0:04:00.656,0:04:05.480
and in fact back in 2000 colleagues at Berkeley and I built a system called Suede,

0:04:05.480,0:04:09.330
which helped automate Wizard-of-Oz testing and speech user interfaces.

0:04:09.330,0:04:15.513
Wizard-of-Oz prototypes make sense when it’s faster and cheapter and easier than building the real thing,

0:04:15.513,0:04:20.465
and that’s going to be especially true any time you have recognition-based interfaces,

0:04:20.465,0:04:23.767
or personalization that’s custom to a user.

0:04:24.490,0:04:28.874
This tradeoff of whether it’s cheaper and faster and easier is also going to depend on your abilities:

0:04:28.874,0:04:33.293
There will be some things that you know how to do quickly and easily and then you might just implement;

0:04:33.293,0:04:37.601
and there’s other things that may be more difficult or more time-consuming for you,

0:04:37.601,0:04:40.044
and those you’ll choose to Wizard-of-Oz.

0:04:40.644,0:04:46.192
Wizard-of-Oz prototypes, like any other kind of prototype, can be higher or lower fidelity.

0:04:46.192,0:04:48.729
And there’s some important tradeoffs here —

0:04:48.729,0:04:53.081
The higher fidelity the prototype, the more it feels like the real thing to users —

0:04:53.081,0:04:58.982
In fact, sometimes you can build an interface that people don’t even know is being powered by a wizard,

0:04:58.982,0:05:04.036
as in the famous story of the mechanical Turk, who is inside a chess machine.

0:05:04.467,0:05:09.209
However, making a high-fidelity user interface is going to take a lot of time,

0:05:09.209,0:05:15.022
and users may be more reluctant to critique a user interface that’s really high-fidelity,

0:05:15.022,0:05:20.600
and so, early on, I recommend quick-and-dirty paper prototypes being the most powerful thing

0:05:20.600,0:05:22.520
to use for Wizard-of-Oz tests.

0:05:22.520,0:05:25.600
Because users will feel like they can give you any kind of feedback;

0:05:25.600,0:05:28.272
it’s clear you haven’t put a lot of time into it yet,

0:05:28.272,0:05:33.110
and it’s fast for you to make, and so you actually haven’t a lot time into it yet.

0:05:33.772,0:05:37.213
A few years ago, my friend Sep Kamvar created a company called Aardvark,

0:05:37.213,0:05:39.926
which is a social search engine.

0:05:39.926,0:05:45.494
It was really good at helping you use your social network to find answers from other people.

0:05:45.971,0:05:50.576
So the way it works is you ask a question via instant message, via Google chat;

0:05:50.576,0:05:55.621
and it’s going to be automatically routed to a friend of yours, or a friend of a friend,

0:05:55.621,0:05:58.601
who has the expertise to answer that question.

0:05:58.601,0:06:04.336
One of the hard parts about building a service like this is: How do you do the routing?

0:06:04.336,0:06:08.017
Eventually they figured out an algorithm to do this; but to bootstrap that —

0:06:08.017,0:06:10.384
both to figure out what algorithm to build,

0:06:10.384,0:06:15.228
and to get a user experience going before they had built anything —

0:06:15.228,0:06:17.716
they had a Wizard-of-Oz prototype.

0:06:17.716,0:06:21.476
So the way that worked was that Aardvark employees would get the question,

0:06:21.476,0:06:25.332
and they would manually select the person that was going to get routed to.

0:06:25.332,0:06:29.200
Damon Horowitz, the co-founder of Aardvark, put it this way in a conference:

0:06:29.200,0:06:33.923
he said that, “If people like it in this super-crappy form, then it’s worth building,

0:06:33.923,0:06:38.261
because they’ll like it even more when we build the real thing!”

0:06:38.277,0:06:43.708
So, if you want to be like Kramer or Aardvark, how can you make a wizard-powered prototype?

0:06:43.708,0:06:46.738
Here’s a sketch of what you’ll need to do in five easy steps:

0:06:47.246,0:06:49.582
First, figure out what scenarios you’re going to support.

0:06:49.582,0:06:54.363
It’s a lot of easier to build a Wizard-of-Oz prototype for a limited set of functionality.

0:06:54.840,0:06:56.372
I like Wizard-of=Oz prototypes

0:06:56.372,0:06:59.892
because it forces you to figure out what should happen in response to human behaviour.

0:07:00.847,0:07:03.717
Then, put together some kind of user interface skeleton —

0:07:03.717,0:07:06.239
the things that the end users are going to see.

0:07:06.593,0:07:11.266
Third, if you need to, develop the “hooks” for the wizard [to] input if you’re going to have a remote wizard

0:07:11.266,0:07:15.277
or if they’re going to control some software functionality from behind the scenes.

0:07:16.046,0:07:17.485
Fourth, and as part of this,

0:07:17.485,0:07:21.491
you’ll need to figure out exactly what kind of input the wizard is allowed to offer:

0:07:21.491,0:07:26.847
Are they going to be selecting from a menu of options? offering free response? speaking text?

0:07:26.847,0:07:31.228
Will you give them a decision tree that controls their behaviour?

0:07:31.228,0:07:33.617
or you’re going let them be more free-form?

0:07:33.617,0:07:36.917
If you have a paper prototype, their role is going to pretty manual —

0:07:36.917,0:07:40.886
adding widgets and moving sliders, and making the interface go.

0:07:41.733,0:07:43.741
When you’re doing this, it’s important to remember that,

0:07:43.741,0:07:49.577
ultimately, the functionality that you’re wizarding now is ultimately going to be replaced with a computer,

0:07:49.577,0:07:53.818
and so at some point down the line, you will have to build some software to make it go.

0:07:53.818,0:07:58.776
And that’s important, because it can be easy to fake stuff that’s not possible to ever happen.

0:07:59.469,0:08:03.430
And, last, as with any prototype that you’re going to test with people,

0:08:03.430,0:08:07.170
practice it first with a friend or a colleague to get the hang of both doing the wizard —

0:08:07.170,0:08:10.095
it’s a very different kind of interaction than you may be used to —

0:08:10.095,0:08:15.442
and also, figuring out what scenarios are most effective, what instructions you’ll need to give people.

0:08:15.442,0:08:19.823
Get out the easy bugs in the user interface before you bring in real users.

0:08:20.300,0:08:25.641
And now you’re ready to run the user interface: Like I said, you’ll want to practice with a friend at first.

0:08:27.380,0:08:32.118
Once you’ve got the really obvious kinks out, and when you’ve got your patter down and the tasks ready,

0:08:32.118,0:08:35.180
then it’s time to recruit some people to come in and try out your prototype.

0:08:35.180,0:08:41.568
You can even go to places like train stations, or airports, or city street corners, or coffee shops

0:08:41.568,0:08:45.814
as a way of going to the people who might be using your system.

0:08:47.230,0:08:50.428
In a Wizard-of-Oz prototype, there’s two roles:

0:08:50.428,0:08:53.270
you’ve got a facilitator who’s going to talk to the user,

0:08:53.270,0:08:55.871
and you’ve got a wizard who’s going to manipulate the prototype.

0:08:55.871,0:09:00.091
If you have the luxury of a team, it’s helpful if there are two different people,

0:09:00.091,0:09:03.745
in part because your mind is going to be full of all the things that you’ll have to do,

0:09:03.745,0:09:08.696
and if you separate these roles, each person can concentrate more, do a better job,

0:09:08.696,0:09:13.568
and you have two sets of eyes looking at what the person’s doing, so you can learn more.

0:09:14.584,0:09:18.920
But can do this yourself if you need to — these can be one person doing two roles.

0:09:19.597,0:09:22.375
If you’re trying to convince somebody that this is actually is a real system,

0:09:22.375,0:09:27.031
then you’ll want your wizard to be hidden or remote somehow, like the Aardvark example.

0:09:27.354,0:09:29.492
And think about how you’d like to get feedback from users —

0:09:29.492,0:09:33.477
this is something that we’ll touch on several times more throughout this course.

0:09:33.477,0:09:36.985
For now, think about whether you’d like to get think-aloud feedback —

0:09:36.985,0:09:39.913
which is, when the person’s using the prototype,

0:09:39.913,0:09:45.436
have them speak aloud what it is that’s going through their mind, what decisions they’re making and why,

0:09:45.436,0:09:51.013
what they’re confused by when they run up against things that they can’t figure out what to do.

0:09:51.721,0:09:58.111
This think-aloud protocol is great for getting stuff that otherwise you might not know or learn.

0:09:58.111,0:10:03.067
However, as you can imagine, thinking aloud can change how people interact with the prototype.

0:10:03.067,0:10:07.325
So, if think-aloud is too distracting, you can ask people at the end of the study

0:10:07.325,0:10:09.865
what was going through their mind and what they were thinking about —

0:10:09.865,0:10:11.821
that’s called a retrospective approach.

0:10:11.821,0:10:14.402
If you like, you can even show them the video of them using it.

0:10:14.402,0:10:18.716
You can then say things like, “When you got stuck here, what was the issue?”

0:10:19.609,0:10:24.051
Third, if you have specific things that you’d like to make sure that your interface does well,

0:10:24.051,0:10:30.155
you can have categories of problems or heuristics that you asked your users to pay attention to.

0:10:30.155,0:10:34.622
And heuristic evaluation is something that we’re going to talk more about later in this course also.

0:10:35.499,0:10:38.065
Once you’re all done, make sure to thank people for their time.

0:10:38.065,0:10:43.162
You may want to give them a gift certificate, or ice-cream, or some other kind of thank-you.

0:10:44.209,0:10:49.711
And, if you haven’t yet told them that there’s actually a wizard behind the scenes operating the user interface,

0:10:49.711,0:10:52.311
the end of the experiment is probably a really good time to that —

0:10:52.311,0:10:54.750
You’ll want to make sure to be honest with people.

0:10:54.750,0:10:57.974
You can use wizards throughout the entire development of your software,

0:10:57.974,0:10:59.942
not just at the very beginning.

0:10:59.942,0:11:06.066
And so, if you have the fully-functional software here, and over time you move towards there,

0:11:06.066,0:11:10.274
you can use wizards to fill in the gap of whatever is not built at that time.

0:11:10.274,0:11:13.523
So, the wizard may do all of the interactions at the very beginning,

0:11:13.523,0:11:16.201
and then, as you get closer and closer to a final project,

0:11:16.201,0:11:19.918
you can have it just fill in only the parts that aren’t done yet.

0:11:20.441,0:11:24.047
As you can see, there’s a lot to like about Wizard-of-Oz prototypes:

0:11:24.463,0:11:27.791
They can be really fast to make, which makes them cheaper,

0:11:27.791,0:11:31.121
and it also means you get to turn the crank on iteration more.

0:11:32.429,0:11:35.618
Because they are quick, you can rapidly create multiple alternatives.

0:11:35.618,0:11:39.078
And you decide that you want the “implementation” to work differently,

0:11:39.078,0:11:42.649
you just give your wizard different instructions; there’s no code to rewrite.

0:11:43.511,0:11:45.141
Because it is in a sense interactive,

0:11:45.141,0:11:49.002
it is more real and you get better feedback than just from paper prototyping,

0:11:49.002,0:11:52.135
although using paper prototypes as the substrate for a wizard

0:11:52.135,0:11:55.724
is an excellent strategy early on in the design process.

0:11:56.308,0:12:00.865
It’s a great way to get feedback about bugs and problems with your user interface design.

0:12:00.865,0:12:05.160
Every single user interface that I’ve ever built, or that I’ve seen built,

0:12:05.160,0:12:08.591
or that students of mine have ever built, has had bugs early on.

0:12:08.591,0:12:11.325
It’s too hard to get right the first time.

0:12:11.325,0:12:15.543
You’re going to iterate and fail fast, so that you can succeed sooner.

0:12:15.543,0:12:18.914
And a Wizard of Oz is a great tool to be able to do that.

0:12:20.053,0:12:22.544
Because you’re getting feedback from users early,

0:12:22.544,0:12:26.437
it’s a great way to place people at the centre of the development process.

0:12:27.006,0:12:29.911
It’s also really wonderful for forward-looking applications

0:12:29.911,0:12:34.608
as we start to think about how sensors and cameras and other futuristic technologies

0:12:34.608,0:12:37.214
might enable other new kinds of user interfaces.

0:12:37.214,0:12:42.230
You can imagine these now by building Wizard-of-Oz prototypes.

0:12:42.522,0:12:46.023
And lastly, strange as it may sound, you actually learn a lot

0:12:46.023,0:12:50.120
about what your application logic of your interface ought to be, and what makes sense

0:12:50.120,0:12:54.268
by playing the wizard and embodying that role in the system.

0:12:54.991,0:12:58.281
But there are some disadvantages too:

0:12:58.281,0:13:03.175
It can be easy to gloss over errors in technology that will ultimately come back to bite you —

0:13:03.175,0:13:06.641
like speech recognition systems for example.

0:13:06.656,0:13:11.635
If, at the end of the day, you’re going to have bugs in your speech system,

0:13:11.635,0:13:14.447
you’ll want to make sure that your Wizard-of-Oz prototype encodes

0:13:14.447,0:13:16.883
what happens when the speech reco[gnition]s screws up.

0:13:16.883,0:13:20.050
Don’t assume it’s going to work perfectly all the time.

0:13:20.050,0:13:24.716
’Cause if you’re not careful, you can end up simulating a technology that doesn’t exist and may never.

0:13:25.393,0:13:27.806
People, for better or for worse, are inconsistent,

0:13:27.806,0:13:32.888
and, to get them to play the wizard role, can require training, so that’s a ramp-up time.

0:13:32.888,0:13:35.997
And because you have to have a physical human there to run it,

0:13:35.997,0:13:38.193
your runtime efficiency is slower,

0:13:38.193,0:13:42.159
and it can be more exhausting for the people that are playing the wizard role.

0:13:42.159,0:13:46.981
Wizards are great for some kind[s] of functionality, like Kramer did with the speech user interface.

0:13:46.981,0:13:51.370
But other stuff, it’s harder to figure out what you can do with a wizard interface,

0:13:51.370,0:13:56.233
and it may not be quite as effective — it may be klunkier.

0:13:56.233,0:13:57.964
Or it might be simply be inappropriate —

0:13:57.964,0:14:05.845
there’re some situations where having a wizard around may not be ok.

0:14:05.845,0:14:09.097
All in all, I think wizards are an incredibly powerful technique,

0:14:09.097,0:14:11.523
and a really fun one to stretch your creativity.

0:14:11.523,0:14:17.075
What kinds of new user interfaces can you imagine by creating a Wizard-of-Oz prototype?

0:14:17.075,0:14:19.761
I’d like to thank Stephen Dow in creating this lecture —

0:14:19.761,0:14:24.461
I drew on a lot of the materials that’s he’s used in writing and speaking about Wizard-of-Oz prototypes.

0:14:24.461,0:14:30.918
And if you’re interested in learning more about wizards, you can see these resources here.