1
00:00:01,362 --> 00:00:04,346
There’s an intrinsic tension in design:

2
00:00:04,346 --> 00:00:09,557
On one hand, you’ll want to be able to get feedback from people as soon as you possibly can.

3
00:00:09,557 --> 00:00:15,620
On the other, in order to be able to get that feedback you need to make something that works.

4
00:00:15,620 --> 00:00:19,371
You’ve learned how to rapidly prototype things with paper prototyping.

5
00:00:19,371 --> 00:00:23,856
And today what we’re going to talk about is how you can test rapid prototypes with users.

6
00:00:24,856 --> 00:00:26,667
So, let’s imagine for a moment —

7
00:00:26,667 --> 00:00:31,747
What if it was possible to create an interactive application without writing much or any code

8
00:00:31,747 --> 00:00:35,943
and get feedback on that interactive application quickly?

9
00:00:35,943 --> 00:00:40,757
I think that’s a pretty powerful dream, and, like Dorothy learned in « The Wizard of Oz »,

10
00:00:40,757 --> 00:00:45,704
dreams can be really powerful, and, sometimes, they can even come true.

11
00:00:45,704 --> 00:00:49,269
And there’s something else that Dorothy learned that we’re going to use in the lecture:

12
00:00:49,269 --> 00:00:51,464
If you’ve seen the movie « The Wizard of Oz », you’ll know that

13
00:00:51,464 --> 00:00:54,700
when Dorothy and her compatriots arrived in the Emerald City,

14
00:00:54,700 --> 00:00:58,741
they saw a big giant wizard that was really scary and freaked them out.

15
00:00:59,633 --> 00:01:03,492
But eventually they learned that that wizard was just a little man behind a curtain,

16
00:01:04,077 --> 00:01:08,440
and the ability of that little man behind the curtain to put on a big realistic show

17
00:01:08,440 --> 00:01:13,356
that, to the visitors to the Emerald City, seemed completely real and immersive,

18
00:01:13,356 --> 00:01:17,527
is exactly what we’re going to accomplish with this video’s prototyping technique.

19
00:01:18,819 --> 00:01:20,579
So, the idea behind Wizard-of-Oz Prototyping

20
00:01:20,579 --> 00:01:24,373
is that we’re going to simulate interactive behaviour and machine functionality

21
00:01:24,373 --> 00:01:28,909
by having a human operator manipulate the levers behind the scenes.

22
00:01:29,571 --> 00:01:32,596
If you want to get a sense for just how this realistic this can be,

23
00:01:32,596 --> 00:01:36,381
watch our friend from « Seinfeld » call a “Movie Phone application”

24
00:01:36,396 --> 00:01:39,978
to try and find out when and where movies are playing.

25
00:01:40,747 --> 00:01:44,537
(Hello, and welcome to Movie Phone.

26
00:01:44,537 --> 00:01:48,655
If you know the name of the movie you’d like to see, press 1.

27
00:01:48,655 --> 00:01:50,487
[beep] C’mon, c’mon…

28
00:01:50,487 --> 00:01:56,156
Using the touch-tone keypad, please enter the first three letters of the movie title, now!

29
00:01:56,156 --> 00:01:58,179
[beep, beep]

30
00:01:58,179 --> 00:02:05,796
You’ve selected « Agent Zero ». If that’s correct, press 1.

31
00:02:05,796 --> 00:02:07,753
What?!

32
00:02:07,753 --> 00:02:09,843
Ugh…)

33
00:02:09,843 --> 00:02:13,468
So George thinks that he’s calling this automated Movie Phone application,

34
00:02:13,468 --> 00:02:17,310
but it turns out that, behind the scenes, it’s just Kramer who’s playing a wizard

35
00:02:17,310 --> 00:02:20,070
and behaving as if he were the machine.

36
00:02:20,070 --> 00:02:23,876
To be honest, he’s not very good — I think that you’d all be able to do better!

37
00:02:24,630 --> 00:02:28,049
Wizard of Oz techniques got their start with speech user interfaces.

38
00:02:28,049 --> 00:02:34,353
The term “Wizard of Oz” was coined by Jeff Kelly in his PhD dissertation around 1980.

39
00:02:34,353 --> 00:02:36,447
And Jeff’s setup at John Hopkins

40
00:02:36,447 --> 00:02:40,101
actually bore more of a resemblance to the Wizard of Oz than you might first imagine;

41
00:02:40,101 --> 00:02:45,778
so, in addition to some one-way mirrors and things like that, there literally was a curtain

42
00:02:45,778 --> 00:02:50,871
separating Jeff, as the wizard, from the participants that were using the speech user interfaces.

43
00:02:50,871 --> 00:02:55,070
And these Wizard-of-Oz prototypes were really important for understanding

44
00:02:55,070 --> 00:02:57,471
what would be an effective speech user interface,

45
00:02:57,471 --> 00:03:01,365
because, at the time, the recognition algorithms weren’t very good,

46
00:03:01,365 --> 00:03:03,253
but we knew they were going to get better.

47
00:03:03,253 --> 00:03:05,739
And Jeff asked the very important questions of:

48
00:03:05,739 --> 00:03:10,763
“If we get the recognition to the point where it was effective, what will the user interaction be?

49
00:03:10,763 --> 00:03:13,612
And what kind of strategies might be [the] most effective?”

50
00:03:13,627 --> 00:03:17,086
So the Wizard of Oz enabled Jeff to time-travel

51
00:03:17,086 --> 00:03:20,625
into a future where speech recognition technology worked better,

52
00:03:20,625 --> 00:03:24,329
to try and understand what the user experience issues of that domain would be.

53
00:03:25,862 --> 00:03:29,380
In general, a Wizard-of-Oz prototype comprises a couple of pieces:

54
00:03:29,380 --> 00:03:33,362
First, you’ve got some kind of user interface that you’re delivering to the user

55
00:03:33,362 --> 00:03:41,790
which feels-ish like a user interface — even if it’s sketches or custom-spoken words.

56
00:03:41,790 --> 00:03:47,036
And, at the same time, there’s not any or much code that goes behind it —

57
00:03:47,036 --> 00:03:53,856
the code and interactivity [that] is being created is a mirage from a human.

58
00:03:53,856 --> 00:03:57,561
Sometimes there’s a user interface that the wizard has,

59
00:03:57,561 --> 00:04:00,656
which they cobbled together to make life better for them —

60
00:04:00,656 --> 00:04:05,480
and in fact back in 2000 colleagues at Berkeley and I built a system called Suede,

61
00:04:05,480 --> 00:04:09,330
which helped automate Wizard-of-Oz testing and speech user interfaces.

62
00:04:09,330 --> 00:04:15,513
Wizard-of-Oz prototypes make sense when it’s faster and cheapter and easier than building the real thing,

63
00:04:15,513 --> 00:04:20,465
and that’s going to be especially true any time you have recognition-based interfaces,

64
00:04:20,465 --> 00:04:23,767
or personalization that’s custom to a user.

65
00:04:24,490 --> 00:04:28,874
This tradeoff of whether it’s cheaper and faster and easier is also going to depend on your abilities:

66
00:04:28,874 --> 00:04:33,293
There will be some things that you know how to do quickly and easily and then you might just implement;

67
00:04:33,293 --> 00:04:37,601
and there’s other things that may be more difficult or more time-consuming for you,

68
00:04:37,601 --> 00:04:40,044
and those you’ll choose to Wizard-of-Oz.

69
00:04:40,644 --> 00:04:46,192
Wizard-of-Oz prototypes, like any other kind of prototype, can be higher or lower fidelity.

70
00:04:46,192 --> 00:04:48,729
And there’s some important tradeoffs here —

71
00:04:48,729 --> 00:04:53,081
The higher fidelity the prototype, the more it feels like the real thing to users —

72
00:04:53,081 --> 00:04:58,982
In fact, sometimes you can build an interface that people don’t even know is being powered by a wizard,

73
00:04:58,982 --> 00:05:04,036
as in the famous story of the mechanical Turk, who is inside a chess machine.

74
00:05:04,467 --> 00:05:09,209
However, making a high-fidelity user interface is going to take a lot of time,

75
00:05:09,209 --> 00:05:15,022
and users may be more reluctant to critique a user interface that’s really high-fidelity,

76
00:05:15,022 --> 00:05:20,600
and so, early on, I recommend quick-and-dirty paper prototypes being the most powerful thing

77
00:05:20,600 --> 00:05:22,520
to use for Wizard-of-Oz tests.

78
00:05:22,520 --> 00:05:25,600
Because users will feel like they can give you any kind of feedback;

79
00:05:25,600 --> 00:05:28,272
it’s clear you haven’t put a lot of time into it yet,

80
00:05:28,272 --> 00:05:33,110
and it’s fast for you to make, and so you actually haven’t a lot time into it yet.

81
00:05:33,772 --> 00:05:37,213
A few years ago, my friend Sep Kamvar created a company called Aardvark,

82
00:05:37,213 --> 00:05:39,926
which is a social search engine.

83
00:05:39,926 --> 00:05:45,494
It was really good at helping you use your social network to find answers from other people.

84
00:05:45,971 --> 00:05:50,576
So the way it works is you ask a question via instant message, via Google chat;

85
00:05:50,576 --> 00:05:55,621
and it’s going to be automatically routed to a friend of yours, or a friend of a friend,

86
00:05:55,621 --> 00:05:58,601
who has the expertise to answer that question.

87
00:05:58,601 --> 00:06:04,336
One of the hard parts about building a service like this is: How do you do the routing?

88
00:06:04,336 --> 00:06:08,017
Eventually they figured out an algorithm to do this; but to bootstrap that —

89
00:06:08,017 --> 00:06:10,384
both to figure out what algorithm to build,

90
00:06:10,384 --> 00:06:15,228
and to get a user experience going before they had built anything —

91
00:06:15,228 --> 00:06:17,716
they had a Wizard-of-Oz prototype.

92
00:06:17,716 --> 00:06:21,476
So the way that worked was that Aardvark employees would get the question,

93
00:06:21,476 --> 00:06:25,332
and they would manually select the person that was going to get routed to.

94
00:06:25,332 --> 00:06:29,200
Damon Horowitz, the co-founder of Aardvark, put it this way in a conference:

95
00:06:29,200 --> 00:06:33,923
he said that, “If people like it in this super-crappy form, then it’s worth building,

96
00:06:33,923 --> 00:06:38,261
because they’ll like it even more when we build the real thing!”

97
00:06:38,277 --> 00:06:43,708
So, if you want to be like Kramer or Aardvark, how can you make a wizard-powered prototype?

98
00:06:43,708 --> 00:06:46,738
Here’s a sketch of what you’ll need to do in five easy steps:

99
00:06:47,246 --> 00:06:49,582
First, figure out what scenarios you’re going to support.

100
00:06:49,582 --> 00:06:54,363
It’s a lot of easier to build a Wizard-of-Oz prototype for a limited set of functionality.

101
00:06:54,840 --> 00:06:56,372
I like Wizard-of=Oz prototypes

102
00:06:56,372 --> 00:06:59,892
because it forces you to figure out what should happen in response to human behaviour.

103
00:07:00,847 --> 00:07:03,717
Then, put together some kind of user interface skeleton —

104
00:07:03,717 --> 00:07:06,239
the things that the end users are going to see.

105
00:07:06,593 --> 00:07:11,266
Third, if you need to, develop the “hooks” for the wizard [to] input if you’re going to have a remote wizard

106
00:07:11,266 --> 00:07:15,277
or if they’re going to control some software functionality from behind the scenes.

107
00:07:16,046 --> 00:07:17,485
Fourth, and as part of this,

108
00:07:17,485 --> 00:07:21,491
you’ll need to figure out exactly what kind of input the wizard is allowed to offer:

109
00:07:21,491 --> 00:07:26,847
Are they going to be selecting from a menu of options? offering free response? speaking text?

110
00:07:26,847 --> 00:07:31,228
Will you give them a decision tree that controls their behaviour?

111
00:07:31,228 --> 00:07:33,617
or you’re going let them be more free-form?

112
00:07:33,617 --> 00:07:36,917
If you have a paper prototype, their role is going to pretty manual —

113
00:07:36,917 --> 00:07:40,886
adding widgets and moving sliders, and making the interface go.

114
00:07:41,733 --> 00:07:43,741
When you’re doing this, it’s important to remember that,

115
00:07:43,741 --> 00:07:49,577
ultimately, the functionality that you’re wizarding now is ultimately going to be replaced with a computer,

116
00:07:49,577 --> 00:07:53,818
and so at some point down the line, you will have to build some software to make it go.

117
00:07:53,818 --> 00:07:58,776
And that’s important, because it can be easy to fake stuff that’s not possible to ever happen.

118
00:07:59,469 --> 00:08:03,430
And, last, as with any prototype that you’re going to test with people,

119
00:08:03,430 --> 00:08:07,170
practice it first with a friend or a colleague to get the hang of both doing the wizard —

120
00:08:07,170 --> 00:08:10,095
it’s a very different kind of interaction than you may be used to —

121
00:08:10,095 --> 00:08:15,442
and also, figuring out what scenarios are most effective, what instructions you’ll need to give people.

122
00:08:15,442 --> 00:08:19,823
Get out the easy bugs in the user interface before you bring in real users.

123
00:08:20,300 --> 00:08:25,641
And now you’re ready to run the user interface: Like I said, you’ll want to practice with a friend at first.

124
00:08:27,380 --> 00:08:32,118
Once you’ve got the really obvious kinks out, and when you’ve got your patter down and the tasks ready,

125
00:08:32,118 --> 00:08:35,180
then it’s time to recruit some people to come in and try out your prototype.

126
00:08:35,180 --> 00:08:41,568
You can even go to places like train stations, or airports, or city street corners, or coffee shops

127
00:08:41,568 --> 00:08:45,814
as a way of going to the people who might be using your system.

128
00:08:47,230 --> 00:08:50,428
In a Wizard-of-Oz prototype, there’s two roles:

129
00:08:50,428 --> 00:08:53,270
you’ve got a facilitator who’s going to talk to the user,

130
00:08:53,270 --> 00:08:55,871
and you’ve got a wizard who’s going to manipulate the prototype.

131
00:08:55,871 --> 00:09:00,091
If you have the luxury of a team, it’s helpful if there are two different people,

132
00:09:00,091 --> 00:09:03,745
in part because your mind is going to be full of all the things that you’ll have to do,

133
00:09:03,745 --> 00:09:08,696
and if you separate these roles, each person can concentrate more, do a better job,

134
00:09:08,696 --> 00:09:13,568
and you have two sets of eyes looking at what the person’s doing, so you can learn more.

135
00:09:14,584 --> 00:09:18,920
But can do this yourself if you need to — these can be one person doing two roles.

136
00:09:19,597 --> 00:09:22,375
If you’re trying to convince somebody that this is actually is a real system,

137
00:09:22,375 --> 00:09:27,031
then you’ll want your wizard to be hidden or remote somehow, like the Aardvark example.

138
00:09:27,354 --> 00:09:29,492
And think about how you’d like to get feedback from users —

139
00:09:29,492 --> 00:09:33,477
this is something that we’ll touch on several times more throughout this course.

140
00:09:33,477 --> 00:09:36,985
For now, think about whether you’d like to get think-aloud feedback —

141
00:09:36,985 --> 00:09:39,913
which is, when the person’s using the prototype,

142
00:09:39,913 --> 00:09:45,436
have them speak aloud what it is that’s going through their mind, what decisions they’re making and why,

143
00:09:45,436 --> 00:09:51,013
what they’re confused by when they run up against things that they can’t figure out what to do.

144
00:09:51,721 --> 00:09:58,111
This think-aloud protocol is great for getting stuff that otherwise you might not know or learn.

145
00:09:58,111 --> 00:10:03,067
However, as you can imagine, thinking aloud can change how people interact with the prototype.

146
00:10:03,067 --> 00:10:07,325
So, if think-aloud is too distracting, you can ask people at the end of the study

147
00:10:07,325 --> 00:10:09,865
what was going through their mind and what they were thinking about —

148
00:10:09,865 --> 00:10:11,821
that’s called a retrospective approach.

149
00:10:11,821 --> 00:10:14,402
If you like, you can even show them the video of them using it.

150
00:10:14,402 --> 00:10:18,716
You can then say things like, “When you got stuck here, what was the issue?”

151
00:10:19,609 --> 00:10:24,051
Third, if you have specific things that you’d like to make sure that your interface does well,

152
00:10:24,051 --> 00:10:30,155
you can have categories of problems or heuristics that you asked your users to pay attention to.

153
00:10:30,155 --> 00:10:34,622
And heuristic evaluation is something that we’re going to talk more about later in this course also.

154
00:10:35,499 --> 00:10:38,065
Once you’re all done, make sure to thank people for their time.

155
00:10:38,065 --> 00:10:43,162
You may want to give them a gift certificate, or ice-cream, or some other kind of thank-you.

156
00:10:44,209 --> 00:10:49,711
And, if you haven’t yet told them that there’s actually a wizard behind the scenes operating the user interface,

157
00:10:49,711 --> 00:10:52,311
the end of the experiment is probably a really good time to that —

158
00:10:52,311 --> 00:10:54,750
You’ll want to make sure to be honest with people.

159
00:10:54,750 --> 00:10:57,974
You can use wizards throughout the entire development of your software,

160
00:10:57,974 --> 00:10:59,942
not just at the very beginning.

161
00:10:59,942 --> 00:11:06,066
And so, if you have the fully-functional software here, and over time you move towards there,

162
00:11:06,066 --> 00:11:10,274
you can use wizards to fill in the gap of whatever is not built at that time.

163
00:11:10,274 --> 00:11:13,523
So, the wizard may do all of the interactions at the very beginning,

164
00:11:13,523 --> 00:11:16,201
and then, as you get closer and closer to a final project,

165
00:11:16,201 --> 00:11:19,918
you can have it just fill in only the parts that aren’t done yet.

166
00:11:20,441 --> 00:11:24,047
As you can see, there’s a lot to like about Wizard-of-Oz prototypes:

167
00:11:24,463 --> 00:11:27,791
They can be really fast to make, which makes them cheaper,

168
00:11:27,791 --> 00:11:31,121
and it also means you get to turn the crank on iteration more.

169
00:11:32,429 --> 00:11:35,618
Because they are quick, you can rapidly create multiple alternatives.

170
00:11:35,618 --> 00:11:39,078
And you decide that you want the “implementation” to work differently,

171
00:11:39,078 --> 00:11:42,649
you just give your wizard different instructions; there’s no code to rewrite.

172
00:11:43,511 --> 00:11:45,141
Because it is in a sense interactive,

173
00:11:45,141 --> 00:11:49,002
it is more real and you get better feedback than just from paper prototyping,

174
00:11:49,002 --> 00:11:52,135
although using paper prototypes as the substrate for a wizard

175
00:11:52,135 --> 00:11:55,724
is an excellent strategy early on in the design process.

176
00:11:56,308 --> 00:12:00,865
It’s a great way to get feedback about bugs and problems with your user interface design.

177
00:12:00,865 --> 00:12:05,160
Every single user interface that I’ve ever built, or that I’ve seen built,

178
00:12:05,160 --> 00:12:08,591
or that students of mine have ever built, has had bugs early on.

179
00:12:08,591 --> 00:12:11,325
It’s too hard to get right the first time.

180
00:12:11,325 --> 00:12:15,543
You’re going to iterate and fail fast, so that you can succeed sooner.

181
00:12:15,543 --> 00:12:18,914
And a Wizard of Oz is a great tool to be able to do that.

182
00:12:20,053 --> 00:12:22,544
Because you’re getting feedback from users early,

183
00:12:22,544 --> 00:12:26,437
it’s a great way to place people at the centre of the development process.

184
00:12:27,006 --> 00:12:29,911
It’s also really wonderful for forward-looking applications

185
00:12:29,911 --> 00:12:34,608
as we start to think about how sensors and cameras and other futuristic technologies

186
00:12:34,608 --> 00:12:37,214
might enable other new kinds of user interfaces.

187
00:12:37,214 --> 00:12:42,230
You can imagine these now by building Wizard-of-Oz prototypes.

188
00:12:42,522 --> 00:12:46,023
And lastly, strange as it may sound, you actually learn a lot

189
00:12:46,023 --> 00:12:50,120
about what your application logic of your interface ought to be, and what makes sense

190
00:12:50,120 --> 00:12:54,268
by playing the wizard and embodying that role in the system.

191
00:12:54,991 --> 00:12:58,281
But there are some disadvantages too:

192
00:12:58,281 --> 00:13:03,175
It can be easy to gloss over errors in technology that will ultimately come back to bite you —

193
00:13:03,175 --> 00:13:06,641
like speech recognition systems for example.

194
00:13:06,656 --> 00:13:11,635
If, at the end of the day, you’re going to have bugs in your speech system,

195
00:13:11,635 --> 00:13:14,447
you’ll want to make sure that your Wizard-of-Oz prototype encodes

196
00:13:14,447 --> 00:13:16,883
what happens when the speech reco[gnition]s screws up.

197
00:13:16,883 --> 00:13:20,050
Don’t assume it’s going to work perfectly all the time.

198
00:13:20,050 --> 00:13:24,716
’Cause if you’re not careful, you can end up simulating a technology that doesn’t exist and may never.

199
00:13:25,393 --> 00:13:27,806
People, for better or for worse, are inconsistent,

200
00:13:27,806 --> 00:13:32,888
and, to get them to play the wizard role, can require training, so that’s a ramp-up time.

201
00:13:32,888 --> 00:13:35,997
And because you have to have a physical human there to run it,

202
00:13:35,997 --> 00:13:38,193
your runtime efficiency is slower,

203
00:13:38,193 --> 00:13:42,159
and it can be more exhausting for the people that are playing the wizard role.

204
00:13:42,159 --> 00:13:46,981
Wizards are great for some kind[s] of functionality, like Kramer did with the speech user interface.

205
00:13:46,981 --> 00:13:51,370
But other stuff, it’s harder to figure out what you can do with a wizard interface,

206
00:13:51,370 --> 00:13:56,233
and it may not be quite as effective — it may be klunkier.

207
00:13:56,233 --> 00:13:57,964
Or it might be simply be inappropriate —

208
00:13:57,964 --> 00:14:05,845
there’re some situations where having a wizard around may not be ok.

209
00:14:05,845 --> 00:14:09,097
All in all, I think wizards are an incredibly powerful technique,

210
00:14:09,097 --> 00:14:11,523
and a really fun one to stretch your creativity.

211
00:14:11,523 --> 00:14:17,075
What kinds of new user interfaces can you imagine by creating a Wizard-of-Oz prototype?

212
00:14:17,075 --> 00:14:19,761
I’d like to thank Stephen Dow in creating this lecture —

213
00:14:19,761 --> 00:14:24,461
I drew on a lot of the materials that’s he’s used in writing and speaking about Wizard-of-Oz prototypes.

214
00:14:24,461 --> 00:14:30,918
And if you’re interested in learning more about wizards, you can see these resources here.