1
00:00:00,001 --> 00:00:06,057
[Dorothy Bishop] I'm going to talk today about evaluating alternative intervention approaches to dyslexia.

2
00:00:06,059 --> 00:00:07,462
[slide with talk title]

3
00:00:07,463 --> 00:00:13,667
The conventional approaches that you get really tend to rely on the fact that we've now got of evidence

4
00:00:13,668 --> 00:00:18,409
that most children with dyslexia have problems in what is called phonological awareness,

5
00:00:18,410 --> 00:00:23,440
that is, they don't necessarily hear all the different sounds in speech,

6
00:00:23,441 --> 00:00:28,380
and therefore have difficulty relating them to letters when they are trying to read.

7
00:00:28,381 --> 00:00:37,516
And most of the interventions that are mainstream these days would focus on trying to train children to identify sounds in words and relate them to letters.

8
00:00:38,971 --> 00:00:45,414
But this sort of intervention has been shown to be effective, and there have been a number of large scale studies.

9
00:00:45,415 --> 00:00:51,725
But nevertheless, it has to be fairly prolonged for some children, and there are children for whom,

10
00:00:51,725 --> 00:00:57,870
even though they can learn this way to actually sound out words and read, they don't necessarily read fluently.

11
00:00:57,912 --> 00:01:03,306
It's still an effort for them, and they don't sort of get to the degree of automaticity that you might expect,

12
00:01:03,307 --> 00:01:07,683
And it's certainly the case that methods that work for many children don't work for all children,

13
00:01:07,684 --> 00:01:10,756
and there is a hard core of children who remain very hard to treat.

14
00:01:10,757 --> 00:01:15,943
It is for this reason that many parents do get very concerned about whether there is something else they should be doing

15
00:01:15,944 --> 00:01:21,997
if they are finding that their child is either not getting intervention, or that the intervention doesn't seem to be working very well.

16
00:01:21,998 --> 00:01:29,446
And there are a whole load of things out there that are on offer, and the problem for the parents, I think -

17
00:01:29,447 --> 00:01:33,982
and/or indeed for adults who themselves, might want to have further intervention for dyslexia -

18
00:01:33,983 --> 00:01:37,691
is that they want to know, "how do I distinguish something that might work for me",

19
00:01:37,691 --> 00:01:41,641
from something that is just some sort of snake-oil merchant who is out there to make money.

20
00:01:41,642 --> 00:01:44,482
And that's what I want to try and address today.

21
00:01:44,482 --> 00:01:51,544
principally from the perspective of how you might evaluate scientific evidence that people put forward.

22
00:01:51,544 --> 00:01:57,486
But perhaps before going onto that, it's worth going into some relatively commonsense things.

23
00:01:57,618 --> 00:02:06,612
I would say that there are certain things that should ring alarm bells if people are advertising some sort of treatment for child dyslexia.

24
00:02:06,649 --> 00:02:13,064
The first thing is if the intervention has been developped by somebody who has no academic track record,

25
00:02:13,065 --> 00:02:17,834
no experience of doing research in this field, and hasn't published anything in this field,

26
00:02:17,834 --> 00:02:24,541
if the intervention isn't endorsed by people in the mainstream dyslexia field,

27
00:02:24,542 --> 00:02:27,633
that should also sound a note of caution.

28
00:02:27,634 --> 00:02:29,820
Of course, the mainstream people aren't always right.

29
00:02:29,821 --> 00:02:32,971
It's possible that somebody with no background will develop something marvelous.

30
00:02:32,972 --> 00:02:37,474
But if that were the case, you would expect it to be pretty quickly picked up by people in the mainstream,

31
00:02:37,475 --> 00:02:40,798
who are really, on the whole, pretty keen to find things that will work.

32
00:02:40,798 --> 00:02:47,811
And you obviously want to look at whether somebody is asking for a lot of money for something that hasn't been proven.

33
00:02:47,812 --> 00:02:56,216
And what is also, to my mind, a worrying sign, is if somebody promoting a treatment is relying heavily just on testimonials

34
00:02:56,217 --> 00:03:04,154
from individuals who claimed to have been cured, rather than having any sort of proper scientific evaluation or kind of controls.

35
00:03:04,155 --> 00:03:10,613
And it's worth noting that human beings have a tendency to be terribly impressed by testimonials,

36
00:03:10,613 --> 00:03:16,749
and even myself, as somebody with a scientific training, I find if, you know, I've got headaches and somebody comes along and says:

37
00:03:16,750 --> 00:03:20,603
"I was cured by such and such, and I went to my herbalist and it worked",

38
00:03:20,603 --> 00:03:27,094
you know, you're often very tempted to be much more swayed by that sort of evidence than by a pagefull of numbers and figures.

39
00:03:27,095 --> 00:03:34,677
And this just a human tendency: we are naturally built to really take advice from other people and to rely on what they tell us.

40
00:03:34,677 --> 00:03:38,494
But in the contexts of these sorts of interventions, that's really quite dangerous,

41
00:03:38,494 --> 00:03:44,271
because, when somebody gives a testimonial, that's just one person, their only individual experience,

42
00:03:44,272 --> 00:03:48,273
And people you don't hear from tend to be the people who tried it, and it didn't work.

43
00:03:48,274 --> 00:03:51,881
And you don't know how many of them there are: there may be thousands of them.

44
00:03:51,882 --> 00:03:56,049
But they're not going to publicize the fact that they tried it and it didn't work.

45
00:03:56,050 --> 00:04:01,945
And so, testimonials are often very much at odds with more scientific evaluations.

46
00:04:01,945 --> 00:04:10,096
.... to turn out that when somebody says there is scientific evidence for what they're doing, how you should interpret that.

47
00:04:10,097 --> 00:04:14,964
And that's jolly difficult even for scientists sometimes: there is disagreement - let alone for the general public.

48
00:04:14,965 --> 00:04:20,727
But again, I think, there are some sort of general rules of thumb that you can go by

49
00:04:20,728 --> 00:04:24,020
for telling that a treatment is likely to be effective.

50
00:04:25,450 --> 00:04:38,229
And when I discuss this, I'm going to illustrate it by taking the example of the Dore treatment - that's DORE, named after Wynford Dore, it's originator.

51
00:04:38,230 --> 00:04:46,360
ANd I'm picking on this largely because it is a non-mainstream treatment that isn't widely accepted by the experts,

52
00:04:46,361 --> 00:04:49,899
and yet it does claim that there is some scientific evidence to support it,

53
00:04:49,900 --> 00:04:54,277
which has lead the scientists to look at it quite critically and quite carefully,

54
00:04:54,277 --> 00:04:57,654
which is what we would do with any scientific evidence that comes along:

55
00:04:57,655 --> 00:05:04,834
once it's out in the public domain and published, people tend to go and look at it as carefully as they possibly can.

56
00:05:04,834 --> 00:05:11,944
Now, the Dore method is interesting to us, because it does illustrate the case where there is disagreement

57
00:05:11,945 --> 00:05:15,197
as to whether the evidence is showing that its' effective or not.

58
00:05:15,198 --> 00:05:20,587
And so, what I want to explain is why it is the case that despite this published evidence,

59
00:05:20,588 --> 00:05:25,324
most of the experts are not impressed of the efficacy of the Dore treatment.

60
00:05:25,325 --> 00:05:32,227
But the general points that I'll make would apply to any other treatment that was out there, whether (?) there was evidence being produced.

61
00:05:32,751 --> 00:05:39,961
So, first of all, what is the Dore method? Well, it's a method that has been proposed for curing problems

62
00:05:39,962 --> 00:05:44,774
that are thought to originate in the part of the brain called the cerebellum, which is at the back of the brain,

63
00:05:44,774 --> 00:05:49,999
and it was developed by Wynford Dore as a method for helping his dyslexic daughter.

64
00:05:49,999 --> 00:05:53,578
He has written a book about the history of how this came to became (?) about,

65
00:05:53,579 --> 00:05:58,973
and he was a classic instance of a parent who was rather desperate to help his daughter who, for many years,

66
00:05:58,973 --> 00:06:03,513
had been through the educational system and failed, and was getting increasingly depressed.

67
00:06:03,514 --> 00:06:10,842
And he tried various things, he talked to various experts, and he ended up with this program that's been put forward,

68
00:06:10,842 --> 00:06:15,541
which is an individualized program, where the child follows various sorts of exercises,

69
00:06:15,542 --> 00:06:21,019
which are done for about ten minutes twice a day, over quite a long period of time,

70
00:06:21,019 --> 00:06:25,265
varying, depending on the severity of the problem, from maybe 6 months to 2 years.

71
00:06:25,928 --> 00:06:30,984
And the child is assessed at regular intervals and different exercises may be prescribed.

72
00:06:30,985 --> 00:06:38,550
Now, the theory behind the Dore method is that dyslexia and other learning difficulties -

73
00:06:38,550 --> 00:06:43,661
it's not just dyslexia it claims to help, but also Attention Deficit problems ...(?) hyperactive -

74
00:06:43,661 --> 00:06:49,043
are thought to arise within the cerebellum: the cerebellum just doesn't develop normally,

75
00:06:49,043 --> 00:06:54,474
and the argument is that you can have different cerebellar impediments in different people,

76
00:06:54,475 --> 00:06:57,198
and that's why you can get this range of different symptoms,

77
00:06:57,199 --> 00:07:04,061
but that you can diagnose them by specific tests of test of mental and physical coordinations.

78
00:07:04,062 --> 00:07:12,679
And what you are then supposed to do is these exercises, which are not anywhere fully described in the public domain,

79
00:07:12,679 --> 00:07:17,623
because they are commercially sensitive, but there are some examples given, and it's clear that what they do

80
00:07:17,623 --> 00:07:22,714
is focused largely on training balance and hand-eye coordination in children.

81
00:07:22,715 --> 00:07:29,890
So you might be asked to stand on a cushion on one leg, or to throw a bean bag from one hand to another

82
00:07:29,891 --> 00:07:37,296
while you are doing that, just stand on a wobble board (?) and balance, or to follow something with your eyes in a particular way.

83
00:07:39,188 --> 00:07:48,267
So, the idea is that these are all things that the cerebellum is involved in, by training up the cerebellum, you may improve its general abilities.

84
00:07:48,727 --> 00:07:53,435
So, what is the evidence for this underlying theory?

85
00:07:53,436 --> 00:07:58,405
Well, it's not a proven theory, but there is some support for it.

86
00:07:58,405 --> 00:08:06,920
Certainly, people trying to look at what is going on in the brain in dyslexia have proposed many different theories

87
00:08:06,921 --> 00:08:09,314
about what the underlying causes might be.

88
00:08:09,315 --> 00:08:14,735
If you look at the brain in a brain scanner of somebody with dyslexia, it typically looks totally normal.

89
00:08:14,735 --> 00:08:19,256
There's certainly no big holes in the head or anything like that, that you are going to see on a scanner.

90
00:08:19,257 --> 00:08:24,865
But the argument is being made that there may be regions of the cerebellum that are perhaps slightly smaller than they should be

91
00:08:24,866 --> 00:08:28,046
or not functioning quite as they should be.

92
00:08:28,047 --> 00:08:33,820
And this theory has some support, although not everybody would agree with it

93
00:08:33,820 --> 00:08:38,843
and there is certainly other theories equally plausible at the moment that are around.

94
00:08:39,444 --> 00:08:44,664
The notion - the cerebellum is important for getting things automated.

95
00:08:44,665 --> 00:08:50,675
So you can - when you learn to drive a car, first of all, it's very slow and effortfull, and you have to think about everything you do.

96
00:08:50,675 --> 00:08:55,185
By the time you are a skilled driver, it's no longe the case that you have to do that,

97
00:08:55,186 --> 00:08:58,061
you just drive around without thinking about it.

98
00:08:58,061 --> 00:09:00,063
You can do all sorts of other things while you are driving.

99
00:09:00,064 --> 00:09:07,321
So, the argument is that with reading, most people, similarly, become very automatic in how they learn to read:

100
00:09:07,322 --> 00:09:14,902
you do it without thinking about it, but for the dylexic it remains effortfull because the cerebellum is not functioning normally

101
00:09:14,902 --> 00:09:19,387
and it's the cerebellum that helps you get your skills automatized.

102
00:09:19,388 --> 00:09:26,874
And in support of this, it has been argued that in many people with dyslexia, there are some associated problems with motor coordination,

103
00:09:26,875 --> 00:09:32,792
..... (?) physical skills and so on, and that too could be a sign of a problem with the cerebellum.

104
00:09:32,793 --> 00:09:39,876
Again, that's fairly controversial, it's not being found in all children, and the arguments go to and fro.

105
00:09:39,876 --> 00:09:47,492
But this is not a sort of theory that is particularly disapproved of by the mainstream. People are debating it.

106
00:09:47,493 --> 00:09:55,052
The difficult stumbling block, though, for the Dore approach to treatment comes with the idea that

107
00:09:55,053 --> 00:10:02,531
if you train the motor skills, that is a sort of coordination between different muscles and movements

108
00:10:02,532 --> 00:10:09,374
and between their eyes and hands, that this will somehow have a knock-on effect on things like reading.

109
00:10:09,375 --> 00:10:16,401
And indeed, David Randall and colleagues, who published this initial study on the treatment,

110
00:10:16,401 --> 00:10:23,410
describe it as something of a leap of faith, because the cerebellum is actually known to be a very complicated organ,

111
00:10:23,411 --> 00:10:27,249
with lots of different regions, which are fairly independent from one another.

112
00:10:27,250 --> 00:10:34,590
So there is no real reason to suppose that if you train one part of the cerebellum, it will have somehow a generalized benefit.

113
00:10:34,591 --> 00:10:40,750
And indeed, you could say: "Well, if it were the case that this is true, if you'd had a chance, you would go to skateboarding,

114
00:10:40,751 --> 00:10:48,710
or playing ping-pong, or things like that, ....... (?) or perhaps ballet dancing, things that require balance and coordination,

115
00:10:48,711 --> 00:10:52,653
that should protect you against dyslexia". There is really not much evidence for that,

116
00:10:52,654 --> 00:10:57,645
on the contrary, there is some very good sportsmen who - gymnasts and people with dyslexia.

117
00:10:57,645 --> 00:11:06,790
So it is hard to see how the logic of saying "Train these motor skills and somehow the whole cerebellum function some day improves2

118
00:11:06,790 --> 00:11:12,876
But what does the published evidence look like? Because the theory might be, you know, questionable,

119
00:11:12,877 --> 00:11:16,956
but basically, what the parents are going to say is, "What matter is, does it work?"

120
00:11:18,215 --> 00:11:24,717
Well, there is a published study on the intervention, which claims that it shows that it really does work

121
00:11:24,717 --> 00:11:28,297
if you compare children who have the intervention and children who don't.

122
00:11:28,298 --> 00:11:35,999
And two papers have been reported - one from the initial phase of the study, and the other from a subsequent phase -

123
00:11:36,866 --> 00:11:45,538
And they are reported in the Journal of Dyslexia which, in 2003, published the first paper

124
00:11:45,663 --> 00:11:53,890
which was on just under - started with a sample statistics on the 300 children who were all attending a .... (?) primary school.

125
00:11:53,890 --> 00:11:58,021
And the researchers went in and screened all the children on the dyslexia screening test,

126
00:11:58,021 --> 00:12:01,457
to pull out children who would be suitable for enrolment in the study.

127
00:12:01,458 --> 00:12:10,647
But the first thing that is more important to note is that these were not children who had a very high rate of diagnosis of dyslexia.

128
00:12:10,648 --> 00:12:19,265
So, there were 35 in the group, and about a third of those came out as having a strong risk of dyslexia on this dyslexia screening test.

129
00:12:19,265 --> 00:12:26,744
Another 21% came out with a mild risk, but about half of these children were not really ...... (?) in this category

130
00:12:26,745 --> 00:12:31,496
and they were just picked because their schools (?) were relatively lower compared to the other children.

131
00:12:31,497 --> 00:12:38,199
And there were only a total of 6 children who had previously been diagnosed with dyslexia, out of the 35.

132
00:12:38,200 --> 00:12:42,989
There were a couple with a diagnosis of dyspraxia and one with ADHD diagnosis.

133
00:12:42,990 --> 00:12:48,464
So this is not really a sample consisting of children really with severe problems on the whole.

134
00:12:48,464 --> 00:12:50,660
There were few in there with major difficulties.

135
00:12:50,661 --> 00:12:58,037
Nevertheless, the originators of the treatment would argue even quite mild problems might be worth treating with this

136
00:12:58,037 --> 00:13:01,890
and so you could argue this study is nevertheless of value.

137
00:13:01,891 --> 00:13:07,834
So what they did, they started out well in this study: they divided the children randomly in treated and untreated groups,

138
00:13:07,834 --> 00:13:13,687
which is, as I am going on to explain later, is an important part of a good study.

139
00:13:13,687 --> 00:13:20,727
And if you look at the results that are described on the promotion materials of the DORE organization,

140
00:13:20,727 --> 00:13:28,476
they are all in Dore's book that he published, "Dyslexia, the miracle cure", he described the results as stunning

141
00:13:28,477 --> 00:13:35,039
and said that reading age increased threefold, comprehension age increased almost fivefold

142
00:13:35,040 --> 00:13:40,640
and writing skills by what he described as "an extraordinary 17-fold".

143
00:13:40,641 --> 00:13:47,856
Of course, everybody reading that think "Wow, my child is going to take off like a rocket if we put him on this intervention."

144
00:13:47,856 --> 00:13:56,806
But unfortunately, these figures are really a classic instance of how statistics can be manipulated in a very misleading way.

145
00:13:57,414 --> 00:14:04,912
So, for a start, they were not based on any comparison between the control children and the untreated children -

146
00:14:04,913 --> 00:14:09,177
sorry, the control children and the treated children.

147
00:14:09,178 --> 00:14:17,568
They were - instead, they just took all the children who would be treated and looked at how they did on a group reading test

148
00:14:17,569 --> 00:14:21,044
that had been administered by the school every year.

149
00:14:21,045 --> 00:14:30,727
And the children had had this on two occasions prior to the intervention - so, 3 months before it started and a year before that -

150
00:14:30,727 --> 00:14:35,345
and on two occasions after the intervention, after this whole long 4-year period.

151
00:14:35,346 --> 00:14:44,263
And what the researchers did was to really just plot the average schools of the group over these 4 time periods

152
00:14:44,264 --> 00:14:49,639
and show that if you compared the amount of change from the first time point to the second,

153
00:14:49,640 --> 00:14:54,394
which was before they had had any treatment, it was a certain amount

154
00:14:54,395 --> 00:15:00,347
and if you then compared the second to the third time point, so the treatment had been going on (?) between those two,

155
00:15:00,347 --> 00:15:02,663
there was a different amount of change.

156
00:15:02,664 --> 00:15:06,672
And then they divided one by the other and showed that there was this threefold improvement.

157
00:15:06,673 --> 00:15:13,301
But it's a very, very misleading way of depicting these data, because if you look at them on a graph, here,

158
00:15:13,301 --> 00:15:18,168
you can see that the only odd thing about the data - well, there's two odd things about the data:

159
00:15:18,169 --> 00:15:23,171
one is that at most time points, these children are reading at absolutely normal levels.

160
00:15:23,172 --> 00:15:27,214
So it's not clear why they are regarded as having risk for dyslexia;

161
00:15:27,214 --> 00:15:32,947
and the one time point when they're not, is the time point 3 months before they are involved in the study,

162
00:15:32,947 --> 00:15:39,363
where there is a bit of a drop. But it's really not an impressive demonstration of change

163
00:15:39,364 --> 00:15:46,794
and this division of one time period by another is very misleading, because it just gives double weighting

164
00:15:46,794 --> 00:15:52,357
to this one low period of three months before the treatment started.

165
00:15:52,358 --> 00:15:57,610
And they did the same thing again with these other figures of massive increases that they talk about,

166
00:15:57,611 --> 00:16:06,830
using data from the SATS tests administered by teachers, which are not really regarded as particularly precise or rigorous tests,

167
00:16:06,831 --> 00:16:14,025
and really group children in a fairly global way at level 2, 3 or 4.

168
00:16:14,026 --> 00:16:20,494
Level 2 is average for 7 year old, 3 is average for 9 year old, and 4 is average for an 11 year old.

169
00:16:20,495 --> 00:16:27,876
And to give you an idea of the sort of misleading nature of these massive changes they talk about,

170
00:16:27,876 --> 00:16:34,125
on the writing test, where there is this incredible change that they talk about, of a 17-fold increase,

171
00:16:34,126 --> 00:16:40,792
the score at age 8, the average score was 2.5, which is about what you'd expect from a 8 year old.

172
00:16:40,792 --> 00:16:47,103
At age 9, it was 2.56, which is a little bit better, but not much.

173
00:16:47,104 --> 00:16:53,486
And then, they argue, the intervention came in, and at age 10, the children scored 2.95.

174
00:16:53,486 --> 00:16:56,888
They are still rather below where they ought to be at the age of 10.

175
00:16:56,888 --> 00:17:01,372
It looks as if on this particular writing assessment, the children were just rather creeping along.

176
00:17:01,373 --> 00:17:09,878
But because the difference between 2.53 and 2.56 is less than the difference between 2.56 and 2.95,

177
00:17:09,879 --> 00:17:17,935
they make a big computation of dividing one by the other, actually coming out with the number 17, which is a wrong number (?): it's actually 13.

178
00:17:17,935 --> 00:17:24,646
So there is a 13-fold change. But if you look at the overall numbers, this is really not so an impressive game at all.

179
00:17:24,647 --> 00:17:27,761
It's really a very misleading way of presenting the numbers.

180
00:17:27,762 --> 00:17:36,509
So, most people would say, this is really smoke in mirrors in terms of using statistics in a way that isn't really valid.

181
00:17:36,925 --> 00:17:42,129
The other thing that is of notice is that all these results that have given so much publicity

182
00:17:42,130 --> 00:17:47,697
in promoting the treatment about these massive changes, haven't talked about the control group at all.

183
00:17:47,697 --> 00:17:51,428
They've just talked about, "Well, we've got these children, before treatment they did this,

184
00:17:51,428 --> 00:17:53,800
and after treatment they did that, and it has all gone up".

185
00:17:53,801 --> 00:17:59,717
And of course, if schools do go up after treatment, it's not necessarily because the treatment works:

186
00:17:59,717 --> 00:18:02,299
There are lots of other reasons you need to bear in mind.

187
00:18:02,951 --> 00:18:07,555
And the first of which is just, on some things, you get better because you get older,

188
00:18:07,556 --> 00:18:13,635
so that if you were to measure shoe sizes before the DORE treatment and after it, it would go up,

189
00:18:13,636 --> 00:18:16,663
but it wouldn't mean that it made your feet grow bigger.

190
00:18:16,664 --> 00:18:24,339
Now, clearly, that's a silly example in most cases, because people try to use measures that don't necessarily change with age,

191
00:18:24,340 --> 00:18:26,643
or that are adjusted in some way for age.

192
00:18:26,644 --> 00:18:31,742
But it's important to bear that in mind when people are talking about changes on things like -

193
00:18:31,743 --> 00:18:38,029
the DORE program, they talk about changes on balance, balance improves dramatically after the program.

194
00:18:38,029 --> 00:18:43,498
These are measures that have not been adjusted for age at all, and so, some of these changes could well be due to the fact

195
00:18:43,499 --> 00:18:48,309
that the children are getting older and getting better at doing these things because of that.

196
00:18:48,310 --> 00:18:56,125
Another uninteresting reason why schools may improve is that the children may be having some other sort of special help.

197
00:18:56,126 --> 00:19:03,315
So, if a child is having reading difficulty, they may very well be getting some special help at the school, in addition to following this program.

198
00:19:03,315 --> 00:19:10,043
And that may be what's causing the change, rather than the particular intervention you are interested in.

199
00:19:10,044 --> 00:19:14,869
What's very well known, of course, is the placebo effect, which is a sort of concept coming from medicine,

200
00:19:14,870 --> 00:19:20,641
which also says that you can get better just because you think you are going to get better, because you think somebody has done something effective.

201
00:19:20,642 --> 00:19:25,611
And in the case of educational treatments, you can see effects where -

202
00:19:25,612 --> 00:19:31,881
because the teachers and the parents and the children themselves are all full of expectations of how this is going to improve them -

203
00:19:31,882 --> 00:19:38,911
there is more motivation: everybody gets positive attention and this itself can cause positive effects.

204
00:19:40,150 --> 00:19:45,830
The fourth reason, which is often neglected, because it really doesn't affect things in medicine so much,

205
00:19:45,831 --> 00:19:50,071
but in education, it's actually rather important, using the sort of thing like reading tests:

206
00:19:50,072 --> 00:19:55,595
you can have practice effects. So you can get better upon some things, just because you've done it before.

207
00:19:55,596 --> 00:20:00,521
And we've seen this quite a lot with language tests, for example, that we give to children,

208
00:20:00,522 --> 00:20:05,695
where, the first time you test a child, they don't know what to expect, they don't know what's coming,

209
00:20:05,696 --> 00:20:09,607
you aske them to do something that's unfamiliar and they are a little bit nervous, maybe.

210
00:20:09,607 --> 00:20:16,122
You test them again on the same thing a month later: they are much, much better, simply because they've done it before

211
00:20:16,122 --> 00:20:19,894
and they are calmer about it, they know what to expect, and so on.

212
00:20:19,894 --> 00:20:26,122
So you can get practice effects that can make quite a difference, just because you know what to expect

213
00:20:26,122 --> 00:20:30,402
and you are familiar with the whole situation of the test.

214
00:20:30,403 --> 00:20:37,856
The fifth reason - and the last one, you'll be pleased to hear - why people may improve for no good reason

215
00:20:37,857 --> 00:20:42,753
is the hardest to explain and it's something known as regression to the mean,

216
00:20:42,754 --> 00:20:49,400
and it's just a statistical artefact, which has to do with, if you pick somebody because they're bad at something,

217
00:20:49,401 --> 00:20:54,259
the odds are, when you test them on a second occasion, they'll be a little bit better (?).

218
00:20:54,260 --> 00:20:59,498
The converse is also true: if you pick somebody who is very good, they tend to get a little bit worse when you test them a second time.

219
00:20:59,499 --> 00:21:08,743
Why should that be? The reason why this occurs is because our measures are not entirely perfect an accurate -

220
00:21:08,744 --> 00:21:13,444
I'm showing a graph here, where we have a measure that is almost perfect,

221
00:21:13,445 --> 00:21:21,953
and you test people on two occasions, and you just will see that their scores on time 1 and time 2 are identical: we are assuming that there is no genune change.

222
00:21:21,954 --> 00:21:26,963
If you do that, then you don't get regression to the mean, because the measure is perfect

223
00:21:26,964 --> 00:21:31,039
and if you test them a second time, they'll get exactly the same sort of score.

224
00:21:31,040 --> 00:21:39,782
And what you can see on the right hand side of the graph here, is people divided up according to the average score they started with.

225
00:21:39,782 --> 00:21:45,397
So we've put people into groups who were very poor to start with, who were medium, less good and so on.

226
00:21:45,397 --> 00:21:49,491
And these are just fictitious data made up to illustrate the point.

227
00:21:49,492 --> 00:21:56,420
So you just generate these numbers by saying, "We've got a measure that has this particular characteristic

228
00:21:56,420 --> 00:22:01,095
that if you measure on one occasion, on another occasion it remains pretty much the same".

229
00:22:01,096 --> 00:22:06,777
So then, you don't get regression to mean and you get people to maintain their position across time.

230
00:22:06,778 --> 00:22:10,011
So if you then see change, you can say "Well, it's genuine change."

231
00:22:10,011 --> 00:22:15,591
But most of our measures are not like that, most of our measures are not perfectly correlated:

232
00:22:15,591 --> 00:22:21,641
that means, you measure them on one time, and another time, and they actually change because of all sorts of things.

233
00:22:21,641 --> 00:22:29,725
Things like the particular test items that you're using, whether you are in a good mood, whether you've made a lucky guess in some items.

234
00:22:29,726 --> 00:22:36,125
And what you can see is that if you do that, that some people's scores go up with time, some people's go down with time.

235
00:22:36,126 --> 00:22:44,983
But on average, if you start with a low score, the odds are, you come a little bit closer to the average when you are tested on another occasion.

236
00:22:44,983 --> 00:22:47,747
If you start with a high score, you get a little bit worse.

237
00:22:47,748 --> 00:22:53,566
And this is nothing to do with genuine change: it's just to do with the fact that our measures are imperfect.

238
00:22:53,567 --> 00:22:59,977
And it has been argued that this is a major reason - all sorts of treatments that work (?) but don't really work.

239
00:22:59,978 --> 00:23:04,401
It's just that it looks as if you've seen a change, and you tend to attribute it to the treatment.

240
00:23:04,401 --> 00:23:10,252
Now, this sounded very depressing, because it means there's all sorts of reasons why we can see change,

241
00:23:10,253 --> 00:23:14,912
and how do we distinguish whether we've got a genuine change due to our treatment?

242
00:23:14,912 --> 00:23:21,473
But the fact is that you can control for most of these things if you do a study that has a control group.

243
00:23:21,474 --> 00:23:30,969
That's why those who are trying to do scientific evaluations are really keen to include control groups in studies and argue that they are essential.

244
00:23:30,970 --> 00:23:37,428
Because if you have another group of children who have been selected to be as similar as possible to your treated group,

245
00:23:37,428 --> 00:23:42,122
and are the same tests before and after the period where the treated group are treated,

246
00:23:42,122 --> 00:23:48,540
you are actually controlling for the effects of maturation, the effects of any other intervention they might be having,

247
00:23:48,541 --> 00:23:53,250
practice effects in particular, and also this dreadful regression to the mean.

248
00:23:53,251 --> 00:23:56,351
All of those things can be then taken into account.

249
00:23:56,351 --> 00:24:02,283
And in so far as they have effects, what you would expect to see is that you may see improvement in your control group

250
00:24:03,212 --> 00:24:06,880
because of these spurious things that we don't really want to see.

251
00:24:06,880 --> 00:24:13,156
And then you can say, "well there is more improvement in the treated group" (?) and it is that difference that is really critical.

252
00:24:13,157 --> 00:24:20,931
It doesn't actually control to use - if you have a group who have not been given any treatment - it doesn't control for placebo effects.

253
00:24:20,931 --> 00:24:27,925
So you've still got the problem that maybe your treated group will improve just because everybody is focusing on them with great excitement.

254
00:24:27,925 --> 00:24:32,778
But you could actually also have control for that, and it's becoming increasingly popular in this field

255
00:24:32,779 --> 00:24:38,341
to say that what you should have is a control group who are actually given some alternative treatment.

256
00:24:38,342 --> 00:24:42,780
So, for example, if you are interested in a treatment that might improve reading,

257
00:24:42,780 --> 00:24:48,264
you could either get children some standard educational treatment that they are getting anyway

258
00:24:48,265 --> 00:24:53,582
So if your claim is that you are doing better than a phonological-based treatment,

259
00:24:53,582 --> 00:24:58,205
you could have a control group given that treatment and see if you are making really that much difference,

260
00:24:58,206 --> 00:25:03,407
or you might prefer to say, "Well, let's treat something else, let's give children training in something completely different

261
00:25:03,407 --> 00:25:06,862
that isn't focused on reading, but nevertheless could benefit them in other ways."

262
00:25:06,863 --> 00:25:10,104
And then you can do that sort of comparison.

263
00:25:10,105 --> 00:25:18,027
So what about the DORE study, because I mentioned at the outset, when talking about this study, that they did have a control group.

264
00:25:18,027 --> 00:25:23,981
But so far, talking about the results, are only mentioned (?) the dramatic changes that they saw,

265
00:25:23,982 --> 00:25:27,122
which ignored the control group.

266
00:25:27,123 --> 00:25:34,593
The interesting thing is that when you look at their control group, it illustrates perfectly the importance of having a control group.

267
00:25:34,593 --> 00:25:39,806
So, on they dylexia risk's score, where a high score is bad,

268
00:25:39,806 --> 00:25:47,228
they had a change in the treated group, from 0.74 to 0.34.

269
00:25:47,228 --> 00:25:51,060
So you think: "Wow, that's great, these children's risks for dyslexia have really come down."

270
00:25:51,061 --> 00:25:56,440
In the control group, the average score changed from 0.72 to 0.44.

271
00:25:56,440 --> 00:25:59,037
Now, you could say: "Well, it's not so big a change."

272
00:25:59,038 --> 00:26:03,849
The trouble is, with groups this size, you can't really tell whether that's meaningful.

273
00:26:03,850 --> 00:26:07,967
But certainly, what is clear is that both groups improved on the dyslexia screening test,

274
00:26:07,968 --> 00:26:12,089
even though the control group had not had the intervention.

275
00:26:12,090 --> 00:26:16,607
So, it really illustrates the point very clearly that on a lot of these measures,

276
00:26:16,608 --> 00:26:21,043
everybody gets better, even if they are not treated.

277
00:26:21,690 --> 00:26:30,834
Now, if we look at the more precise data that they presented, they presented average scores on the different subtests from the dyslexia screening tests,

278
00:26:30,835 --> 00:26:36,090
I won't talk about all of them, I have got a fuller presentation

279
00:26:36,091 --> 00:26:39,408
where I do talk about all the different measures they use

280
00:26:39,409 --> 00:26:42,662
and I don't want to sort of be accused of delberately hiding things,

281
00:26:42,663 --> 00:26:46,291
but I think the tests that people would be most interested in are the literacy tests.

282
00:26:46,291 --> 00:26:52,752
So, you undertake the DORE treatment because you want to get better at reading and writing, if you are a parent of a dyslexic child, at any rate.

283
00:26:52,752 --> 00:27:03,100
So, looking at the results on those tests, what they found was that there were a total of 4 tests that had to do with literacy directly.

284
00:27:03,101 --> 00:27:07,784
And on one of those, it looked as if the treated group did better than the untreated group.

285
00:27:07,785 --> 00:27:13,853
But there is a problem with that, though, because on this reading test, the untreated -

286
00:27:13,854 --> 00:27:20,498
the control group are actually right on the average score for their age at the start of the treatment - at the start of the study.

287
00:27:20,498 --> 00:27:27,497
So, in a sense, you could argue, "Is there really room for improvement ...... (?) school absolutely average,

288
00:27:27,498 --> 00:27:35,425
whereas it just so happened that the children who had treatment started a little bit lower and therefore had more improvement.

289
00:27:35,426 --> 00:27:38,544
And their improvement was not dramatic, one has to say as well.

290
00:27:38,544 --> 00:27:43,222
Their school went up from 3 to 3.5, on a scale of 0 to 10.

291
00:27:43,222 --> 00:27:51,733
On the other measures, again it illustrated that on two of them, everybody improved, regardless of whether they had the treatment.

292
00:27:51,733 --> 00:27:54,896
And on the third one, nobody changed very much at all.

293
00:27:54,896 --> 00:28:03,226
So, this is not dramatic evidence of improvement but you could argue: "Well, nevertheless there was one measure that looked a little bit promising."

294
00:28:03,227 --> 00:28:11,782
But they then, in the second phase of the study, went on to give the control group the same treatment, and they published this in 2007.

295
00:28:11,782 --> 00:28:16,456
So we now don't anymore have a control group as everybody has been treated:

296
00:28:16,456 --> 00:28:21,587
one group early on, and the other group with a delayed time scale.

297
00:28:21,587 --> 00:28:30,068
And they presented the data between time 1 at the start of the study and right at the end of the study, when everybody had had this treatment.

298
00:28:30,069 --> 00:28:37,502
But when you look at the results there, it's clear that there really is a, you know, no persistent improvement in reading.

299
00:28:37,502 --> 00:28:41,723
In fact, the mean scores for the children having the delayed treatment on the reading test

300
00:28:41,724 --> 00:28:45,453
have now really gone down, rather than up, at the end of treatment.

301
00:28:45,454 --> 00:28:51,185
And the general impression, I would say, is that there is nothing very stunning going on here,

302
00:28:51,186 --> 00:28:57,615
certainly nothing that matches the description that you get on the promotion materials for the intervention.

303
00:28:57,616 --> 00:29:08,095
So, overall, I would argue that the evidence for gains associated with this treatment is really not at all compelling.

304
00:29:08,096 --> 00:29:17,265
First of all, the claims that are made for stunning changes are all coming from analyses where they didn't incorporate the controls

305
00:29:17,265 --> 00:29:21,820
and they just tried to argue that any change you see at the time must be due to the treatment.

306
00:29:21,820 --> 00:29:25,091
and not taking into account all these other factors.

307
00:29:25,092 --> 00:29:30,355
And on reading measures, where there was control group data available,

308
00:29:30,356 --> 00:29:37,098
there was an initial small gain in the treated group, but it wasn't sustained by the end of the study.

309
00:29:37,099 --> 00:29:41,359
So, it really doesn't look terribly promising.

310
00:29:42,420 --> 00:29:46,989
Now, this is why in general, I think it's true to say this:

311
00:29:46,989 --> 00:29:52,641
I don't know of anybody in the dyslexic community who is an advocate - in the academic community

312
00:29:52,641 --> 00:29:57,753
who is an advocate of the DORE treatment, other than people that are directly associated with the DORE organization.

313
00:29:57,754 --> 00:30:05,466
And so, the reason really is just that the evidence is not at all compelling,

314
00:30:05,466 --> 00:30:09,184
although the study was small and you could argue a larger study should be done.

315
00:30:09,184 --> 00:30:14,779
There is a real mismatch between the claims that are being made and the evidence that is available.

316
00:30:14,780 --> 00:30:21,717
But the interesting thing is also why so many people seem to nevertheless regard this as an effective treatment.

317
00:30:21,717 --> 00:30:29,504
If the testimonials are to be believed, there are many satisfied customers and happy parents who feel that their children have been helped.

318
00:30:29,505 --> 00:30:35,596
I think there is quite an interesting set of reasons why this may be so.

319
00:30:35,597 --> 00:30:41,405
And one is that there is a well known - in the psychological field - well known human tendency

320
00:30:41,406 --> 00:30:47,147
to think that something that you've put in a lot of time and money too, was worthwhile.

321
00:30:47,148 --> 00:30:55,566
It's called cognitive dissonance, and it means that if you've actually put in the effort, you tend to feel that there was an effect.

322
00:30:55,567 --> 00:31:01,531
You have to somehow resolve this sort of inconsistency, otherwise, in your mind.

323
00:31:01,532 --> 00:31:06,852
And this was beautifully illustrated, not by the trial of the DORE treatment, but in another trial,

324
00:31:06,852 --> 00:31:11,242
which was a very nicely well-conducted trial of something called Sunflower therapy,

325
00:31:11,243 --> 00:31:19,165
which is a rather holistic approach to intervention for dyslexia that involves kinesiology and physical manipulation,

326
00:31:19,166 --> 00:31:23,499
massage, homeopathy, herbal remedies and neurolingusitc programming.

327
00:31:23,500 --> 00:31:30,274
And there was a very rigorous study done for this, and what was interesting about it

328
00:31:30,275 --> 00:31:39,296
was that, like so many of these things, they didn't really find a lot of evidence for any better change in the clinical versus the control group,

329
00:31:39,297 --> 00:31:43,357
although, to some extent, both groups were securing (?) their schools were improving.

330
00:31:43,867 --> 00:31:49,837
What they did find, though, is that the children themselves had higher self-esteem if they had undergone the Sunflower treatment,

331
00:31:49,838 --> 00:31:56,296
but that also, 57% of the parents did think that Sunflower therapy was effective in treating their child.

332
00:31:56,955 --> 00:32:04,260
So there is a clear mismatch between what the study showed of the objective evidence on the children's learning difficulties,

333
00:32:04,261 --> 00:32:06,682
and what the parents actually thought.

334
00:32:06,683 --> 00:32:12,230
It is possible that this could be related to the fact that the children's scores did improve,

335
00:32:12,231 --> 00:32:17,595
but if you didn't know that the control children had also improved, you might attribute that to the therapy -

336
00:32:17,595 --> 00:32:23,572
but also to the fact that people were again being given a lot of encouragement,

337
00:32:23,573 --> 00:32:29,845
there was a lot invested in that treatment, and then there might well (?) have been some sort of sense of cognitive dissonance there.

338
00:32:29,846 --> 00:32:41,088
There's also a strong human tendency to be impressed by certain kinds of explanation that get more biological about dyslexia,

339
00:32:41,088 --> 00:32:48,683
particularly those such as the DORE treatment that get more neurological and claim to be doing something to the brain in treating dyslexia.

340
00:32:48,684 --> 00:32:54,007
There was a beautiful study done - published - in 2008, not on dyslexia,

341
00:32:54,007 --> 00:32:59,816
but just more generally on people's tendency to be impressed by scientific eyplanations.

342
00:32:59,817 --> 00:33:09,268
and what these researchers did was to give people explanations of psychological phenomena that are well known

343
00:33:09,269 --> 00:33:14,108
and they either gave them a good explanation or they gave them a not very good explanation

344
00:33:14,109 --> 00:33:21,341
that was more like just a re-description of the effect, and asked people to judge whether this was a good explanation or not.

345
00:33:21,342 --> 00:33:25,603
And what was fascinating about this study was that in general, people were quite good at doing this:

346
00:33:25,604 --> 00:33:33,309
even if they had no familiarity or background in psychology, they could distinguish a good explanation from a bad one.

347
00:33:33,309 --> 00:33:39,629
But what they found was that if they added some verbiage that just talked about the brain in various ways,

348
00:33:39,629 --> 00:33:45,245
and said, "This result came about because brain scans showed it", or "because we looked at the frontal lobes",

349
00:33:45,246 --> 00:33:48,371
people were much more impressed with the bad explanations.

350
00:33:48,371 --> 00:33:53,128
So a good explanation didn't get any better when you added all this neuroscience waffle,

351
00:33:53,129 --> 00:33:59,340
but if you added neuroscience waffle to a bad explanation, people thought it not so bad.

352
00:33:59,341 --> 00:34:05,954
And so, there is a tendency to be very impressed by anything that talks about, adds the brain in to an ........ (?) explanation.

353
00:34:05,955 --> 00:34:12,994
And I think this is used by people who then try and add spurious neuroscience sometimes

354
00:34:12,995 --> 00:34:18,171
to their accounts of their particular promissing theory.

355
00:34:18,172 --> 00:34:23,808
And it really is not - we shouldn't allow ourselves to be mislead.

356
00:34:24,541 --> 00:34:34,228
So I think, to sum up, there are a number of barriers to objective evaluation of intelligence,

357
00:34:34,229 --> 00:34:38,309
which, to some extent, functions about (?) our human condition,

358
00:34:38,309 --> 00:34:44,095
that we are not naturally good at taking in lots of numbers and looking at graphs

359
00:34:44,096 --> 00:34:47,758
and trying to sort of take into account alternative explanations.

360
00:34:47,759 --> 00:34:52,649
We tend to be impressed when we hear other people tell us that something has worked

361
00:34:52,649 --> 00:34:57,183
and it's hard - you have to almost guard yourself against the tendency to do that

362
00:34:57,184 --> 00:35:02,389
and to look rather for the hard evidence, to look for the actual numerical data.

363
00:35:03,249 --> 00:35:08,301
We have to be very careful when people start giving us explanations that have got a lot of neuroscience in them

364
00:35:08,301 --> 00:35:12,194
and check out, is this real neuroscience or is it just put in there to impress us?

365
00:35:13,808 --> 00:35:19,508
We have to be aware of the effect of cognitive dissonance and the tendency to believe some things

366
00:35:19,509 --> 00:35:23,866
simply because we have invested time and money in it

367
00:35:23,867 --> 00:35:31,401
and, most importantly, we have to bear in mind that there will be effects on children's performance

368
00:35:31,401 --> 00:35:37,974
of maturation, of our expectations, of just get practising on things,

369
00:35:37,975 --> 00:35:42,705
and there are also these dreadful statistical artefacts that can make it look as if a change has occurred,

370
00:35:42,706 --> 00:35:45,838
when it's really not particularly impressive.

371
00:35:45,838 --> 00:35:50,363
But I think, if one bears these things in mind, the bottom line is really:

372
00:35:50,364 --> 00:35:54,575
"Look for evidence from studies that have got adequate controls"

373
00:35:54,576 --> 00:36:01,508
and if you do, you'll be in - a standard, I think, by how far you can see improvements in children, even if they haven't had the treatment.

374
00:36:01,508 --> 00:36:06,140
and there are lots of things that will make things a lot better, just with the passage of time.

375
00:36:06,141 --> 00:36:11,339
But if you really want to demonstrate that there has been an effective treatment,

376
00:36:11,339 --> 00:36:15,804
you do have to show an improvement relative to a control group,

377
00:36:15,804 --> 00:36:20,955
rather than just, somebody started out not so good and is now a little bit better after the treatment.

378
00:36:20,956 --> 00:36:29,953
I hope that that might give you some useful indicators when trying to look at new treatments that are out there and on offer,

379
00:36:29,954 --> 00:36:35,111
and for a more detailed account of some of this work, there are various -

380
00:36:35,112 --> 00:36:40,104
there is a powerpoint presentation with notes on my website on this topic.