0:00:00.001,0:00:06.057 [Dorothy Bishop] I'm going to talk today about evaluating alternative intervention approaches to dyslexia. 0:00:06.059,0:00:07.462 [slide with talk title] 0:00:07.463,0:00:13.667 The conventional approaches that you get really tend to rely on the fact that we've now got of evidence 0:00:13.668,0:00:18.409 that most children with dyslexia have problems in what is called phonological awareness, 0:00:18.410,0:00:23.440 that is, they don't necessarily hear all the different sounds in speech, 0:00:23.441,0:00:28.380 and therefore have difficulty relating them to letters when they are trying to read. 0:00:28.381,0:00:37.516 And most of the interventions that are mainstream these days would focus on trying to train children to identify sounds in words and relate them to letters. 0:00:38.971,0:00:45.414 But this sort of intervention has been shown to be effective, and there have been a number of large scale studies. 0:00:45.415,0:00:51.725 But nevertheless, it has to be fairly prolonged for some children, and there are children for whom, 0:00:51.725,0:00:57.870 even though they can learn this way to actually sound out words and read, they don't necessarily read fluently. 0:00:57.912,0:01:03.306 It's still an effort for them, and they don't sort of get to the degree of automaticity that you might expect, 0:01:03.307,0:01:07.683 And it's certainly the case that methods that work for many children don't work for all children, 0:01:07.684,0:01:10.756 and there is a hard core of children who remain very hard to treat. 0:01:10.757,0:01:15.943 It is for this reason that many parents do get very concerned about whether there is something else they should be doing 0:01:15.944,0:01:21.997 if they are finding that their child is either not getting intervention, or that the intervention doesn't seem to be working very well. 0:01:21.998,0:01:29.446 And there are a whole load of things out there that are on offer, and the problem for the parents, I think - 0:01:29.447,0:01:33.982 and/or indeed for adults who themselves, might want to have further intervention for dyslexia - 0:01:33.983,0:01:37.691 is that they want to know, "how do I distinguish something that might work for me", 0:01:37.691,0:01:41.641 from something that is just some sort of snake-oil merchant who is out there to make money. 0:01:41.642,0:01:44.482 And that's what I want to try and address today. 0:01:44.482,0:01:51.544 principally from the perspective of how you might evaluate scientific evidence that people put forward. 0:01:51.544,0:01:57.486 But perhaps before going onto that, it's worth going into some relatively commonsense things. 0:01:57.618,0:02:06.612 I would say that there are certain things that should ring alarm bells if people are advertising some sort of treatment for child dyslexia. 0:02:06.649,0:02:13.064 The first thing is if the intervention has been developped by somebody who has no academic track record, 0:02:13.065,0:02:17.834 no experience of doing research in this field, and hasn't published anything in this field, 0:02:17.834,0:02:24.541 if the intervention isn't endorsed by people in the mainstream dyslexia field, 0:02:24.542,0:02:27.633 that should also sound a note of caution. 0:02:27.634,0:02:29.820 Of course, the mainstream people aren't always right. 0:02:29.821,0:02:32.971 It's possible that somebody with no background will develop something marvelous. 0:02:32.972,0:02:37.474 But if that were the case, you would expect it to be pretty quickly picked up by people in the mainstream, 0:02:37.475,0:02:40.798 who are really, on the whole, pretty keen to find things that will work. 0:02:40.798,0:02:47.811 And you obviously want to look at whether somebody is asking for a lot of money for something that hasn't been proven. 0:02:47.812,0:02:56.216 And what is also, to my mind, a worrying sign, is if somebody promoting a treatment is relying heavily just on testimonials 0:02:56.217,0:03:04.154 from individuals who claimed to have been cured, rather than having any sort of proper scientific evaluation or kind of controls. 0:03:04.155,0:03:10.613 And it's worth noting that human beings have a tendency to be terribly impressed by testimonials, 0:03:10.613,0:03:16.749 and even myself, as somebody with a scientific training, I find if, you know, I've got headaches and somebody comes along and says: 0:03:16.750,0:03:20.603 "I was cured by such and such, and I went to my herbalist and it worked", 0:03:20.603,0:03:27.094 you know, you're often very tempted to be much more swayed by that sort of evidence than by a pagefull of numbers and figures. 0:03:27.095,0:03:34.677 And this just a human tendency: we are naturally built to really take advice from other people and to rely on what they tell us. 0:03:34.677,0:03:38.494 But in the contexts of these sorts of interventions, that's really quite dangerous, 0:03:38.494,0:03:44.271 because, when somebody gives a testimonial, that's just one person, their only individual experience, 0:03:44.272,0:03:48.273 And people you don't hear from tend to be the people who tried it, and it didn't work. 0:03:48.274,0:03:51.881 And you don't know how many of them there are: there may be thousands of them. 0:03:51.882,0:03:56.049 But they're not going to publicize the fact that they tried it and it didn't work. 0:03:56.050,0:04:01.945 And so, testimonials are often very much at odds with more scientific evaluations. 0:04:01.945,0:04:10.096 .... to turn out that when somebody says there is scientific evidence for what they're doing, how you should interpret that. 0:04:10.097,0:04:14.964 And that's jolly difficult even for scientists sometimes: there is disagreement - let alone for the general public. 0:04:14.965,0:04:20.727 But again, I think, there are some sort of general rules of thumb that you can go by 0:04:20.728,0:04:24.020 for telling that a treatment is likely to be effective. 0:04:25.450,0:04:38.229 And when I discuss this, I'm going to illustrate it by taking the example of the Dore treatment - that's DORE, named after Wynford Dore, it's originator. 0:04:38.230,0:04:46.360 ANd I'm picking on this largely because it is a non-mainstream treatment that isn't widely accepted by the experts, 0:04:46.361,0:04:49.899 and yet it does claim that there is some scientific evidence to support it, 0:04:49.900,0:04:54.277 which has lead the scientists to look at it quite critically and quite carefully, 0:04:54.277,0:04:57.654 which is what we would do with any scientific evidence that comes along: 0:04:57.655,0:05:04.834 once it's out in the public domain and published, people tend to go and look at it as carefully as they possibly can. 0:05:04.834,0:05:11.944 Now, the Dore method is interesting to us, because it does illustrate the case where there is disagreement 0:05:11.945,0:05:15.197 as to whether the evidence is showing that its' effective or not. 0:05:15.198,0:05:20.587 And so, what I want to explain is why it is the case that despite this published evidence, 0:05:20.588,0:05:25.324 most of the experts are not impressed of the efficacy of the Dore treatment. 0:05:25.325,0:05:32.227 But the general points that I'll make would apply to any other treatment that was out there, whether (?) there was evidence being produced. 0:05:32.751,0:05:39.961 So, first of all, what is the Dore method? Well, it's a method that has been proposed for curing problems 0:05:39.962,0:05:44.774 that are thought to originate in the part of the brain called the cerebellum, which is at the back of the brain, 0:05:44.774,0:05:49.999 and it was developed by Wynford Dore as a method for helping his dyslexic daughter. 0:05:49.999,0:05:53.578 He has written a book about the history of how this came to became (?) about, 0:05:53.579,0:05:58.973 and he was a classic instance of a parent who was rather desperate to help his daughter who, for many years, 0:05:58.973,0:06:03.513 had been through the educational system and failed, and was getting increasingly depressed. 0:06:03.514,0:06:10.842 And he tried various things, he talked to various experts, and he ended up with this program that's been put forward, 0:06:10.842,0:06:15.541 which is an individualized program, where the child follows various sorts of exercises, 0:06:15.542,0:06:21.019 which are done for about ten minutes twice a day, over quite a long period of time, 0:06:21.019,0:06:25.265 varying, depending on the severity of the problem, from maybe 6 months to 2 years. 0:06:25.928,0:06:30.984 And the child is assessed at regular intervals and different exercises may be prescribed. 0:06:30.985,0:06:38.550 Now, the theory behind the Dore method is that dyslexia and other learning difficulties - 0:06:38.550,0:06:43.661 it's not just dyslexia it claims to help, but also Attention Deficit problems ...(?) hyperactive - 0:06:43.661,0:06:49.043 are thought to arise within the cerebellum: the cerebellum just doesn't develop normally, 0:06:49.043,0:06:54.474 and the argument is that you can have different cerebellar impediments in different people, 0:06:54.475,0:06:57.198 and that's why you can get this range of different symptoms, 0:06:57.199,0:07:04.061 but that you can diagnose them by specific tests of test of mental and physical coordinations. 0:07:04.062,0:07:12.679 And what you are then supposed to do is these exercises, which are not anywhere fully described in the public domain, 0:07:12.679,0:07:17.623 because they are commercially sensitive, but there are some examples given, and it's clear that what they do 0:07:17.623,0:07:22.714 is focused largely on training balance and hand-eye coordination in children. 0:07:22.715,0:07:29.890 So you might be asked to stand on a cushion on one leg, or to throw a bean bag from one hand to another 0:07:29.891,0:07:37.296 while you are doing that, just stand on a wobble board (?) and balance, or to follow something with your eyes in a particular way. 0:07:39.188,0:07:48.267 So, the idea is that these are all things that the cerebellum is involved in, by training up the cerebellum, you may improve its general abilities. 0:07:48.727,0:07:53.435 So, what is the evidence for this underlying theory? 0:07:53.436,0:07:58.405 Well, it's not a proven theory, but there is some support for it. 0:07:58.405,0:08:06.920 Certainly, people trying to look at what is going on in the brain in dyslexia have proposed many different theories 0:08:06.921,0:08:09.314 about what the underlying causes might be. 0:08:09.315,0:08:14.735 If you look at the brain in a brain scanner of somebody with dyslexia, it typically looks totally normal. 0:08:14.735,0:08:19.256 There's certainly no big holes in the head or anything like that, that you are going to see on a scanner. 0:08:19.257,0:08:24.865 But the argument is being made that there may be regions of the cerebellum that are perhaps slightly smaller than they should be 0:08:24.866,0:08:28.046 or not functioning quite as they should be. 0:08:28.047,0:08:33.820 And this theory has some support, although not everybody would agree with it 0:08:33.820,0:08:38.843 and there is certainly other theories equally plausible at the moment that are around. 0:08:39.444,0:08:44.664 The notion - the cerebellum is important for getting things automated. 0:08:44.665,0:08:50.675 So you can - when you learn to drive a car, first of all, it's very slow and effortfull, and you have to think about everything you do. 0:08:50.675,0:08:55.185 By the time you are a skilled driver, it's no longe the case that you have to do that, 0:08:55.186,0:08:58.061 you just drive around without thinking about it. 0:08:58.061,0:09:00.063 You can do all sorts of other things while you are driving. 0:09:00.064,0:09:07.321 So, the argument is that with reading, most people, similarly, become very automatic in how they learn to read: 0:09:07.322,0:09:14.902 you do it without thinking about it, but for the dylexic it remains effortfull because the cerebellum is not functioning normally 0:09:14.902,0:09:19.387 and it's the cerebellum that helps you get your skills automatized. 0:09:19.388,0:09:26.874 And in support of this, it has been argued that in many people with dyslexia, there are some associated problems with motor coordination, 0:09:26.875,0:09:32.792 ..... (?) physical skills and so on, and that too could be a sign of a problem with the cerebellum. 0:09:32.793,0:09:39.876 Again, that's fairly controversial, it's not being found in all children, and the arguments go to and fro. 0:09:39.876,0:09:47.492 But this is not a sort of theory that is particularly disapproved of by the mainstream. People are debating it. 0:09:47.493,0:09:55.052 The difficult stumbling block, though, for the Dore approach to treatment comes with the idea that 0:09:55.053,0:10:02.531 if you train the motor skills, that is a sort of coordination between different muscles and movements 0:10:02.532,0:10:09.374 and between their eyes and hands, that this will somehow have a knock-on effect on things like reading. 0:10:09.375,0:10:16.401 And indeed, David Randall and colleagues, who published this initial study on the treatment, 0:10:16.401,0:10:23.410 describe it as something of a leap of faith, because the cerebellum is actually known to be a very complicated organ, 0:10:23.411,0:10:27.249 with lots of different regions, which are fairly independent from one another. 0:10:27.250,0:10:34.590 So there is no real reason to suppose that if you train one part of the cerebellum, it will have somehow a generalized benefit. 0:10:34.591,0:10:40.750 And indeed, you could say: "Well, if it were the case that this is true, if you'd had a chance, you would go to skateboarding, 0:10:40.751,0:10:48.710 or playing ping-pong, or things like that, ....... (?) or perhaps ballet dancing, things that require balance and coordination, 0:10:48.711,0:10:52.653 that should protect you against dyslexia". There is really not much evidence for that, 0:10:52.654,0:10:57.645 on the contrary, there is some very good sportsmen who - gymnasts and people with dyslexia. 0:10:57.645,0:11:06.790 So it is hard to see how the logic of saying "Train these motor skills and somehow the whole cerebellum function some day improves2 0:11:06.790,0:11:12.876 But what does the published evidence look like? Because the theory might be, you know, questionable, 0:11:12.877,0:11:16.956 but basically, what the parents are going to say is, "What matter is, does it work?" 0:11:18.215,0:11:24.717 Well, there is a published study on the intervention, which claims that it shows that it really does work 0:11:24.717,0:11:28.297 if you compare children who have the intervention and children who don't. 0:11:28.298,0:11:35.999 And two papers have been reported - one from the initial phase of the study, and the other from a subsequent phase - 0:11:36.866,0:11:45.538 And they are reported in the Journal of Dyslexia which, in 2003, published the first paper 0:11:45.663,0:11:53.890 which was on just under - started with a sample statistics on the 300 children who were all attending a .... (?) primary school. 0:11:53.890,0:11:58.021 And the researchers went in and screened all the children on the dyslexia screening test, 0:11:58.021,0:12:01.457 to pull out children who would be suitable for enrolment in the study. 0:12:01.458,0:12:10.647 But the first thing that is more important to note is that these were not children who had a very high rate of diagnosis of dyslexia. 0:12:10.648,0:12:19.265 So, there were 35 in the group, and about a third of those came out as having a strong risk of dyslexia on this dyslexia screening test. 0:12:19.265,0:12:26.744 Another 21% came out with a mild risk, but about half of these children were not really ...... (?) in this category 0:12:26.745,0:12:31.496 and they were just picked because their schools (?) were relatively lower compared to the other children. 0:12:31.497,0:12:38.199 And there were only a total of 6 children who had previously been diagnosed with dyslexia, out of the 35. 0:12:38.200,0:12:42.989 There were a couple with a diagnosis of dyspraxia and one with ADHD diagnosis. 0:12:42.990,0:12:48.464 So this is not really a sample consisting of children really with severe problems on the whole. 0:12:48.464,0:12:50.660 There were few in there with major difficulties. 0:12:50.661,0:12:58.037 Nevertheless, the originators of the treatment would argue even quite mild problems might be worth treating with this 0:12:58.037,0:13:01.890 and so you could argue this study is nevertheless of value. 0:13:01.891,0:13:07.834 So what they did, they started out well in this study: they divided the children randomly in treated and untreated groups, 0:13:07.834,0:13:13.687 which is, as I am going on to explain later, is an important part of a good study. 0:13:13.687,0:13:20.727 And if you look at the results that are described on the promotion materials of the DORE organization, 0:13:20.727,0:13:28.476 they are all in Dore's book that he published, "Dyslexia, the miracle cure", he described the results as stunning 0:13:28.477,0:13:35.039 and said that reading age increased threefold, comprehension age increased almost fivefold 0:13:35.040,0:13:40.640 and writing skills by what he described as "an extraordinary 17-fold". 0:13:40.641,0:13:47.856 Of course, everybody reading that think "Wow, my child is going to take off like a rocket if we put him on this intervention." 0:13:47.856,0:13:56.806 But unfortunately, these figures are really a classic instance of how statistics can be manipulated in a very misleading way. 0:13:57.414,0:14:04.912 So, for a start, they were not based on any comparison between the control children and the untreated children - 0:14:04.913,0:14:09.177 sorry, the control children and the treated children. 0:14:09.178,0:14:17.568 They were - instead, they just took all the children who would be treated and looked at how they did on a group reading test 0:14:17.569,0:14:21.044 that had been administered by the school every year. 0:14:21.045,0:14:30.727 And the children had had this on two occasions prior to the intervention - so, 3 months before it started and a year before that - 0:14:30.727,0:14:35.345 and on two occasions after the intervention, after this whole long 4-year period. 0:14:35.346,0:14:44.263 And what the researchers did was to really just plot the average schools of the group over these 4 time periods 0:14:44.264,0:14:49.639 and show that if you compared the amount of change from the first time point to the second, 0:14:49.640,0:14:54.394 which was before they had had any treatment, it was a certain amount 0:14:54.395,0:15:00.347 and if you then compared the second to the third time point, so the treatment had been going on (?) between those two, 0:15:00.347,0:15:02.663 there was a different amount of change. 0:15:02.664,0:15:06.672 And then they divided one by the other and showed that there was this threefold improvement. 0:15:06.673,0:15:13.301 But it's a very, very misleading way of depicting these data, because if you look at them on a graph, here, 0:15:13.301,0:15:18.168 you can see that the only odd thing about the data - well, there's two odd things about the data: 0:15:18.169,0:15:23.171 one is that at most time points, these children are reading at absolutely normal levels. 0:15:23.172,0:15:27.214 So it's not clear why they are regarded as having risk for dyslexia; 0:15:27.214,0:15:32.947 and the one time point when they're not, is the time point 3 months before they are involved in the study, 0:15:32.947,0:15:39.363 where there is a bit of a drop. But it's really not an impressive demonstration of change 0:15:39.364,0:15:46.794 and this division of one time period by another is very misleading, because it just gives double weighting 0:15:46.794,0:15:52.357 to this one low period of three months before the treatment started. 0:15:52.358,0:15:57.610 And they did the same thing again with these other figures of massive increases that they talk about, 0:15:57.611,0:16:06.830 using data from the SATS tests administered by teachers, which are not really regarded as particularly precise or rigorous tests, 0:16:06.831,0:16:14.025 and really group children in a fairly global way at level 2, 3 or 4. 0:16:14.026,0:16:20.494 Level 2 is average for 7 year old, 3 is average for 9 year old, and 4 is average for an 11 year old. 0:16:20.495,0:16:27.876 And to give you an idea of the sort of misleading nature of these massive changes they talk about, 0:16:27.876,0:16:34.125 on the writing test, where there is this incredible change that they talk about, of a 17-fold increase, 0:16:34.126,0:16:40.792 the score at age 8, the average score was 2.5, which is about what you'd expect from a 8 year old. 0:16:40.792,0:16:47.103 At age 9, it was 2.56, which is a little bit better, but not much. 0:16:47.104,0:16:53.486 And then, they argue, the intervention came in, and at age 10, the children scored 2.95. 0:16:53.486,0:16:56.888 They are still rather below where they ought to be at the age of 10. 0:16:56.888,0:17:01.372 It looks as if on this particular writing assessment, the children were just rather creeping along. 0:17:01.373,0:17:09.878 But because the difference between 2.53 and 2.56 is less than the difference between 2.56 and 2.95, 0:17:09.879,0:17:17.935 they make a big computation of dividing one by the other, actually coming out with the number 17, which is a wrong number (?): it's actually 13. 0:17:17.935,0:17:24.646 So there is a 13-fold change. But if you look at the overall numbers, this is really not so an impressive game at all. 0:17:24.647,0:17:27.761 It's really a very misleading way of presenting the numbers. 0:17:27.762,0:17:36.509 So, most people would say, this is really smoke in mirrors in terms of using statistics in a way that isn't really valid. 0:17:36.925,0:17:42.129 The other thing that is of notice is that all these results that have given so much publicity 0:17:42.130,0:17:47.697 in promoting the treatment about these massive changes, haven't talked about the control group at all. 0:17:47.697,0:17:51.428 They've just talked about, "Well, we've got these children, before treatment they did this, 0:17:51.428,0:17:53.800 and after treatment they did that, and it has all gone up". 0:17:53.801,0:17:59.717 And of course, if schools do go up after treatment, it's not necessarily because the treatment works: 0:17:59.717,0:18:02.299 There are lots of other reasons you need to bear in mind. 0:18:02.951,0:18:07.555 And the first of which is just, on some things, you get better because you get older, 0:18:07.556,0:18:13.635 so that if you were to measure shoe sizes before the DORE treatment and after it, it would go up, 0:18:13.636,0:18:16.663 but it wouldn't mean that it made your feet grow bigger. 0:18:16.664,0:18:24.339 Now, clearly, that's a silly example in most cases, because people try to use measures that don't necessarily change with age, 0:18:24.340,0:18:26.643 or that are adjusted in some way for age. 0:18:26.644,0:18:31.742 But it's important to bear that in mind when people are talking about changes on things like - 0:18:31.743,0:18:38.029 the DORE program, they talk about changes on balance, balance improves dramatically after the program. 0:18:38.029,0:18:43.498 These are measures that have not been adjusted for age at all, and so, some of these changes could well be due to the fact 0:18:43.499,0:18:48.309 that the children are getting older and getting better at doing these things because of that. 0:18:48.310,0:18:56.125 Another uninteresting reason why schools may improve is that the children may be having some other sort of special help. 0:18:56.126,0:19:03.315 So, if a child is having reading difficulty, they may very well be getting some special help at the school, in addition to following this program. 0:19:03.315,0:19:10.043 And that may be what's causing the change, rather than the particular intervention you are interested in. 0:19:10.044,0:19:14.869 What's very well known, of course, is the placebo effect, which is a sort of concept coming from medicine, 0:19:14.870,0:19:20.641 which also says that you can get better just because you think you are going to get better, because you think somebody has done something effective. 0:19:20.642,0:19:25.611 And in the case of educational treatments, you can see effects where - 0:19:25.612,0:19:31.881 because the teachers and the parents and the children themselves are all full of expectations of how this is going to improve them - 0:19:31.882,0:19:38.911 there is more motivation: everybody gets positive attention and this itself can cause positive effects. 0:19:40.150,0:19:45.830 The fourth reason, which is often neglected, because it really doesn't affect things in medicine so much, 0:19:45.831,0:19:50.071 but in education, it's actually rather important, using the sort of thing like reading tests: 0:19:50.072,0:19:55.595 you can have practice effects. So you can get better upon some things, just because you've done it before. 0:19:55.596,0:20:00.521 And we've seen this quite a lot with language tests, for example, that we give to children, 0:20:00.522,0:20:05.695 where, the first time you test a child, they don't know what to expect, they don't know what's coming, 0:20:05.696,0:20:09.607 you aske them to do something that's unfamiliar and they are a little bit nervous, maybe. 0:20:09.607,0:20:16.122 You test them again on the same thing a month later: they are much, much better, simply because they've done it before 0:20:16.122,0:20:19.894 and they are calmer about it, they know what to expect, and so on. 0:20:19.894,0:20:26.122 So you can get practice effects that can make quite a difference, just because you know what to expect 0:20:26.122,0:20:30.402 and you are familiar with the whole situation of the test. 0:20:30.403,0:20:37.856 The fifth reason - and the last one, you'll be pleased to hear - why people may improve for no good reason 0:20:37.857,0:20:42.753 is the hardest to explain and it's something known as regression to the mean, 0:20:42.754,0:20:49.400 and it's just a statistical artefact, which has to do with, if you pick somebody because they're bad at something, 0:20:49.401,0:20:54.259 the odds are, when you test them on a second occasion, they'll be a little bit better (?). 0:20:54.260,0:20:59.498 The converse is also true: if you pick somebody who is very good, they tend to get a little bit worse when you test them a second time. 0:20:59.499,0:21:08.743 Why should that be? The reason why this occurs is because our measures are not entirely perfect an accurate - 0:21:08.744,0:21:13.444 I'm showing a graph here, where we have a measure that is almost perfect, 0:21:13.445,0:21:21.953 and you test people on two occasions, and you just will see that their scores on time 1 and time 2 are identical: we are assuming that there is no genune change. 0:21:21.954,0:21:26.963 If you do that, then you don't get regression to the mean, because the measure is perfect 0:21:26.964,0:21:31.039 and if you test them a second time, they'll get exactly the same sort of score. 0:21:31.040,0:21:39.782 And what you can see on the right hand side of the graph here, is people divided up according to the average score they started with. 0:21:39.782,0:21:45.397 So we've put people into groups who were very poor to start with, who were medium, less good and so on. 0:21:45.397,0:21:49.491 And these are just fictitious data made up to illustrate the point. 0:21:49.492,0:21:56.420 So you just generate these numbers by saying, "We've got a measure that has this particular characteristic 0:21:56.420,0:22:01.095 that if you measure on one occasion, on another occasion it remains pretty much the same". 0:22:01.096,0:22:06.777 So then, you don't get regression to mean and you get people to maintain their position across time. 0:22:06.778,0:22:10.011 So if you then see change, you can say "Well, it's genuine change." 0:22:10.011,0:22:15.591 But most of our measures are not like that, most of our measures are not perfectly correlated: 0:22:15.591,0:22:21.641 that means, you measure them on one time, and another time, and they actually change because of all sorts of things. 0:22:21.641,0:22:29.725 Things like the particular test items that you're using, whether you are in a good mood, whether you've made a lucky guess in some items. 0:22:29.726,0:22:36.125 And what you can see is that if you do that, that some people's scores go up with time, some people's go down with time. 0:22:36.126,0:22:44.983 But on average, if you start with a low score, the odds are, you come a little bit closer to the average when you are tested on another occasion. 0:22:44.983,0:22:47.747 If you start with a high score, you get a little bit worse. 0:22:47.748,0:22:53.566 And this is nothing to do with genuine change: it's just to do with the fact that our measures are imperfect. 0:22:53.567,0:22:59.977 And it has been argued that this is a major reason - all sorts of treatments that work (?) but don't really work. 0:22:59.978,0:23:04.401 It's just that it looks as if you've seen a change, and you tend to attribute it to the treatment. 0:23:04.401,0:23:10.252 Now, this sounded very depressing, because it means there's all sorts of reasons why we can see change, 0:23:10.253,0:23:14.912 and how do we distinguish whether we've got a genuine change due to our treatment? 0:23:14.912,0:23:21.473 But the fact is that you can control for most of these things if you do a study that has a control group. 0:23:21.474,0:23:30.969 That's why those who are trying to do scientific evaluations are really keen to include control groups in studies and argue that they are essential. 0:23:30.970,0:23:37.428 Because if you have another group of children who have been selected to be as similar as possible to your treated group, 0:23:37.428,0:23:42.122 and are the same tests before and after the period where the treated group are treated, 0:23:42.122,0:23:48.540 you are actually controlling for the effects of maturation, the effects of any other intervention they might be having, 0:23:48.541,0:23:53.250 practice effects in particular, and also this dreadful regression to the mean. 0:23:53.251,0:23:56.351 All of those things can be then taken into account. 0:23:56.351,0:24:02.283 And in so far as they have effects, what you would expect to see is that you may see improvement in your control group 0:24:03.212,0:24:06.880 because of these spurious things that we don't really want to see. 0:24:06.880,0:24:13.156 And then you can say, "well there is more improvement in the treated group" (?) and it is that difference that is really critical. 0:24:13.157,0:24:20.931 It doesn't actually control to use - if you have a group who have not been given any treatment - it doesn't control for placebo effects. 0:24:20.931,0:24:27.925 So you've still got the problem that maybe your treated group will improve just because everybody is focusing on them with great excitement. 0:24:27.925,0:24:32.778 But you could actually also have control for that, and it's becoming increasingly popular in this field 0:24:32.779,0:24:38.341 to say that what you should have is a control group who are actually given some alternative treatment. 0:24:38.342,0:24:42.780 So, for example, if you are interested in a treatment that might improve reading, 0:24:42.780,0:24:48.264 you could either get children some standard educational treatment that they are getting anyway 0:24:48.265,0:24:53.582 So if your claim is that you are doing better than a phonological-based treatment, 0:24:53.582,0:24:58.205 you could have a control group given that treatment and see if you are making really that much difference, 0:24:58.206,0:25:03.407 or you might prefer to say, "Well, let's treat something else, let's give children training in something completely different 0:25:03.407,0:25:06.862 that isn't focused on reading, but nevertheless could benefit them in other ways." 0:25:06.863,0:25:10.104 And then you can do that sort of comparison. 0:25:10.105,0:25:18.027 So what about the DORE study, because I mentioned at the outset, when talking about this study, that they did have a control group. 0:25:18.027,0:25:23.981 But so far, talking about the results, are only mentioned (?) the dramatic changes that they saw, 0:25:23.982,0:25:27.122 which ignored the control group. 0:25:27.123,0:25:34.593 The interesting thing is that when you look at their control group, it illustrates perfectly the importance of having a control group. 0:25:34.593,0:25:39.806 So, on they dylexia risk's score, where a high score is bad, 0:25:39.806,0:25:47.228 they had a change in the treated group, from 0.74 to 0.34. 0:25:47.228,0:25:51.060 So you think: "Wow, that's great, these children's risks for dyslexia have really come down." 0:25:51.061,0:25:56.440 In the control group, the average score changed from 0.72 to 0.44. 0:25:56.440,0:25:59.037 Now, you could say: "Well, it's not so big a change." 0:25:59.038,0:26:03.849 The trouble is, with groups this size, you can't really tell whether that's meaningful. 0:26:03.850,0:26:07.967 But certainly, what is clear is that both groups improved on the dyslexia screening test, 0:26:07.968,0:26:12.089 even though the control group had not had the intervention. 0:26:12.090,0:26:16.607 So, it really illustrates the point very clearly that on a lot of these measures, 0:26:16.608,0:26:21.043 everybody gets better, even if they are not treated. 0:26:21.690,0:26:30.834 Now, if we look at the more precise data that they presented, they presented average scores on the different subtests from the dyslexia screening tests, 0:26:30.835,0:26:36.090 I won't talk about all of them, I have got a fuller presentation 0:26:36.091,0:26:39.408 where I do talk about all the different measures they use 0:26:39.409,0:26:42.662 and I don't want to sort of be accused of delberately hiding things, 0:26:42.663,0:26:46.291 but I think the tests that people would be most interested in are the literacy tests. 0:26:46.291,0:26:52.752 So, you undertake the DORE treatment because you want to get better at reading and writing, if you are a parent of a dyslexic child, at any rate. 0:26:52.752,0:27:03.100 So, looking at the results on those tests, what they found was that there were a total of 4 tests that had to do with literacy directly. 0:27:03.101,0:27:07.784 And on one of those, it looked as if the treated group did better than the untreated group. 0:27:07.785,0:27:13.853 But there is a problem with that, though, because on this reading test, the untreated - 0:27:13.854,0:27:20.498 the control group are actually right on the average score for their age at the start of the treatment - at the start of the study. 0:27:20.498,0:27:27.497 So, in a sense, you could argue, "Is there really room for improvement ...... (?) school absolutely average, 0:27:27.498,0:27:35.425 whereas it just so happened that the children who had treatment started a little bit lower and therefore had more improvement. 0:27:35.426,0:27:38.544 And their improvement was not dramatic, one has to say as well. 0:27:38.544,0:27:43.222 Their school went up from 3 to 3.5, on a scale of 0 to 10. 0:27:43.222,0:27:51.733 On the other measures, again it illustrated that on two of them, everybody improved, regardless of whether they had the treatment. 0:27:51.733,0:27:54.896 And on the third one, nobody changed very much at all. 0:27:54.896,0:28:03.226 So, this is not dramatic evidence of improvement but you could argue: "Well, nevertheless there was one measure that looked a little bit promising." 0:28:03.227,0:28:11.782 But they then, in the second phase of the study, went on to give the control group the same treatment, and they published this in 2007. 0:28:11.782,0:28:16.456 So we now don't anymore have a control group as everybody has been treated: 0:28:16.456,0:28:21.587 one group early on, and the other group with a delayed time scale. 0:28:21.587,0:28:30.068 And they presented the data between time 1 at the start of the study and right at the end of the study, when everybody had had this treatment. 0:28:30.069,0:28:37.502 But when you look at the results there, it's clear that there really is a, you know, no persistent improvement in reading. 0:28:37.502,0:28:41.723 In fact, the mean scores for the children having the delayed treatment on the reading test 0:28:41.724,0:28:45.453 have now really gone down, rather than up, at the end of treatment. 0:28:45.454,0:28:51.185 And the general impression, I would say, is that there is nothing very stunning going on here, 0:28:51.186,0:28:57.615 certainly nothing that matches the description that you get on the promotion materials for the intervention. 0:28:57.616,0:29:08.095 So, overall, I would argue that the evidence for gains associated with this treatment is really not at all compelling. 0:29:08.096,0:29:17.265 First of all, the claims that are made for stunning changes are all coming from analyses where they didn't incorporate the controls 0:29:17.265,0:29:21.820 and they just tried to argue that any change you see at the time must be due to the treatment. 0:29:21.820,0:29:25.091 and not taking into account all these other factors. 0:29:25.092,0:29:30.355 And on reading measures, where there was control group data available, 0:29:30.356,0:29:37.098 there was an initial small gain in the treated group, but it wasn't sustained by the end of the study. 0:29:37.099,0:29:41.359 So, it really doesn't look terribly promising. 0:29:42.420,0:29:46.989 Now, this is why in general, I think it's true to say this: 0:29:46.989,0:29:52.641 I don't know of anybody in the dyslexic community who is an advocate - in the academic community 0:29:52.641,0:29:57.753 who is an advocate of the DORE treatment, other than people that are directly associated with the DORE organization. 0:29:57.754,0:30:05.466 And so, the reason really is just that the evidence is not at all compelling, 0:30:05.466,0:30:09.184 although the study was small and you could argue a larger study should be done. 0:30:09.184,0:30:14.779 There is a real mismatch between the claims that are being made and the evidence that is available. 0:30:14.780,0:30:21.717 But the interesting thing is also why so many people seem to nevertheless regard this as an effective treatment. 0:30:21.717,0:30:29.504 If the testimonials are to be believed, there are many satisfied customers and happy parents who feel that their children have been helped. 0:30:29.505,0:30:35.596 I think there is quite an interesting set of reasons why this may be so. 0:30:35.597,0:30:41.405 And one is that there is a well known - in the psychological field - well known human tendency 0:30:41.406,0:30:47.147 to think that something that you've put in a lot of time and money too, was worthwhile. 0:30:47.148,0:30:55.566 It's called cognitive dissonance, and it means that if you've actually put in the effort, you tend to feel that there was an effect. 0:30:55.567,0:31:01.531 You have to somehow resolve this sort of inconsistency, otherwise, in your mind. 0:31:01.532,0:31:06.852 And this was beautifully illustrated, not by the trial of the DORE treatment, but in another trial, 0:31:06.852,0:31:11.242 which was a very nicely well-conducted trial of something called Sunflower therapy, 0:31:11.243,0:31:19.165 which is a rather holistic approach to intervention for dyslexia that involves kinesiology and physical manipulation, 0:31:19.166,0:31:23.499 massage, homeopathy, herbal remedies and neurolingusitc programming. 0:31:23.500,0:31:30.274 And there was a very rigorous study done for this, and what was interesting about it 0:31:30.275,0:31:39.296 was that, like so many of these things, they didn't really find a lot of evidence for any better change in the clinical versus the control group, 0:31:39.297,0:31:43.357 although, to some extent, both groups were securing (?) their schools were improving. 0:31:43.867,0:31:49.837 What they did find, though, is that the children themselves had higher self-esteem if they had undergone the Sunflower treatment, 0:31:49.838,0:31:56.296 but that also, 57% of the parents did think that Sunflower therapy was effective in treating their child. 0:31:56.955,0:32:04.260 So there is a clear mismatch between what the study showed of the objective evidence on the children's learning difficulties, 0:32:04.261,0:32:06.682 and what the parents actually thought. 0:32:06.683,0:32:12.230 It is possible that this could be related to the fact that the children's scores did improve, 0:32:12.231,0:32:17.595 but if you didn't know that the control children had also improved, you might attribute that to the therapy - 0:32:17.595,0:32:23.572 but also to the fact that people were again being given a lot of encouragement, 0:32:23.573,0:32:29.845 there was a lot invested in that treatment, and then there might well (?) have been some sort of sense of cognitive dissonance there. 0:32:29.846,0:32:41.088 There's also a strong human tendency to be impressed by certain kinds of explanation that get more biological about dyslexia, 0:32:41.088,0:32:48.683 particularly those such as the DORE treatment that get more neurological and claim to be doing something to the brain in treating dyslexia. 0:32:48.684,0:32:54.007 There was a beautiful study done - published - in 2008, not on dyslexia, 0:32:54.007,0:32:59.816 but just more generally on people's tendency to be impressed by scientific eyplanations. 0:32:59.817,0:33:09.268 and what these researchers did was to give people explanations of psychological phenomena that are well known 0:33:09.269,0:33:14.108 and they either gave them a good explanation or they gave them a not very good explanation 0:33:14.109,0:33:21.341 that was more like just a re-description of the effect, and asked people to judge whether this was a good explanation or not. 0:33:21.342,0:33:25.603 And what was fascinating about this study was that in general, people were quite good at doing this: 0:33:25.604,0:33:33.309 even if they had no familiarity or background in psychology, they could distinguish a good explanation from a bad one. 0:33:33.309,0:33:39.629 But what they found was that if they added some verbiage that just talked about the brain in various ways, 0:33:39.629,0:33:45.245 and said, "This result came about because brain scans showed it", or "because we looked at the frontal lobes", 0:33:45.246,0:33:48.371 people were much more impressed with the bad explanations. 0:33:48.371,0:33:53.128 So a good explanation didn't get any better when you added all this neuroscience waffle, 0:33:53.129,0:33:59.340 but if you added neuroscience waffle to a bad explanation, people thought it not so bad. 0:33:59.341,0:34:05.954 And so, there is a tendency to be very impressed by anything that talks about, adds the brain in to an ........ (?) explanation. 0:34:05.955,0:34:12.994 And I think this is used by people who then try and add spurious neuroscience sometimes 0:34:12.995,0:34:18.171 to their accounts of their particular promissing theory. 0:34:18.172,0:34:23.808 And it really is not - we shouldn't allow ourselves to be mislead. 0:34:24.541,0:34:34.228 So I think, to sum up, there are a number of barriers to objective evaluation of intelligence, 0:34:34.229,0:34:38.309 which, to some extent, functions about (?) our human condition, 0:34:38.309,0:34:44.095 that we are not naturally good at taking in lots of numbers and looking at graphs 0:34:44.096,0:34:47.758 and trying to sort of take into account alternative explanations. 0:34:47.759,0:34:52.649 We tend to be impressed when we hear other people tell us that something has worked 0:34:52.649,0:34:57.183 and it's hard - you have to almost guard yourself against the tendency to do that 0:34:57.184,0:35:02.389 and to look rather for the hard evidence, to look for the actual numerical data. 0:35:03.249,0:35:08.301 We have to be very careful when people start giving us explanations that have got a lot of neuroscience in them 0:35:08.301,0:35:12.194 and check out, is this real neuroscience or is it just put in there to impress us? 0:35:13.808,0:35:19.508 We have to be aware of the effect of cognitive dissonance and the tendency to believe some things 0:35:19.509,0:35:23.866 simply because we have invested time and money in it 0:35:23.867,0:35:31.401 and, most importantly, we have to bear in mind that there will be effects on children's performance 0:35:31.401,0:35:37.974 of maturation, of our expectations, of just get practising on things, 0:35:37.975,0:35:42.705 and there are also these dreadful statistical artefacts that can make it look as if a change has occurred, 0:35:42.706,0:35:45.838 when it's really not particularly impressive. 0:35:45.838,0:35:50.363 But I think, if one bears these things in mind, the bottom line is really: 0:35:50.364,0:35:54.575 "Look for evidence from studies that have got adequate controls" 0:35:54.576,0:36:01.508 and if you do, you'll be in - a standard, I think, by how far you can see improvements in children, even if they haven't had the treatment. 0:36:01.508,0:36:06.140 and there are lots of things that will make things a lot better, just with the passage of time. 0:36:06.141,0:36:11.339 But if you really want to demonstrate that there has been an effective treatment, 0:36:11.339,0:36:15.804 you do have to show an improvement relative to a control group, 0:36:15.804,0:36:20.955 rather than just, somebody started out not so good and is now a little bit better after the treatment. 0:36:20.956,0:36:29.953 I hope that that might give you some useful indicators when trying to look at new treatments that are out there and on offer, 0:36:29.954,0:36:35.111 and for a more detailed account of some of this work, there are various - 0:36:35.112,0:36:40.104 there is a powerpoint presentation with notes on my website on this topic.