[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:00.06,0:00:01.09,Default,,0000,0000,0000,,So we spent a bunch of time
Dialogue: 0,0:00:01.09,0:00:03.03,Default,,0000,0000,0000,,in the last couple of lectures
Dialogue: 0,0:00:03.03,0:00:05.08,Default,,0000,0000,0000,,talking about different kinds of testing
Dialogue: 0,0:00:05.08,0:00:08.02,Default,,0000,0000,0000,,about unit testing versus integration testing
Dialogue: 0,0:00:08.02,0:00:10.01,Default,,0000,0000,0000,,We talked about how do you use RSpec
Dialogue: 0,0:00:10.01,0:00:12.05,Default,,0000,0000,0000,,to really isolate the parts of your code you want to test
Dialogue: 0,0:00:12.05,0:00:14.09,Default,,0000,0000,0000,,you’ve also, you know, because of homework 3,
Dialogue: 0,0:00:14.09,0:00:18.02,Default,,0000,0000,0000,,and other stuff, we have been doing BDD,
Dialogue: 0,0:00:18.02,0:00:20.06,Default,,0000,0000,0000,,where we’ve been using Cucumber to turn user stories
Dialogue: 0,0:00:20.06,0:00:22.10,Default,,0000,0000,0000,,into, essentially, integration and acceptance tests
Dialogue: 0,0:00:22.10,0:00:25.06,Default,,0000,0000,0000,,So you’ve seen testing in a couple of different levels
Dialogue: 0,0:00:25.06,0:00:27.06,Default,,0000,0000,0000,,and the goal here is sort of to do a few remarks
Dialogue: 0,0:00:27.06,0:00:29.09,Default,,0000,0000,0000,,to, you know, let’s back up a little bit
Dialogue: 0,0:00:29.09,0:00:33.00,Default,,0000,0000,0000,,and see the big picture, and tie those things together
Dialogue: 0,0:00:33.00,0:00:34.10,Default,,0000,0000,0000,,So this sort of spans material
Dialogue: 0,0:00:34.10,0:00:37.00,Default,,0000,0000,0000,,that covers three or four sections in the book
Dialogue: 0,0:00:37.00,0:00:39.06,Default,,0000,0000,0000,,and I want to just hit the high points in lecture
Dialogue: 0,0:00:39.06,0:00:41.05,Default,,0000,0000,0000,,So a question that comes up
Dialogue: 0,0:00:41.05,0:00:43.02,Default,,0000,0000,0000,,I’m sure it’s come up for all of you
Dialogue: 0,0:00:43.02,0:00:44.05,Default,,0000,0000,0000,,as you have been doing homework
Dialogue: 0,0:00:44.05,0:00:45.07,Default,,0000,0000,0000,,is: “How much testing is enough?”
Dialogue: 0,0:00:45.07,0:00:48.05,Default,,0000,0000,0000,,And, sadly, for a long time
Dialogue: 0,0:00:48.05,0:00:51.01,Default,,0000,0000,0000,,kind of if you asked this question in industry
Dialogue: 0,0:00:51.01,0:00:52.02,Default,,0000,0000,0000,,the answer was basically
Dialogue: 0,0:00:52.02,0:00:53.02,Default,,0000,0000,0000,,“Well, we have a shipping deadline,
Dialogue: 0,0:00:53.02,0:00:54.10,Default,,0000,0000,0000,,so however much testing we can do
Dialogue: 0,0:00:54.10,0:00:56.07,Default,,0000,0000,0000,,before that deadline, that’s how much.”
Dialogue: 0,0:00:56.07,0:00:58.02,Default,,0000,0000,0000,,That’s what you have time for.
Dialogue: 0,0:00:58.02,0:01:00.00,Default,,0000,0000,0000,,So, you know, that’s a little flip
Dialogue: 0,0:01:00.00,0:01:01.01,Default,,0000,0000,0000,,obviously not very good
Dialogue: 0,0:01:01.01,0:01:02.05,Default,,0000,0000,0000,,So you can do a bit better, right?
Dialogue: 0,0:01:02.05,0:01:03.07,Default,,0000,0000,0000,,There’re some static measures
Dialogue: 0,0:01:03.07,0:01:06.00,Default,,0000,0000,0000,,like how many lines of code does your app have
Dialogue: 0,0:01:06.00,0:01:08.02,Default,,0000,0000,0000,,and how many lines of tests do you have?
Dialogue: 0,0:01:08.02,0:01:10.03,Default,,0000,0000,0000,,And it’s not unusual in industry
Dialogue: 0,0:01:10.03,0:01:12.07,Default,,0000,0000,0000,,in a well-tested piece of software
Dialogue: 0,0:01:12.07,0:01:14.06,Default,,0000,0000,0000,,for the number of lines of tests
Dialogue: 0,0:01:14.06,0:01:17.07,Default,,0000,0000,0000,,to go far beyond the number of lines of code
Dialogue: 0,0:01:17.07,0:01:19.08,Default,,0000,0000,0000,,So, integer multiples are not unusual
Dialogue: 0,0:01:19.08,0:01:21.08,Default,,0000,0000,0000,,And I think even for sort of, you know,
Dialogue: 0,0:01:21.08,0:01:23.02,Default,,0000,0000,0000,,research code or classwork
Dialogue: 0,0:01:23.02,0:01:26.08,Default,,0000,0000,0000,,a ratio of, you know, maybe 1.5 is not unreasonable
Dialogue: 0,0:01:26.08,0:01:30.00,Default,,0000,0000,0000,,so one and a half times the amount of test code
Dialogue: 0,0:01:30.00,0:01:32.02,Default,,0000,0000,0000,,as you have application code
Dialogue: 0,0:01:32.02,0:01:34.02,Default,,0000,0000,0000,,And in a lot of production systems
Dialogue: 0,0:01:34.02,0:01:35.03,Default,,0000,0000,0000,,where they really care about testing
Dialogue: 0,0:01:35.03,0:01:36.09,Default,,0000,0000,0000,,it is much higher than that
Dialogue: 0,0:01:36.09,0:01:38.02,Default,,0000,0000,0000,,So maybe a better question to ask:
Dialogue: 0,0:01:38.02,0:01:39.05,Default,,0000,0000,0000,,Rather than saying “How much testing is enough?”
Dialogue: 0,0:01:39.05,0:01:42.05,Default,,0000,0000,0000,,is to ask “How good is the testing I am doing now?
Dialogue: 0,0:01:42.05,0:01:44.04,Default,,0000,0000,0000,,How thorough is it?”
Dialogue: 0,0:01:44.04,0:01:45.06,Default,,0000,0000,0000,,Later in this semester
Dialogue: 0,0:01:45.06,0:01:46.06,Default,,0000,0000,0000,,Professor Sen will talk about
Dialogue: 0,0:01:46.06,0:01:48.02,Default,,0000,0000,0000,,a little bit about formal methods
Dialogue: 0,0:01:48.02,0:01:50.08,Default,,0000,0000,0000,,and sort of what’s at the frontiers of testing and debugging
Dialogue: 0,0:01:50.08,0:01:52.07,Default,,0000,0000,0000,,But a couple of things that we can talk about
Dialogue: 0,0:01:52.07,0:01:54.01,Default,,0000,0000,0000,,based on what you already know
Dialogue: 0,0:01:54.01,0:01:57.07,Default,,0000,0000,0000,,is some basic concepts about test coverage
Dialogue: 0,0:01:57.07,0:01:59.05,Default,,0000,0000,0000,,And although I would say
Dialogue: 0,0:01:59.05,0:02:01.00,Default,,0000,0000,0000,,you know, we’ve been saying all along
Dialogue: 0,0:02:01.00,0:02:03.00,Default,,0000,0000,0000,,formal methods, they don’t really work on big systems
Dialogue: 0,0:02:03.00,0:02:05.03,Default,,0000,0000,0000,,I think that statement, in my personal opinion
Dialogue: 0,0:02:05.03,0:02:07.00,Default,,0000,0000,0000,,is actually a lot less true than it used to be
Dialogue: 0,0:02:07.00,0:02:09.02,Default,,0000,0000,0000,,I think there are a number of specific places
Dialogue: 0,0:02:09.02,0:02:10.05,Default,,0000,0000,0000,,especially in testing and debugging
Dialogue: 0,0:02:10.05,0:02:12.08,Default,,0000,0000,0000,,where formal methods are actually making fast progress
Dialogue: 0,0:02:12.08,0:02:15.08,Default,,0000,0000,0000,,and Koushik Sen is one of the leaders in that
Dialogue: 0,0:02:15.08,0:02:17.09,Default,,0000,0000,0000,,So you’ll have the opportunity to hear more about that later
Dialogue: 0,0:02:17.09,0:02:21.04,Default,,0000,0000,0000,,but for the moment I think, kind of bread and butter
Dialogue: 0,0:02:21.04,0:02:22.07,Default,,0000,0000,0000,,is let’s talk about coverage measurement
Dialogue: 0,0:02:22.07,0:02:24.05,Default,,0000,0000,0000,,because this is where the rubber meets the road
Dialogue: 0,0:02:24.05,0:02:26.02,Default,,0000,0000,0000,,in terms of how you’d be evaluated
Dialogue: 0,0:02:26.02,0:02:28.06,Default,,0000,0000,0000,,if you are doing this for real
Dialogue: 0,0:02:28.06,0:02:29.05,Default,,0000,0000,0000,,So what’s some basics?
Dialogue: 0,0:02:29.05,0:02:30.08,Default,,0000,0000,0000,,Here’s a really simple class you can use
Dialogue: 0,0:02:30.08,0:02:32.09,Default,,0000,0000,0000,,to talk about different ways to measure
Dialogue: 0,0:02:32.09,0:02:34.08,Default,,0000,0000,0000,,how our test covers this code
Dialogue: 0,0:02:34.08,0:02:36.06,Default,,0000,0000,0000,,And there’re a few different levels
Dialogue: 0,0:02:36.06,0:02:37.08,Default,,0000,0000,0000,,with different terminologies
Dialogue: 0,0:02:37.08,0:02:40.07,Default,,0000,0000,0000,,It’s not really universal across all software houses
Dialogue: 0,0:02:40.07,0:02:42.06,Default,,0000,0000,0000,,But one common set of terminology
Dialogue: 0,0:02:42.06,0:02:43.06,Default,,0000,0000,0000,,that the book exposes
Dialogue: 0,0:02:43.06,0:02:44.07,Default,,0000,0000,0000,,is we could talk about S0
Dialogue: 0,0:02:44.07,0:02:47.04,Default,,0000,0000,0000,,where we’d just mean you’ve called every method once
Dialogue: 0,0:02:47.04,0:02:50.04,Default,,0000,0000,0000,,So you know, if you call foo, and you call bar, you’re done
Dialogue: 0,0:02:50.04,0:02:52.02,Default,,0000,0000,0000,,That’s S0 coverage: not terribly thorough
Dialogue: 0,0:02:52.02,0:02:54.07,Default,,0000,0000,0000,,A little more stringent, S1, is
Dialogue: 0,0:02:54.07,0:02:56.01,Default,,0000,0000,0000,,you could say, we’re calling every method
Dialogue: 0,0:02:56.01,0:02:57.03,Default,,0000,0000,0000,,from every place that it could be called
Dialogue: 0,0:02:57.03,0:02:58.08,Default,,0000,0000,0000,,So what does that mean?
Dialogue: 0,0:02:58.08,0:03:00.01,Default,,0000,0000,0000,,It means, for example
Dialogue: 0,0:03:00.01,0:03:01.01,Default,,0000,0000,0000,,it’s not enough to call bar
Dialogue: 0,0:03:01.01,0:03:02.10,Default,,0000,0000,0000,,You have to make sure that you have to call it
Dialogue: 0,0:03:02.10,0:03:05.06,Default,,0000,0000,0000,,at least once from in here
Dialogue: 0,0:03:05.06,0:03:07.02,Default,,0000,0000,0000,,as well as calling it once
Dialogue: 0,0:03:07.02,0:03:10.04,Default,,0000,0000,0000,,from any exterior function that might call it
Dialogue: 0,0:03:10.04,0:03:12.08,Default,,0000,0000,0000,,C0 which is what SimpleCov measures
Dialogue: 0,0:03:12.08,0:03:15.10,Default,,0000,0000,0000,,(those of you who’ve gotten SimpleCov up and running)
Dialogue: 0,0:03:15.10,0:03:18.05,Default,,0000,0000,0000,,basically says you’ve executed every statement
Dialogue: 0,0:03:18.05,0:03:20.00,Default,,0000,0000,0000,,you’ve touched every statement in your code once
Dialogue: 0,0:03:20.00,0:03:22.05,Default,,0000,0000,0000,,But the caveat there is that
Dialogue: 0,0:03:22.05,0:03:25.06,Default,,0000,0000,0000,,conditionals really just count as a single statement
Dialogue: 0,0:03:25.06,0:03:28.09,Default,,0000,0000,0000,,So, if you, no matter which branch of this “if” you took
Dialogue: 0,0:03:28.09,0:03:31.07,Default,,0000,0000,0000,,as long as you touched one of the other branch
Dialogue: 0,0:03:31.07,0:03:33.04,Default,,0000,0000,0000,,you’ve executed the “if’ statement
Dialogue: 0,0:03:33.04,0:03:35.07,Default,,0000,0000,0000,,So even C0 is still, you know, sort of superficial coverage
Dialogue: 0,0:03:35.07,0:03:37.03,Default,,0000,0000,0000,,But, as we will see
Dialogue: 0,0:03:37.03,0:03:39.02,Default,,0000,0000,0000,,the way that you will want to read this information is:
Dialogue: 0,0:03:39.02,0:03:41.08,Default,,0000,0000,0000,,if you are getting {\i1}bad{\i0} coverage at the C0 level
Dialogue: 0,0:03:41.08,0:03:44.01,Default,,0000,0000,0000,,then you have really really bad coverage
Dialogue: 0,0:03:44.01,0:03:46.01,Default,,0000,0000,0000,,So if you are not kind of making
Dialogue: 0,0:03:46.01,0:03:47.04,Default,,0000,0000,0000,,this simple level of superficial coverage
Dialogue: 0,0:03:47.04,0:03:50.00,Default,,0000,0000,0000,,then your testing is probably deficient
Dialogue: 0,0:03:50.00,0:03:51.09,Default,,0000,0000,0000,,C1 is the next step up from that
Dialogue: 0,0:03:51.09,0:03:53.07,Default,,0000,0000,0000,,We could say:
Dialogue: 0,0:03:53.07,0:03:55.02,Default,,0000,0000,0000,,Well, we have to take every branch in both directions
Dialogue: 0,0:03:55.02,0:03:56.06,Default,,0000,0000,0000,,So, when we are doing this “if” statement
Dialogue: 0,0:03:56.06,0:03:58.07,Default,,0000,0000,0000,,we have to make sure that
Dialogue: 0,0:03:58.07,0:03:59.09,Default,,0000,0000,0000,,we do the “if x” part once
Dialogue: 0,0:03:59.09,0:04:05.01,Default,,0000,0000,0000,,and the “if not x” part at least once to meet C1
Dialogue: 0,0:04:05.01,0:04:08.04,Default,,0000,0000,0000,,You can augment that with decision coverage
Dialogue: 0,0:04:08.04,0:04:09.06,Default,,0000,0000,0000,,saying: Well, if we’re gonna…
Dialogue: 0,0:04:09.06,0:04:12.04,Default,,0000,0000,0000,,If we have “if” statments where the condition
Dialogue: 0,0:04:12.04,0:04:13.09,Default,,0000,0000,0000,,is made up of multiple terms
Dialogue: 0,0:04:13.09,0:04:15.07,Default,,0000,0000,0000,,we have to make sure that every subexpression
Dialogue: 0,0:04:15.07,0:04:17.10,Default,,0000,0000,0000,,has been evaluated both directions
Dialogue: 0,0:04:17.10,0:04:19.07,Default,,0000,0000,0000,,In other words, that means that
Dialogue: 0,0:04:19.07,0:04:22.04,Default,,0000,0000,0000,,if we’re going to fail this “if” statement
Dialogue: 0,0:04:22.04,0:04:24.03,Default,,0000,0000,0000,,we have to make sure to fail it at least once
Dialogue: 0,0:04:24.03,0:04:26.04,Default,,0000,0000,0000,,because y was false in at least once because z was false
Dialogue: 0,0:04:26.04,0:04:28.09,Default,,0000,0000,0000,,In other words, any subexpression that could
Dialogue: 0,0:04:28.09,0:04:31.02,Default,,0000,0000,0000,,independently change the outcome of the condition
Dialogue: 0,0:04:31.02,0:04:34.05,Default,,0000,0000,0000,,has to be exercised in both directions
Dialogue: 0,0:04:34.05,0:04:36.00,Default,,0000,0000,0000,,And then,
Dialogue: 0,0:04:36.00,0:04:38.05,Default,,0000,0000,0000,,kind of, the one that, you know, a lot of people aspire to
Dialogue: 0,0:04:38.05,0:04:41.03,Default,,0000,0000,0000,,but there is disagreement on how much more valuable it is
Dialogue: 0,0:04:41.03,0:04:42.08,Default,,0000,0000,0000,,is you take every path through the code
Dialogue: 0,0:04:42.08,0:04:45.05,Default,,0000,0000,0000,,Obviously, this is kind of difficult because
Dialogue: 0,0:04:45.05,0:04:48.03,Default,,0000,0000,0000,,it tends to be exponential in the number of conditions
Dialogue: 0,0:04:48.03,0:04:53.01,Default,,0000,0000,0000,,And in general it’s difficult
Dialogue: 0,0:04:53.01,0:04:55.03,Default,,0000,0000,0000,,to evaluate if you’ve taken every path through the code
Dialogue: 0,0:04:55.03,0:04:57.00,Default,,0000,0000,0000,,There are formal techniques that you can use
Dialogue: 0,0:04:57.00,0:04:58.08,Default,,0000,0000,0000,,to tell you where the holes are
Dialogue: 0,0:04:58.08,0:05:01.03,Default,,0000,0000,0000,,but the bottom line is that
Dialogue: 0,0:05:01.03,0:05:03.00,Default,,0000,0000,0000,,in most commercial software houses
Dialogue: 0,0:05:03.00,0:05:04.09,Default,,0000,0000,0000,,there is, I would say, not complete consensus
Dialogue: 0,0:05:04.09,0:05:06.07,Default,,0000,0000,0000,,on how much more valuable C2 is
Dialogue: 0,0:05:06.07,0:05:08.07,Default,,0000,0000,0000,,compared to C0 or C1
Dialogue: 0,0:05:08.07,0:05:10.01,Default,,0000,0000,0000,,So, I think, for the purpose of our class
Dialogue: 0,0:05:10.01,0:05:11.07,Default,,0000,0000,0000,,you get exposed to the idea
Dialogue: 0,0:05:11.07,0:05:13.02,Default,,0000,0000,0000,,of how you use coverage information
Dialogue: 0,0:05:13.02,0:05:16.04,Default,,0000,0000,0000,,SimpleCov takes advantage of some built-in Ruby features
Dialogue: 0,0:05:16.04,0:05:18.01,Default,,0000,0000,0000,,to give you C0 coverage
Dialogue: 0,0:05:18.01,0:05:19.06,Default,,0000,0000,0000,,[It] does really nice reports
Dialogue: 0,0:05:19.06,0:05:21.02,Default,,0000,0000,0000,,We can sort of see it
Dialogue: 0,0:05:21.02,0:05:22.10,Default,,0000,0000,0000,,at the level of individual lines in your file
Dialogue: 0,0:05:22.10,0:05:24.09,Default,,0000,0000,0000,,You can see what your coverage is
Dialogue: 0,0:05:24.09,0:05:27.02,Default,,0000,0000,0000,,and I think that’s kind of a, you know
Dialogue: 0,0:05:27.02,0:05:31.02,Default,,0000,0000,0000,,a good start for where we are
Dialogue: 0,0:05:31.02,0:05:33.08,Default,,0000,0000,0000,,So, having see a sort of different flavours of tests
Dialogue: 0,0:05:33.08,0:05:37.02,Default,,0000,0000,0000,,Stepping back and looking back at the big picture
Dialogue: 0,0:05:37.02,0:05:38.10,Default,,0000,0000,0000,,what are the different kind of tests
Dialogue: 0,0:05:38.10,0:05:40.08,Default,,0000,0000,0000,,that we’ve seen concretely?
Dialogue: 0,0:05:40.08,0:05:42.03,Default,,0000,0000,0000,,and what are the tradeoffs
Dialogue: 0,0:05:42.03,0:05:43.09,Default,,0000,0000,0000,,between using those different kinds of tests?
Dialogue: 0,0:05:43.09,0:05:47.02,Default,,0000,0000,0000,,So we’ve seen at the level of individual classes or methods
Dialogue: 0,0:05:47.02,0:05:50.01,Default,,0000,0000,0000,,we use RSpec, with extensive use of mocking and stubbing
Dialogue: 0,0:05:50.01,0:05:53.00,Default,,0000,0000,0000,,So, for example when we do testing methods in the model
Dialogue: 0,0:05:53.00,0:05:55.06,Default,,0000,0000,0000,,that will be an example of unit testing
Dialogue: 0,0:05:55.06,0:05:59.02,Default,,0000,0000,0000,,We also did something that is pretty similar to
Dialogue: 0,0:05:59.02,0:06:00.10,Default,,0000,0000,0000,,functional or module testing
Dialogue: 0,0:06:00.10,0:06:02.07,Default,,0000,0000,0000,,where there is more than one module participating
Dialogue: 0,0:06:02.07,0:06:04.06,Default,,0000,0000,0000,,So, for example when we did controller specs
Dialogue: 0,0:06:04.06,0:06:07.08,Default,,0000,0000,0000,,we saw that—we simulate a POST action
Dialogue: 0,0:06:07.08,0:06:09.03,Default,,0000,0000,0000,,but remember that the POST action
Dialogue: 0,0:06:09.03,0:06:10.09,Default,,0000,0000,0000,,has to go through the routing subsystem
Dialogue: 0,0:06:10.09,0:06:12.04,Default,,0000,0000,0000,,before it gets to the controller
Dialogue: 0,0:06:12.04,0:06:14.05,Default,,0000,0000,0000,,Once the controller is done it will try to render a view
Dialogue: 0,0:06:14.05,0:06:16.01,Default,,0000,0000,0000,,So in fact there’s other pieces
Dialogue: 0,0:06:16.01,0:06:17.07,Default,,0000,0000,0000,,that collaborate with the controller
Dialogue: 0,0:06:17.07,0:06:19.10,Default,,0000,0000,0000,,that have to be working in order for controller specs to pass
Dialogue: 0,0:06:19.10,0:06:21.05,Default,,0000,0000,0000,,So that’s somewhere inbetween:
Dialogue: 0,0:06:21.05,0:06:23.04,Default,,0000,0000,0000,,where we’re doing more than a single method
Dialogue: 0,0:06:23.04,0:06:25.00,Default,,0000,0000,0000,,touching more than a single class
Dialogue: 0,0:06:25.00,0:06:27.00,Default,,0000,0000,0000,,but we’re still concentrating [our] attention
Dialogue: 0,0:06:27.00,0:06:28.09,Default,,0000,0000,0000,,on a fairly narrow slice of the system at a time
Dialogue: 0,0:06:28.09,0:06:31.04,Default,,0000,0000,0000,,and we’re still using mocking and stubbing extensively
Dialogue: 0,0:06:31.04,0:06:35.03,Default,,0000,0000,0000,,to sort of isolate that behaviour that we want to test
Dialogue: 0,0:06:35.03,0:06:36.09,Default,,0000,0000,0000,,And then at the level of Cucumber scenarios
Dialogue: 0,0:06:36.09,0:06:38.05,Default,,0000,0000,0000,,these are more like integration or system tests
Dialogue: 0,0:06:38.05,0:06:41.07,Default,,0000,0000,0000,,They exercise complete paths throughout the application
Dialogue: 0,0:06:41.07,0:06:43.04,Default,,0000,0000,0000,,They probably touch a lot of different modules
Dialogue: 0,0:06:43.04,0:06:46.00,Default,,0000,0000,0000,,They make minimal use of mocks and stubs
Dialogue: 0,0:06:46.00,0:06:48.03,Default,,0000,0000,0000,,because part of the goal of an integration test
Dialogue: 0,0:06:48.03,0:06:50.10,Default,,0000,0000,0000,,is exactly to test the interaction between pieces
Dialogue: 0,0:06:50.10,0:06:53.02,Default,,0000,0000,0000,,So you don’t want to stub or control those interactions
Dialogue: 0,0:06:53.02,0:06:54.08,Default,,0000,0000,0000,,You actually want to let the system do
Dialogue: 0,0:06:54.08,0:06:56.03,Default,,0000,0000,0000,,what it would really do
Dialogue: 0,0:06:56.03,0:06:58.02,Default,,0000,0000,0000,,if this was a scenario happening in production
Dialogue: 0,0:06:58.02,0:07:00.07,Default,,0000,0000,0000,,So how would we compare these different kinds of tests?
Dialogue: 0,0:07:00.07,0:07:02.04,Default,,0000,0000,0000,,There’s a few different axes we can look at
Dialogue: 0,0:07:02.04,0:07:05.01,Default,,0000,0000,0000,,One of them is how long they take to run
Dialogue: 0,0:07:05.01,0:07:06.09,Default,,0000,0000,0000,,Now, both RSpec and Cucumber
Dialogue: 0,0:07:06.09,0:07:09.01,Default,,0000,0000,0000,,have, kind of, high startup times and stuff like that
Dialogue: 0,0:07:09.01,0:07:10.01,Default,,0000,0000,0000,,But, as you’ll see
Dialogue: 0,0:07:10.01,0:07:11.09,Default,,0000,0000,0000,,as you start adding more and more RSpec tests
Dialogue: 0,0:07:11.09,0:07:14.04,Default,,0000,0000,0000,,and using autotest to run them in the background
Dialogue: 0,0:07:14.04,0:07:17.09,Default,,0000,0000,0000,,by and large, once RSpec kind of gets off the launching pad
Dialogue: 0,0:07:17.09,0:07:19.09,Default,,0000,0000,0000,,it runs specs really fast
Dialogue: 0,0:07:19.09,0:07:21.10,Default,,0000,0000,0000,,whereas running Cucumber features just takes a long time
Dialogue: 0,0:07:21.10,0:07:24.06,Default,,0000,0000,0000,,as it essentially fires up your entire application
Dialogue: 0,0:07:24.06,0:07:26.01,Default,,0000,0000,0000,,And later in this semester
Dialogue: 0,0:07:26.01,0:07:28.09,Default,,0000,0000,0000,,we’ll see a way to make Cucumber even slower—
Dialogue: 0,0:07:28.09,0:07:30.07,Default,,0000,0000,0000,,which is to have it fire up an entire browser
Dialogue: 0,0:07:30.07,0:07:33.04,Default,,0000,0000,0000,,basically act like a puppet, remote-controlling Firefox
Dialogue: 0,0:07:33.04,0:07:35.08,Default,,0000,0000,0000,,so you can test Javascript code
Dialogue: 0,0:07:35.08,0:07:37.00,Default,,0000,0000,0000,,We’ll do that when we actually—
Dialogue: 0,0:07:37.00,0:07:40.03,Default,,0000,0000,0000,,I think we’ll be able to work with our friends at SourceLabs
Dialogue: 0,0:07:40.03,0:07:42.08,Default,,0000,0000,0000,,so you can do that in the cloud—That will be exciting
Dialogue: 0,0:07:42.08,0:07:45.08,Default,,0000,0000,0000,,So, “run fast” versus “run slow”
Dialogue: 0,0:07:45.08,0:07:46.07,Default,,0000,0000,0000,,Resolution:
Dialogue: 0,0:07:46.07,0:07:48.02,Default,,0000,0000,0000,,If an error happens in your unit tests
Dialogue: 0,0:07:48.02,0:07:49.08,Default,,0000,0000,0000,,it’s usually pretty easy
Dialogue: 0,0:07:49.08,0:07:52.03,Default,,0000,0000,0000,,to figure out and track down what the source of that error is
Dialogue: 0,0:07:52.03,0:07:53.07,Default,,0000,0000,0000,,because the tests are so isolated
Dialogue: 0,0:07:53.07,0:07:56.02,Default,,0000,0000,0000,,You’ve stubbed out everything that doesn’t matter
Dialogue: 0,0:07:56.02,0:07:58.02,Default,,0000,0000,0000,,and you’re focusing on only the behaviour of interest
Dialogue: 0,0:07:58.02,0:07:59.08,Default,,0000,0000,0000,,So, if you’ve done a good job of doing that
Dialogue: 0,0:07:59.08,0:08:01.10,Default,,0000,0000,0000,,when something goes wrong in one of your tests
Dialogue: 0,0:08:01.10,0:08:03.04,Default,,0000,0000,0000,,there’s not a lot of places
Dialogue: 0,0:08:03.04,0:08:04.09,Default,,0000,0000,0000,,that something could have gone wrong
Dialogue: 0,0:08:04.09,0:08:07.04,Default,,0000,0000,0000,,In contrast, if you’re running a Cucumber scenario
Dialogue: 0,0:08:07.04,0:08:08.09,Default,,0000,0000,0000,,that’s got, you know, 10 steps
Dialogue: 0,0:08:08.09,0:08:10.03,Default,,0000,0000,0000,,and every step is touching
Dialogue: 0,0:08:10.03,0:08:11.06,Default,,0000,0000,0000,,a whole bunch of pieces of the app
Dialogue: 0,0:08:11.06,0:08:12.09,Default,,0000,0000,0000,,it could take a long time
Dialogue: 0,0:08:12.09,0:08:14.08,Default,,0000,0000,0000,,to actually get to the bottom of a bug
Dialogue: 0,0:08:14.08,0:08:16.01,Default,,0000,0000,0000,,So it is kind of a tradeoff
Dialogue: 0,0:08:16.01,0:08:17.05,Default,,0000,0000,0000,,between how well can you localize errors
Dialogue: 0,0:08:17.05,0:08:20.06,Default,,0000,0000,0000,,Coverage:
Dialogue: 0,0:08:20.06,0:08:23.00,Default,,0000,0000,0000,,It’s possible if you write a good suite
Dialogue: 0,0:08:23.00,0:08:24.07,Default,,0000,0000,0000,,of unit and functional tests
Dialogue: 0,0:08:24.07,0:08:26.02,Default,,0000,0000,0000,,you can get really high coverage
Dialogue: 0,0:08:26.02,0:08:27.08,Default,,0000,0000,0000,,You can run your SimpleCov report
Dialogue: 0,0:08:27.08,0:08:30.08,Default,,0000,0000,0000,,and you can actually identify specific lines in your files
Dialogue: 0,0:08:30.08,0:08:32.04,Default,,0000,0000,0000,,that have not been exercised by any test
Dialogue: 0,0:08:32.04,0:08:34.02,Default,,0000,0000,0000,,and then you can go right at tests that cover them
Dialogue: 0,0:08:34.02,0:08:36.01,Default,,0000,0000,0000,,So, figuring out how to improve your coverage
Dialogue: 0,0:08:36.01,0:08:37.06,Default,,0000,0000,0000,,for example at the C0 level
Dialogue: 0,0:08:37.06,0:08:40.02,Default,,0000,0000,0000,,is something much more easily done with unit tests
Dialogue: 0,0:08:40.02,0:08:42.02,Default,,0000,0000,0000,,whereas, with a Cucumber test—
Dialogue: 0,0:08:42.02,0:08:43.08,Default,,0000,0000,0000,,with a Cucumber scenario—
Dialogue: 0,0:08:43.08,0:08:45.08,Default,,0000,0000,0000,,you {\i1}are{\i0} touching a lot of parts of the code
Dialogue: 0,0:08:45.08,0:08:47.08,Default,,0000,0000,0000,,but you are doing it very sparsely
Dialogue: 0,0:08:47.08,0:08:49.04,Default,,0000,0000,0000,,So, if your goal is to get your coverage up
Dialogue: 0,0:08:49.04,0:08:51.03,Default,,0000,0000,0000,,use the tools at that are at the unit levels
Dialogue: 0,0:08:51.03,0:08:53.01,Default,,0000,0000,0000,,so that you can focusing on understanding
Dialogue: 0,0:08:53.01,0:08:54.07,Default,,0000,0000,0000,,what parts or my code are undertested
Dialogue: 0,0:08:54.07,0:08:56.06,Default,,0000,0000,0000,,and then you can write very targeted tests
Dialogue: 0,0:08:56.06,0:08:58.09,Default,,0000,0000,0000,,just to focus on them
Dialogue: 0,0:08:58.09,0:09:01.04,Default,,0000,0000,0000,,And, sort of, you know, putting those pieces together
Dialogue: 0,0:09:01.04,0:09:03.04,Default,,0000,0000,0000,,the unit tests
Dialogue: 0,0:09:03.04,0:09:05.06,Default,,0000,0000,0000,,because of their isolation and their fine resolution
Dialogue: 0,0:09:05.06,0:09:07.04,Default,,0000,0000,0000,,tend to use a lot of mocks
Dialogue: 0,0:09:07.04,0:09:09.01,Default,,0000,0000,0000,,to isolate the behaviours you don’t care about
Dialogue: 0,0:09:09.01,0:09:11.02,Default,,0000,0000,0000,,But that means that, by definition
Dialogue: 0,0:09:11.02,0:09:12.07,Default,,0000,0000,0000,,you’re not testing the interfaces
Dialogue: 0,0:09:12.07,0:09:14.10,Default,,0000,0000,0000,,and it’s sort of a “received wisdom” in software
Dialogue: 0,0:09:14.10,0:09:16.07,Default,,0000,0000,0000,,that a lot of the interesting bugs
Dialogue: 0,0:09:16.07,0:09:18.08,Default,,0000,0000,0000,,occur at the interfaces between pieces
Dialogue: 0,0:09:18.08,0:09:20.08,Default,,0000,0000,0000,,and not sort of within a class or within a method—
Dialogue: 0,0:09:20.08,0:09:22.04,Default,,0000,0000,0000,,those are sort of the easy bugs to track down
Dialogue: 0,0:09:22.04,0:09:24.03,Default,,0000,0000,0000,,And at the other extreme
Dialogue: 0,0:09:24.03,0:09:26.08,Default,,0000,0000,0000,,the more you get towards the integration testing extreme
Dialogue: 0,0:09:26.08,0:09:29.07,Default,,0000,0000,0000,,you’re supposed to rely less and less on mocks
Dialogue: 0,0:09:29.07,0:09:30.09,Default,,0000,0000,0000,,for that exact reason
Dialogue: 0,0:09:30.09,0:09:32.07,Default,,0000,0000,0000,,Now we saw, if you’re testing something like
Dialogue: 0,0:09:32.07,0:09:34.02,Default,,0000,0000,0000,,say, in a service-oriented architecture
Dialogue: 0,0:09:34.02,0:09:35.09,Default,,0000,0000,0000,,where you have to interact with the remote site
Dialogue: 0,0:09:35.09,0:09:37.03,Default,,0000,0000,0000,,you still end up
Dialogue: 0,0:09:37.03,0:09:38.09,Default,,0000,0000,0000,,having to do a fair amount of mocking and stubbing
Dialogue: 0,0:09:38.09,0:09:40.03,Default,,0000,0000,0000,,so that you don’t rely on the Internet
Dialogue: 0,0:09:40.03,0:09:41.07,Default,,0000,0000,0000,,in order for your tests to pass
Dialogue: 0,0:09:41.07,0:09:43.01,Default,,0000,0000,0000,,but, generally speaking
Dialogue: 0,0:09:43.01,0:09:47.01,Default,,0000,0000,0000,,you’re trying to remove as many of the mocks that you can
Dialogue: 0,0:09:47.01,0:09:48.10,Default,,0000,0000,0000,,and let the system run the way it would run in real life
Dialogue: 0,0:09:48.10,0:09:52.07,Default,,0000,0000,0000,,So, the good news is you {\i1}are{\i0} testing the interfaces
Dialogue: 0,0:09:52.07,0:09:54.07,Default,,0000,0000,0000,,{\i1}but{\i0} when something goes wrong in one of the interfaces
Dialogue: 0,0:09:54.07,0:09:57.05,Default,,0000,0000,0000,,because your resolution is not as good
Dialogue: 0,0:09:57.05,0:10:00.03,Default,,0000,0000,0000,,it may take longer to figure out what it is
Dialogue: 0,0:10:00.03,0:10:05.02,Default,,0000,0000,0000,,So, what’s sort of the high-order bit from this tradeoff
Dialogue: 0,0:10:05.02,0:10:07.02,Default,,0000,0000,0000,,is you don’t really want to rely
Dialogue: 0,0:10:07.02,0:10:08.08,Default,,0000,0000,0000,,too heavily on any one kind of test
Dialogue: 0,0:10:08.08,0:10:10.08,Default,,0000,0000,0000,,They serve different purposes and, depending on
Dialogue: 0,0:10:10.08,0:10:13.04,Default,,0000,0000,0000,,are you trying to exercise your interfaces more
Dialogue: 0,0:10:13.04,0:10:15.09,Default,,0000,0000,0000,,or are you trying to improve your fine-grain coverage
Dialogue: 0,0:10:15.09,0:10:18.00,Default,,0000,0000,0000,,that affects how you develop your test suite
Dialogue: 0,0:10:18.00,0:10:20.06,Default,,0000,0000,0000,,and you’ll evolve it along with your software
Dialogue: 0,0:10:20.06,0:10:24.01,Default,,0000,0000,0000,,So, we’ve used a certain set of terminology in testing
Dialogue: 0,0:10:24.01,0:10:26.03,Default,,0000,0000,0000,,It’s the terminology that, by and large
Dialogue: 0,0:10:26.03,0:10:29.00,Default,,0000,0000,0000,,is most commonly used in the Rails community
Dialogue: 0,0:10:29.00,0:10:30.06,Default,,0000,0000,0000,,but there’s some variation
Dialogue: 0,0:10:30.06,0:10:33.07,Default,,0000,0000,0000,,[and] some other terms that you might hear
Dialogue: 0,0:10:33.07,0:10:35.02,Default,,0000,0000,0000,,if you go get a job somewhere
Dialogue: 0,0:10:35.02,0:10:36.09,Default,,0000,0000,0000,,and you hear about mutation testing
Dialogue: 0,0:10:36.09,0:10:38.07,Default,,0000,0000,0000,,which we haven’t done
Dialogue: 0,0:10:38.07,0:10:40.02,Default,,0000,0000,0000,,This is an interesting idea that was, I think, invented by
Dialogue: 0,0:10:40.02,0:10:43.04,Default,,0000,0000,0000,,Ammann and Offutt, who have, sort of
Dialogue: 0,0:10:43.04,0:10:44.09,Default,,0000,0000,0000,,the definitive book on software testing
Dialogue: 0,0:10:44.09,0:10:46.05,Default,,0000,0000,0000,,The idea is:
Dialogue: 0,0:10:46.05,0:10:48.00,Default,,0000,0000,0000,,Suppose I introduced a deliberate bug into my code
Dialogue: 0,0:10:48.00,0:10:49.05,Default,,0000,0000,0000,,does that force some test to fail?
Dialogue: 0,0:10:49.05,0:10:53.00,Default,,0000,0000,0000,,Because, if I changed, you know, “if x” to “if not x”
Dialogue: 0,0:10:53.00,0:10:56.01,Default,,0000,0000,0000,,and no tests fail, then either I’m missing some coverage
Dialogue: 0,0:10:56.01,0:10:59.02,Default,,0000,0000,0000,,or my app is very strange and somehow nondeterministic
Dialogue: 0,0:10:59.02,0:11:03.10,Default,,0000,0000,0000,,Fuzz testing, which Koushik Sen may talk more about
Dialogue: 0,0:11:03.10,0:11:07.08,Default,,0000,0000,0000,,basically, this is the “10,000 monkeys at typewriters
Dialogue: 0,0:11:07.08,0:11:09.02,Default,,0000,0000,0000,,throwing random input at your code”
Dialogue: 0,0:11:09.02,0:11:10.04,Default,,0000,0000,0000,,What’s interesting about it is that
Dialogue: 0,0:11:10.04,0:11:11.06,Default,,0000,0000,0000,,those tests we’ve been doing
Dialogue: 0,0:11:11.06,0:11:13.09,Default,,0000,0000,0000,,essentially are crafted to test the app
Dialogue: 0,0:11:13.09,0:11:15.06,Default,,0000,0000,0000,,the way it was designed
Dialogue: 0,0:11:15.06,0:11:16.09,Default,,0000,0000,0000,,and these, you know, fuzz testing
Dialogue: 0,0:11:16.09,0:11:19.06,Default,,0000,0000,0000,,is about testing the app in ways it {\i1}wasn’t{\i0} meant to be used
Dialogue: 0,0:11:19.06,0:11:22.10,Default,,0000,0000,0000,,So, what happens if you throw enormous form submissions
Dialogue: 0,0:11:22.10,0:11:25.04,Default,,0000,0000,0000,,What happens if you put control characters in your forms?
Dialogue: 0,0:11:25.04,0:11:27.06,Default,,0000,0000,0000,,What happens if you submit the same thing over and over?
Dialogue: 0,0:11:27.06,0:11:29.09,Default,,0000,0000,0000,,And, Koushik has a statistic that
Dialogue: 0,0:11:29.09,0:11:32.03,Default,,0000,0000,0000,,Microsoft finds up to 20% of their bugs
Dialogue: 0,0:11:32.03,0:11:34.06,Default,,0000,0000,0000,,using some variation of fuzz testing
Dialogue: 0,0:11:34.06,0:11:36.03,Default,,0000,0000,0000,,and that about 25%
Dialogue: 0,0:11:36.03,0:11:39.02,Default,,0000,0000,0000,,of the common Unix command-line programs
Dialogue: 0,0:11:39.02,0:11:40.09,Default,,0000,0000,0000,,can be made to crash
Dialogue: 0,0:11:40.09,0:11:44.02,Default,,0000,0000,0000,,[when] put through aggressive fuzz testing
Dialogue: 0,0:11:44.02,0:11:46.09,Default,,0000,0000,0000,,Defining-use coverage is something that we haven’t done
Dialogue: 0,0:11:46.09,0:11:48.09,Default,,0000,0000,0000,,but it’s another interesting concept
Dialogue: 0,0:11:48.09,0:11:50.09,Default,,0000,0000,0000,,The idea is that at any point in my program
Dialogue: 0,0:11:50.09,0:11:52.06,Default,,0000,0000,0000,,there’s a place where I define—
Dialogue: 0,0:11:52.06,0:11:54.05,Default,,0000,0000,0000,,or I assign a value to some variable—
Dialogue: 0,0:11:54.05,0:11:56.00,Default,,0000,0000,0000,,and then there’s a place downstream
Dialogue: 0,0:11:56.00,0:11:57.08,Default,,0000,0000,0000,,where presumably I’m going to consume that value—
Dialogue: 0,0:11:57.08,0:11:59.06,Default,,0000,0000,0000,,someone’s going to use that value
Dialogue: 0,0:11:59.06,0:12:01.01,Default,,0000,0000,0000,,Have I covered every pair?
Dialogue: 0,0:12:01.01,0:12:02.06,Default,,0000,0000,0000,,In other words, do I have tests where every pair
Dialogue: 0,0:12:02.06,0:12:04.05,Default,,0000,0000,0000,,of defining a variable and using it somewhere
Dialogue: 0,0:12:04.05,0:12:07.01,Default,,0000,0000,0000,,is executed at some part of my test suites
Dialogue: 0,0:12:07.01,0:12:10.07,Default,,0000,0000,0000,,It’s sometimes called DU-coverage
Dialogue: 0,0:12:10.07,0:12:14.01,Default,,0000,0000,0000,,And other terms that I think are not as widely used anymore
Dialogue: 0,0:12:14.01,0:12:17.07,Default,,0000,0000,0000,,blackbox versus whitebox, or blackbox versus glassbox
Dialogue: 0,0:12:17.07,0:12:20.02,Default,,0000,0000,0000,,Roughly, a blackbox test is one that is written from
Dialogue: 0,0:12:20.02,0:12:22.04,Default,,0000,0000,0000,,the point of view of the external specification of the thing
Dialogue: 0,0:12:22.04,0:12:24.02,Default,,0000,0000,0000,,[For example:] “This is a hash table
Dialogue: 0,0:12:24.02,0:12:26.02,Default,,0000,0000,0000,,When I put in a key I should get back a value
Dialogue: 0,0:12:26.02,0:12:28.01,Default,,0000,0000,0000,,If I delete the key the value shouldn’t be there”
Dialogue: 0,0:12:28.01,0:12:29.10,Default,,0000,0000,0000,,That’s a blackbox test because it doesn’t say
Dialogue: 0,0:12:29.10,0:12:32.03,Default,,0000,0000,0000,,anything about how the hash table is implemented
Dialogue: 0,0:12:32.03,0:12:34.07,Default,,0000,0000,0000,,and it doesn’t try to stress the implementation
Dialogue: 0,0:12:34.07,0:12:36.06,Default,,0000,0000,0000,,A corresponding whitebox test might be:
Dialogue: 0,0:12:36.06,0:12:38.01,Default,,0000,0000,0000,,“I know something about the hash function
Dialogue: 0,0:12:38.01,0:12:39.10,Default,,0000,0000,0000,,and I’m going to deliberately create
Dialogue: 0,0:12:39.10,0:12:41.09,Default,,0000,0000,0000,,hash keys in my test cases
Dialogue: 0,0:12:41.09,0:12:43.08,Default,,0000,0000,0000,,that cause a lot of hash collisions
Dialogue: 0,0:12:43.08,0:12:45.10,Default,,0000,0000,0000,,to make sure that I’m testing that part of the functionality”
Dialogue: 0,0:12:45.10,0:12:49.01,Default,,0000,0000,0000,,Now, a C0 test coverage tool, like SimpleCov
Dialogue: 0,0:12:49.01,0:12:52.00,Default,,0000,0000,0000,,would reveal that if all you had is blackbox tests
Dialogue: 0,0:12:52.00,0:12:53.03,Default,,0000,0000,0000,,you might find that
Dialogue: 0,0:12:53.03,0:12:55.06,Default,,0000,0000,0000,,the collision coverage code wasn’t being hit very often
Dialogue: 0,0:12:55.06,0:12:56.08,Default,,0000,0000,0000,,And that might tip you off and say:
Dialogue: 0,0:12:56.08,0:12:58.03,Default,,0000,0000,0000,,“Ok, if I really want to strengthen that—
Dialogue: 0,0:12:58.03,0:13:00.01,Default,,0000,0000,0000,,for one, if I want to boost coverage for those tests
Dialogue: 0,0:13:00.01,0:13:02.01,Default,,0000,0000,0000,,now I have to write a whitebox or a glassbox test
Dialogue: 0,0:13:02.01,0:13:04.06,Default,,0000,0000,0000,,I have to look inside, see what the implementation does
Dialogue: 0,0:13:04.06,0:13:05.06,Default,,0000,0000,0000,,and find specific ways
Dialogue: 0,0:13:05.06,0:13:10.06,Default,,0000,0000,0000,,to try to break the implementation in evil ways”
Dialogue: 0,0:13:10.06,0:13:13.08,Default,,0000,0000,0000,,So, I think, testing is a kind of a way of life, right?
Dialogue: 0,0:13:13.08,0:13:16.07,Default,,0000,0000,0000,,We’ve gotten away from the phase of
Dialogue: 0,0:13:16.07,0:13:18.03,Default,,0000,0000,0000,,“We’d build the whole thing and then we’d test it”
Dialogue: 0,0:13:18.03,0:13:19.09,Default,,0000,0000,0000,,and we’ve gotten into the phase of
Dialogue: 0,0:13:19.09,0:13:20.08,Default,,0000,0000,0000,,“We’re testing as we go”
Dialogue: 0,0:13:20.08,0:13:22.05,Default,,0000,0000,0000,,Testing is really more like a development tool
Dialogue: 0,0:13:22.05,0:13:24.02,Default,,0000,0000,0000,,and like so many development tools
Dialogue: 0,0:13:24.02,0:13:25.06,Default,,0000,0000,0000,,the effectiveness of it depends
Dialogue: 0,0:13:25.06,0:13:27.01,Default,,0000,0000,0000,,on whether you’re using it in a tasteful manner
Dialogue: 0,0:13:27.01,0:13:31.00,Default,,0000,0000,0000,,So, you could say: “Well, let’s see—I kicked the tires
Dialogue: 0,0:13:31.00,0:13:33.05,Default,,0000,0000,0000,,You know, I fired up the browser, I tried a couple of things
Dialogue: 0,0:13:33.05,0:13:35.10,Default,,0000,0000,0000,,(claps hand) Looks like it works! Deploy it!”
Dialogue: 0,0:13:35.10,0:13:38.04,Default,,0000,0000,0000,,That’s obviously a little more cavalier than you’d want to be
Dialogue: 0,0:13:38.04,0:13:41.02,Default,,0000,0000,0000,,And, by the way, one of the things that we discovered
Dialogue: 0,0:13:41.02,0:13:43.08,Default,,0000,0000,0000,,with this online course just starting up
Dialogue: 0,0:13:43.08,0:13:45.09,Default,,0000,0000,0000,,when 60,000 people are enrolled in the course
Dialogue: 0,0:13:45.09,0:13:48.10,Default,,0000,0000,0000,,and 0.1% of those people have a problem
Dialogue: 0,0:13:48.10,0:13:50.08,Default,,0000,0000,0000,,you’d get 60 emails
Dialogue: 0,0:13:50.08,0:13:53.08,Default,,0000,0000,0000,,The corollary is: when your site is used by a lot of people
Dialogue: 0,0:13:53.08,0:13:55.09,Default,,0000,0000,0000,,some stupid bug that you didn’t find
Dialogue: 0,0:13:55.09,0:13:57.02,Default,,0000,0000,0000,,but that could have found by testing
Dialogue: 0,0:13:57.02,0:13:59.08,Default,,0000,0000,0000,,could very quickly generate *a lot* of pain
Dialogue: 0,0:13:59.08,0:14:02.02,Default,,0000,0000,0000,,On the other hand, you don’t want to be dogmatic and say
Dialogue: 0,0:14:02.02,0:14:04.06,Default,,0000,0000,0000,,“Uh, until we have 100% coverage and every test is green
Dialogue: 0,0:14:04.06,0:14:06.00,Default,,0000,0000,0000,,we absolutely will not ship”
Dialogue: 0,0:14:06.00,0:14:07.01,Default,,0000,0000,0000,,That’s not healthy either
Dialogue: 0,0:14:07.01,0:14:08.05,Default,,0000,0000,0000,,And the test quality
Dialogue: 0,0:14:08.05,0:14:10.06,Default,,0000,0000,0000,,doesn’t necessarily correlate with the statement
Dialogue: 0,0:14:10.06,0:14:11.06,Default,,0000,0000,0000,,unless you can say something
Dialogue: 0,0:14:11.06,0:14:12.07,Default,,0000,0000,0000,,about the quality of your tests
Dialogue: 0,0:14:12.07,0:14:14.03,Default,,0000,0000,0000,,just because you’ve executed every line
Dialogue: 0,0:14:14.03,0:14:17.01,Default,,0000,0000,0000,,doesn’t mean that you’ve tested the interesting cases
Dialogue: 0,0:14:17.01,0:14:18.07,Default,,0000,0000,0000,,So, somewhere in between, you could say
Dialogue: 0,0:14:18.07,0:14:20.01,Default,,0000,0000,0000,,“Well, we’ll use coverage tools to identify
Dialogue: 0,0:14:20.01,0:14:23.00,Default,,0000,0000,0000,,undertested or poorly-tested parts of the code
Dialogue: 0,0:14:23.00,0:14:24.07,Default,,0000,0000,0000,,and we’ll use them as a guideline
Dialogue: 0,0:14:24.07,0:14:27.01,Default,,0000,0000,0000,,to sort of help improve our overall confidence level”
Dialogue: 0,0:14:27.01,0:14:29.02,Default,,0000,0000,0000,,But remember, Agile is about embracing change
Dialogue: 0,0:14:29.02,0:14:30.03,Default,,0000,0000,0000,,and dealing with it
Dialogue: 0,0:14:30.03,0:14:32.00,Default,,0000,0000,0000,,Part of change is things would change that will cause
Dialogue: 0,0:14:32.00,0:14:33.04,Default,,0000,0000,0000,,bugs that you didn’t foresee
Dialogue: 0,0:14:33.04,0:14:34.03,Default,,0000,0000,0000,,and the right reaction is:
Dialogue: 0,0:14:34.03,0:14:36.03,Default,,0000,0000,0000,,Be comfortable enough for the testing tools
Dialogue: 0,0:14:36.03,0:14:37.06,Default,,0000,0000,0000,,[so] that you can quickly find those bugs
Dialogue: 0,0:14:37.06,0:14:39.02,Default,,0000,0000,0000,,Write a test that reproduces that bug
Dialogue: 0,0:14:39.02,0:14:40.06,Default,,0000,0000,0000,,And then make the test green
Dialogue: 0,0:14:40.06,0:14:41.06,Default,,0000,0000,0000,,Then you’ll really fix it
Dialogue: 0,0:14:41.06,0:14:43.00,Default,,0000,0000,0000,,That means, the way that you really fix a bug is
Dialogue: 0,0:14:43.00,0:14:45.05,Default,,0000,0000,0000,,if you created a test that correctly failed
Dialogue: 0,0:14:45.05,0:14:46.09,Default,,0000,0000,0000,,to reproduce that bug
Dialogue: 0,0:14:46.09,0:14:48.06,Default,,0000,0000,0000,,and then you went back and fixed the code
Dialogue: 0,0:14:48.06,0:14:49.06,Default,,0000,0000,0000,,to make those tests pass
Dialogue: 0,0:14:49.06,0:14:51.07,Default,,0000,0000,0000,,Similarly, you don’t want to say
Dialogue: 0,0:14:51.07,0:14:53.04,Default,,0000,0000,0000,,“Well, unit tests give you better coverage
Dialogue: 0,0:14:53.04,0:14:54.07,Default,,0000,0000,0000,,They’re more thorough and detailed
Dialogue: 0,0:14:54.07,0:14:56.04,Default,,0000,0000,0000,,So let’s focus all our energy on that”
Dialogue: 0,0:14:56.04,0:14:57.06,Default,,0000,0000,0000,,as opposed to
Dialogue: 0,0:14:57.06,0:14:58.09,Default,,0000,0000,0000,,“Oh, focus on integration tests
Dialogue: 0,0:14:58.09,0:15:00.01,Default,,0000,0000,0000,,because they’re more realistic, right?
Dialogue: 0,0:15:00.01,0:15:01.06,Default,,0000,0000,0000,,They reflect what the customer said they want
Dialogue: 0,0:15:01.06,0:15:03.03,Default,,0000,0000,0000,,So, if the integration tests are passing
Dialogue: 0,0:15:03.03,0:15:05.07,Default,,0000,0000,0000,,by definition we’re meeting a customer need”
Dialogue: 0,0:15:05.07,0:15:07.03,Default,,0000,0000,0000,,Again, both extremes are kind of unhealthy
Dialogue: 0,0:15:07.03,0:15:09.08,Default,,0000,0000,0000,,because each one of these can find problems
Dialogue: 0,0:15:09.08,0:15:11.03,Default,,0000,0000,0000,,that would be missed by the other
Dialogue: 0,0:15:11.03,0:15:12.06,Default,,0000,0000,0000,,So, having a good combination of them
Dialogue: 0,0:15:12.06,0:15:15.04,Default,,0000,0000,0000,,is kind of all it is all about
Dialogue: 0,0:15:15.04,0:15:18.07,Default,,0000,0000,0000,,The last thing I want to leave you with is, I think
Dialogue: 0,0:15:18.07,0:15:20.04,Default,,0000,0000,0000,,in terms of testing, is “TDD versus
Dialogue: 0,0:15:20.04,0:15:22.00,Default,,0000,0000,0000,,what I call conventional debugging—
Dialogue: 0,0:15:22.00,0:15:24.00,Default,,0000,0000,0000,,i.e., the way that we all kind of do it
Dialogue: 0,0:15:24.00,0:15:25.05,Default,,0000,0000,0000,,even though we say we don’t”
Dialogue: 0,0:15:25.05,0:15:26.06,Default,,0000,0000,0000,,and we’re all trying to get better, right?
Dialogue: 0,0:15:26.06,0:15:27.08,Default,,0000,0000,0000,,We’re all kind of in the gutter
Dialogue: 0,0:15:27.08,0:15:29.04,Default,,0000,0000,0000,,Some of us are looking up at the stars
Dialogue: 0,0:15:29.04,0:15:31.01,Default,,0000,0000,0000,,trying to improve our practices
Dialogue: 0,0:15:31.01,0:15:33.10,Default,,0000,0000,0000,,But, having now lived with this for 3 or 4 years myself
Dialogue: 0,0:15:33.10,0:15:35.09,Default,,0000,0000,0000,,and—I’ll be honest—3 years ago I didn’t do TDD
Dialogue: 0,0:15:35.09,0:15:37.08,Default,,0000,0000,0000,,I do it now, because I find that it’s better
Dialogue: 0,0:15:37.08,0:15:40.08,Default,,0000,0000,0000,,and here’s my distillation of why I think it works for me
Dialogue: 0,0:15:40.08,0:15:43.03,Default,,0000,0000,0000,,Sorry, the colours are a little weird
Dialogue: 0,0:15:43.03,0:15:45.00,Default,,0000,0000,0000,,but on the left column of the table
Dialogue: 0,0:15:45.00,0:15:46.03,Default,,0000,0000,0000,,[it] says “Conventional debugging”
Dialogue: 0,0:15:46.03,0:15:47.04,Default,,0000,0000,0000,,and the right side says “TDD”
Dialogue: 0,0:15:47.04,0:15:49.07,Default,,0000,0000,0000,,So what’s the way I used to write code?
Dialogue: 0,0:15:49.07,0:15:51.06,Default,,0000,0000,0000,,Maybe some of you still do this
Dialogue: 0,0:15:51.06,0:15:53.01,Default,,0000,0000,0000,,I write a whole bunch of lines
Dialogue: 0,0:15:53.01,0:15:54.04,Default,,0000,0000,0000,,maybe a few tens of lines of code
Dialogue: 0,0:15:54.04,0:15:55.06,Default,,0000,0000,0000,,I’m {\i1}sure{\i0} they’re right—
Dialogue: 0,0:15:55.06,0:15:56.06,Default,,0000,0000,0000,,I mean, I {\i1}am{\i0} a good programmer, right?
Dialogue: 0,0:15:56.06,0:15:57.10,Default,,0000,0000,0000,,This is not that hard
Dialogue: 0,0:15:57.10,0:15:59.00,Default,,0000,0000,0000,,I run it – It doesn’t work
Dialogue: 0,0:15:59.00,0:16:01.10,Default,,0000,0000,0000,,Ok, fire up the debugger – Start putting in printf’s
Dialogue: 0,0:16:01.10,0:16:04.09,Default,,0000,0000,0000,,If I’d been using TDD what would I do instead?
Dialogue: 0,0:16:04.09,0:16:08.02,Default,,0000,0000,0000,,Well I’d write a {\i1}few{\i0} lines of code, having written a test first
Dialogue: 0,0:16:08.02,0:16:10.07,Default,,0000,0000,0000,,So as soon as the test goes from red to green
Dialogue: 0,0:16:10.07,0:16:12.06,Default,,0000,0000,0000,,I know I wrote code that works—
Dialogue: 0,0:16:12.06,0:16:15.01,Default,,0000,0000,0000,,or at least the parts of the behaviour that I had in mind
Dialogue: 0,0:16:15.01,0:16:16.10,Default,,0000,0000,0000,,Those parts of the behaviour work, because I had a test
Dialogue: 0,0:16:16.10,0:16:19.06,Default,,0000,0000,0000,,Ok, back to conventional debugging:
Dialogue: 0,0:16:19.06,0:16:21.07,Default,,0000,0000,0000,,I’m running my program, trying to find the bugs
Dialogue: 0,0:16:21.07,0:16:23.03,Default,,0000,0000,0000,,I start putting in printf’s everywhere
Dialogue: 0,0:16:23.03,0:16:24.06,Default,,0000,0000,0000,,to print out the values of things
Dialogue: 0,0:16:24.06,0:16:25.06,Default,,0000,0000,0000,,which by the way is a lot fun
Dialogue: 0,0:16:25.06,0:16:26.07,Default,,0000,0000,0000,,when you’re trying to read them
Dialogue: 0,0:16:26.07,0:16:28.01,Default,,0000,0000,0000,,out of the 500 lines of log output
Dialogue: 0,0:16:28.01,0:16:29.04,Default,,0000,0000,0000,,that you’d get in a Rails app
Dialogue: 0,0:16:29.04,0:16:30.09,Default,,0000,0000,0000,,trying to find {\i1}your{\i0} printf’s
Dialogue: 0,0:16:30.09,0:16:32.04,Default,,0000,0000,0000,,you know, “I know what I’ll do—
Dialogue: 0,0:16:32.04,0:16:34.01,Default,,0000,0000,0000,,I’ll put in 75 asterisks before and after
Dialogue: 0,0:16:34.01,0:16:36.04,Default,,0000,0000,0000,,That will make it readable” (laughter)
Dialogue: 0,0:16:36.04,0:16:38.07,Default,,0000,0000,0000,,Who don’t—Ok, raise your hands if you don’t do this!
Dialogue: 0,0:16:38.07,0:16:40.09,Default,,0000,0000,0000,,Thank you for your honesty. (laughter) Ok.
Dialogue: 0,0:16:40.09,0:16:43.01,Default,,0000,0000,0000,,Or— Or I could do the other thing, I could say:
Dialogue: 0,0:16:43.01,0:16:45.03,Default,,0000,0000,0000,,Instead of printing the value of a variable
Dialogue: 0,0:16:45.03,0:16:47.04,Default,,0000,0000,0000,,why don’t I write a test that inspects it
Dialogue: 0,0:16:47.04,0:16:48.08,Default,,0000,0000,0000,,in such an expectation which should
Dialogue: 0,0:16:48.08,0:16:50.09,Default,,0000,0000,0000,,and I’ll know immediately in bright red letters
Dialogue: 0,0:16:50.09,0:16:53.03,Default,,0000,0000,0000,,if that expectation wasn’t met
Dialogue: 0,0:16:53.03,0:16:56.00,Default,,0000,0000,0000,,Ok, I’m back on the conventional debugging side:
Dialogue: 0,0:16:56.00,0:16:58.09,Default,,0000,0000,0000,,I break out the big guns: I pull out the Ruby debugger
Dialogue: 0,0:16:58.09,0:17:02.04,Default,,0000,0000,0000,,I set a debug breakpoint, and I now start {\i1}tweaking{\i0} and say
Dialogue: 0,0:17:02.04,0:17:04.08,Default,,0000,0000,0000,,“Oh, let’s see, I have to get past that ‘if’ statement
Dialogue: 0,0:17:04.08,0:17:06.00,Default,,0000,0000,0000,,so I have to set that thing
Dialogue: 0,0:17:06.00,0:17:07.06,Default,,0000,0000,0000,,Oh, I have to call that method and so I need to…”
Dialogue: 0,0:17:07.06,0:17:08.06,Default,,0000,0000,0000,,No!
Dialogue: 0,0:17:08.06,0:17:10.09,Default,,0000,0000,0000,,I {\i1}could{\i0} instead—if I’m going to do that anyway—
Dialogue: 0,0:17:10.09,0:17:13.00,Default,,0000,0000,0000,,let’s just do it in a file, set up some mocks and stubs
Dialogue: 0,0:17:13.00,0:17:16.04,Default,,0000,0000,0000,,to control the code path, make it go the way I want
Dialogue: 0,0:17:16.04,0:17:19.01,Default,,0000,0000,0000,,And now, “Ok, for sure I’ve fixed it!
Dialogue: 0,0:17:19.01,0:17:22.01,Default,,0000,0000,0000,,I’ll get out of the debugger, run it all again!”
Dialogue: 0,0:17:22.01,0:17:24.02,Default,,0000,0000,0000,,And, of course, 9 times out of 10, you didn’t fix it
Dialogue: 0,0:17:24.02,0:17:26.07,Default,,0000,0000,0000,,or you kind of partly fixed it but you didn’t completely fix it
Dialogue: 0,0:17:26.07,0:17:30.04,Default,,0000,0000,0000,,and now I have to do all these manual things all over again
Dialogue: 0,0:17:30.04,0:17:32.09,Default,,0000,0000,0000,,{\i1}or{\i0} I already have a bunch of tests
Dialogue: 0,0:17:32.09,0:17:34.03,Default,,0000,0000,0000,,and I can just rerun them automatically
Dialogue: 0,0:17:34.03,0:17:35.06,Default,,0000,0000,0000,,and I could, if some of them fail
Dialogue: 0,0:17:35.06,0:17:36.09,Default,,0000,0000,0000,,“Oh, I didn’t fix the whole thing
Dialogue: 0,0:17:36.09,0:17:38.04,Default,,0000,0000,0000,,No problem, I’ll just go back!”
Dialogue: 0,0:17:38.04,0:17:39.10,Default,,0000,0000,0000,,So, the bottom line is that
Dialogue: 0,0:17:39.10,0:17:41.10,Default,,0000,0000,0000,,you know, you {\i1}could{\i0} do it on the left side
Dialogue: 0,0:17:41.10,0:17:45.00,Default,,0000,0000,0000,,but you’re using the same techniques in both cases
Dialogue: 0,0:17:45.00,0:17:48.06,Default,,0000,0000,0000,,The only difference is, in one case you’re doing it manually
Dialogue: 0,0:17:48.06,0:17:50.00,Default,,0000,0000,0000,,which is boring and error-prone
Dialogue: 0,0:17:50.00,0:17:51.08,Default,,0000,0000,0000,,In the other case you’re doing a little more work
Dialogue: 0,0:17:51.08,0:17:53.10,Default,,0000,0000,0000,,but you can make it automatic and repeatable
Dialogue: 0,0:17:53.10,0:17:55.07,Default,,0000,0000,0000,,and have, you know, some high confidence
Dialogue: 0,0:17:55.07,0:17:57.00,Default,,0000,0000,0000,,that as you change things in your code
Dialogue: 0,0:17:57.00,0:17:58.09,Default,,0000,0000,0000,,you are not breaking stuff that used to work
Dialogue: 0,0:17:58.09,0:18:00.09,Default,,0000,0000,0000,,and basically it’s more productive
Dialogue: 0,0:18:00.09,0:18:02.05,Default,,0000,0000,0000,,So you’re doing all the same things
Dialogue: 0,0:18:02.05,0:18:04.04,Default,,0000,0000,0000,,but with a, kind of, “delta” extra work
Dialogue: 0,0:18:04.04,0:18:07.09,Default,,0000,0000,0000,,you are using your effort at a much higher leverage
Dialogue: 0,0:18:07.09,0:18:10.04,Default,,0000,0000,0000,,So that’s kind of my view of why TDD is a good thing
Dialogue: 0,0:18:10.04,0:18:11.09,Default,,0000,0000,0000,,It’s really, it doesn’t require new skills
Dialogue: 0,0:18:11.09,0:18:15.01,Default,,0000,0000,0000,,It just requires [you] to refactor your existing skills
Dialogue: 0,0:18:15.01,0:18:18.01,Default,,0000,0000,0000,,I also tried when I—again, honest confessions, right?—
Dialogue: 0,0:18:18.01,0:18:19.03,Default,,0000,0000,0000,,when I started doing this it was like
Dialogue: 0,0:18:19.03,0:18:21.05,Default,,0000,0000,0000,,“Ok, I gonna be teaching a course on Rails
Dialogue: 0,0:18:21.05,0:18:22.06,Default,,0000,0000,0000,,I should really focus on testing
Dialogue: 0,0:18:22.06,0:18:24.03,Default,,0000,0000,0000,,So I went back to some code I had written
Dialogue: 0,0:18:24.03,0:18:26.09,Default,,0000,0000,0000,,that was {\i1}working{\i0}—you know, that was decent code—
Dialogue: 0,0:18:26.09,0:18:29.01,Default,,0000,0000,0000,,and I started trying to write tests for it
Dialogue: 0,0:18:29.01,0:18:31.02,Default,,0000,0000,0000,,and it was *so painful*
Dialogue: 0,0:18:31.02,0:18:33.03,Default,,0000,0000,0000,,because the code wasn’t written in way that was testable
Dialogue: 0,0:18:33.03,0:18:34.10,Default,,0000,0000,0000,,There were all kinds of interactions
Dialogue: 0,0:18:34.10,0:18:36.04,Default,,0000,0000,0000,,There were, like, nested conditionals
Dialogue: 0,0:18:36.04,0:18:38.08,Default,,0000,0000,0000,,And if you wanted to isolate a particular statement
Dialogue: 0,0:18:38.08,0:18:41.07,Default,,0000,0000,0000,,and have it test—to trigger test—just that statement
Dialogue: 0,0:18:41.07,0:18:44.00,Default,,0000,0000,0000,,the amount of stuff you’d have to set up in your test
Dialogue: 0,0:18:44.00,0:18:45.01,Default,,0000,0000,0000,,to have it happen—
Dialogue: 0,0:18:45.01,0:18:46.04,Default,,0000,0000,0000,,remember when talked about mock train wrecks—
Dialogue: 0,0:18:46.04,0:18:48.01,Default,,0000,0000,0000,,you have to set up all this infrastructure
Dialogue: 0,0:18:48.01,0:18:49.06,Default,,0000,0000,0000,,just to get {\i1}one{\i0} line of code
Dialogue: 0,0:18:49.06,0:18:51.02,Default,,0000,0000,0000,,and you do that and you go
Dialogue: 0,0:18:51.02,0:18:52.07,Default,,0000,0000,0000,,“Gawd, testing is really not worth it!
Dialogue: 0,0:18:52.07,0:18:54.03,Default,,0000,0000,0000,,I wrote 20 lines of setup
Dialogue: 0,0:18:54.03,0:18:56.06,Default,,0000,0000,0000,,so that I could test two lines in my function!”
Dialogue: 0,0:18:56.06,0:18:58.08,Default,,0000,0000,0000,,What that’s really telling you—as I now realize—
Dialogue: 0,0:18:58.08,0:19:00.04,Default,,0000,0000,0000,,is your function is {\i1}bad{\i0}
Dialogue: 0,0:19:00.04,0:19:01.05,Default,,0000,0000,0000,,It’s a badly written function
Dialogue: 0,0:19:01.05,0:19:02.05,Default,,0000,0000,0000,,It’s not a testable function
Dialogue: 0,0:19:02.05,0:19:03.09,Default,,0000,0000,0000,,It’s got too many moving parts
Dialogue: 0,0:19:03.09,0:19:06.03,Default,,0000,0000,0000,,whose dependencies {\i1}can{\i0} be broken
Dialogue: 0,0:19:06.03,0:19:07.07,Default,,0000,0000,0000,,There’s no seams in my function
Dialogue: 0,0:19:07.07,0:19:11.01,Default,,0000,0000,0000,,that allow me to individually test the different behaviours
Dialogue: 0,0:19:11.01,0:19:12.08,Default,,0000,0000,0000,,And once you start doing Test First Development
Dialogue: 0,0:19:12.08,0:19:15.04,Default,,0000,0000,0000,,because you have to write your tests in small chunks
Dialogue: 0,0:19:15.04,0:19:17.05,Default,,0000,0000,0000,,it kind of make this problem go away
Dialogue: 0,0:19:17.05,9:59:59.99,Default,,0000,0000,0000,,So that’s been my epiphany