0:00:15.088,0:00:21.142
So over the past few centuries,[br]microscopes have revolutionized our world.

0:00:23.204,0:00:27.563
They revealed to us a tiny world[br]of objects, life and structures

0:00:27.563,0:00:30.567
that are too small for us[br]to see with our naked eyes.

0:00:30.567,0:00:34.064
They are a tremendous contribution[br]to science and technology.

0:00:34.064,0:00:37.839
Today I'd like to introduce you[br]to a new type of microscope,

0:00:37.839,0:00:40.230
a microscope for changes.

0:00:40.230,0:00:43.438
It doesn't use optics[br]like a regular microscope

0:00:43.438,0:00:45.350
to make small objects bigger,

0:00:45.350,0:00:49.673
but instead it uses a video camera[br]and image processing

0:00:49.673,0:00:54.387
to reveal to us the tiniest motions[br]and color changes in objects and people,

0:00:54.526,0:00:58.368
changes that are impossible[br]for us to see with our naked eyes.

0:00:58.906,0:01:02.970
And it lets us look at our world[br]in a completely new way.

0:01:03.295,0:01:05.715
So what do I mean by color changes?

0:01:07.093,0:01:09.896
Our skin, for example,[br]changes its color very slightly

0:01:09.896,0:01:12.211
when the blood flows under it.

0:01:12.211,0:01:14.443
That change is incredibly subtle,

0:01:14.443,0:01:16.872
which is why, when you look[br]at other people,

0:01:16.872,0:01:19.171
when you look at the person[br]sitting next to you,

0:01:19.171,0:01:21.920
you don't see their skin[br]or their face changing color.

0:01:21.920,0:01:27.150
When we look at this video of Steve here,[br]it appears to us like a static picture,

0:01:27.849,0:01:31.292
but once we look at this video[br]through our new, special microscope,

0:01:31.292,0:01:34.550
suddenly we see[br]a completely different image.

0:01:35.276,0:01:39.200
What you see here are small changes[br]in the color of Steve's skin,

0:01:39.200,0:01:43.087
magnified 100 times[br]so that they become visible.

0:01:43.539,0:01:46.259
We can actually see a human pulse.

0:01:46.554,0:01:49.729
We can see how fast[br]Steve's heart is beating,

0:01:49.729,0:01:53.994
but we can also see the actual way[br]that the blood flows in his face.

0:01:55.152,0:01:58.055
And we can do that not just[br]to visualize the pulse,

0:01:58.055,0:02:01.448
but also to actually recover[br]our heart rates,

0:02:01.448,0:02:03.510
and measure our heart rates.

0:02:03.510,0:02:07.692
And we can do it with regular cameras[br]and without touching the patients.

0:02:07.692,0:02:12.768
So here you see the pulse and heart rate[br]we extracted from a neonatal baby

0:02:12.768,0:02:16.250
from a video we took[br]with a regular DSLR camera,

0:02:16.250,0:02:18.364
and the heart rate measurement we get

0:02:18.364,0:02:22.588
is as accurate as the one you'd get[br]with a standard monitor in a hospital.

0:02:23.384,0:02:26.457
And it doesn't even have to be[br]a video we recorded.

0:02:26.457,0:02:29.391
We can do it essentially[br]with other videos as well.

0:02:29.391,0:02:32.565
So I just took a short clip[br]from "Batman Begins" here

0:02:32.565,0:02:35.226
just to show Christian Bale's pulse.

0:02:35.226,0:02:37.282
(Laughter)

0:02:37.282,0:02:39.417
And you know, presumably[br]he's wearing makeup,

0:02:39.417,0:02:41.389
the lighting here is kind of challenging,

0:02:41.389,0:02:44.392
but still, just from the video,[br]we're able to extract his pulse

0:02:44.392,0:02:46.224
and show it quite well.

0:02:46.224,0:02:47.995
So how do we do all that?

0:02:47.995,0:02:52.315
We basically analyze the changes[br]in the light that are recorded

0:02:52.315,0:02:54.928
at every pixel in the video over time,

0:02:54.928,0:02:56.648
and then we crank up those changes.

0:02:56.648,0:02:59.494
We make them bigger[br]so that we can see them.

0:02:59.494,0:03:01.576
The tricky part is that those signals,

0:03:01.576,0:03:04.359
those changes that we're after,[br]are extremely subtle,

0:03:04.359,0:03:07.353
so we have to be very careful[br]when you try to separate them

0:03:07.353,0:03:10.240
from noise that always exists in videos.

0:03:10.240,0:03:13.682
So we use some clever[br]image processing techniques

0:03:13.682,0:03:17.736
to get a very accurate measurement[br]of the color at each pixel in the video,

0:03:17.736,0:03:20.569
and then the way[br]the color changes over time,

0:03:20.569,0:03:23.227
and then we amplify those changes.

0:03:23.227,0:03:27.081
We make them bigger to create those types[br]of enhanced videos, or magnified videos,

0:03:27.081,0:03:29.552
that actually show us those changes.

0:03:32.007,0:03:36.227
But it turns out we can do that[br]not just to show tiny changes in color,

0:03:36.227,0:03:38.379
but also tiny motions,

0:03:38.379,0:03:42.064
and that's because the light[br]that gets recorded in our cameras

0:03:42.064,0:03:45.189
will change not only if the color[br]of the object changes,

0:03:45.189,0:03:47.305
but also if the object moves.

0:03:47.905,0:03:52.543
So this is my daughter[br]when she was about two months old.

0:03:56.157,0:03:59.397
It's a video I recorded[br]about three years ago.

0:03:59.397,0:04:02.807
And as new parents, we all want[br]to make sure our babies are healthy,

0:04:02.807,0:04:05.373
that they're breathing,[br]that they're alive, of course.

0:04:05.373,0:04:07.465
So I too got one of those baby monitors

0:04:07.465,0:04:10.072
so that I could see my daughter[br]when she was asleep.

0:04:10.072,0:04:13.590
And this is pretty much what you'll see[br]with a standard baby monitor.

0:04:13.590,0:04:15.686
You can see the baby's sleeping,

0:04:15.686,0:04:17.729
but there's not too much information[br]there.

0:04:17.729,0:04:19.516
There's not too much we can see.

0:04:19.516,0:04:22.358
Wouldn't it be better,[br]or more informative, or more useful,

0:04:22.358,0:04:25.261
if instead we could look[br]at the view like this.

0:04:25.261,0:04:30.310
So here I took the motions[br]and I magnified them 30 times,

0:04:31.217,0:04:33.708
and then I could clearly see[br]that my daughter

0:04:33.708,0:04:35.428
was indeed alive and breathing.

0:04:35.428,0:04:37.565
(Laughter)

0:04:38.092,0:04:39.891
Here is a side-by-side comparison.

0:04:39.891,0:04:42.440
So again, in the source video,[br]in the original video,

0:04:42.440,0:04:44.260
there's not too much we can see,

0:04:44.260,0:04:48.212
but once we magnify the motions,[br]the breathing becomes much more visible.

0:04:48.212,0:04:50.801
And it turns out,[br]there's a lot of phenomena

0:04:50.801,0:04:54.474
we can reveal and magnify[br]with our new motion microscope.

0:04:54.474,0:04:58.909
We can see how our veins and arteries[br]are pulsing in our bodies.

0:04:59.752,0:05:02.560
We can see that our eyes[br]are constantly moving

0:05:02.560,0:05:04.776
in this wobbly motion.

0:05:04.776,0:05:06.354
And that's actually my eye,

0:05:06.354,0:05:09.414
and again this video was taken[br]right after my daughter was born,

0:05:09.414,0:05:13.103
so you can see I wasn't getting[br]too much sleep. (Laughter)

0:05:13.539,0:05:16.403
Even when a person is sitting still,

0:05:16.403,0:05:18.997
there's a lot of information[br]we can extract

0:05:18.997,0:05:21.989
about their breathing patterns,[br]small facial expressions.

0:05:22.672,0:05:24.623
Maybe we could use those motions

0:05:24.623,0:05:27.488
to tell us something about[br]our thoughts or our emotions.

0:05:29.003,0:05:32.385
We can also magnify[br]small mechanical movements,

0:05:32.385,0:05:34.337
like vibrations in engines,

0:05:34.337,0:05:38.017
that can help engineers detect[br]and diagnose machinery problems,

0:05:40.130,0:05:45.547
or see how our buildings and structures[br]sway in the wind and react to forces.

0:05:45.547,0:05:50.333
Those are all things that our society[br]knows how to measure in various ways,

0:05:50.333,0:05:52.875
but measuring those motions is one thing,

0:05:52.875,0:05:55.479
and actually seeing those motions[br]as they happen

0:05:55.479,0:05:57.614
is a whole different thing.

0:05:58.450,0:06:02.021
And ever since we discovered[br]this new technology,

0:06:02.021,0:06:03.723
we made our code available online

0:06:03.723,0:06:06.269
so that others could use[br]and experiment with it.

0:06:08.005,0:06:09.809
It's very simple to use.

0:06:09.809,0:06:11.853
It can work on your own videos.

0:06:11.853,0:06:15.357
Our collaborators at Quantum Research[br]even created this nice website

0:06:15.357,0:06:18.036
where you can upload your videos[br]and process them online,

0:06:18.036,0:06:21.603
so even if you don't have any experience[br]in computer science or programming,

0:06:21.603,0:06:24.509
you can still very easily experiment[br]with this new microscope.

0:06:24.509,0:06:26.941
And I'd like to show you[br]just a couple of examples

0:06:26.941,0:06:28.919
of what others have done with it.

0:06:32.363,0:06:37.259
So this video was made[br]by a YouTube user called Tamez85.

0:06:37.259,0:06:38.807
I don't know who that user is,

0:06:38.807,0:06:41.105
but he, or she, used our code

0:06:41.105,0:06:43.910
to magnify small belly movements[br]during pregnancy.

0:06:44.933,0:06:46.420
It's kind of creepy.

0:06:46.420,0:06:48.818
(Laughter)

0:06:48.818,0:06:52.782
People have used it to magnify[br]pulsing veins in their hands.

0:06:53.532,0:06:56.699
And you know it's not real science[br]unless you use guinea pigs,

0:06:58.037,0:07:00.584
and apparently this guinea pig[br]is called Tiffany,

0:07:00.584,0:07:04.007
and this YouTube user claims[br]it is the first rodent on Earth

0:07:04.007,0:07:05.780
that was motion-magnified.

0:07:06.604,0:07:08.811
You can also do some art with it.

0:07:08.811,0:07:12.123
So this video was sent to me[br]by a design student at Yale.

0:07:12.123,0:07:14.516
She wanted to see[br]if there's any difference

0:07:14.516,0:07:16.072
in the way her classmates move.

0:07:16.072,0:07:20.361
She made them all stand still,[br]and then magnified their motions.

0:07:20.361,0:07:23.457
It's like seeing still pictures[br]come to life.

0:07:23.714,0:07:26.077
And the nice thing with all those examples

0:07:26.077,0:07:28.315
is that we had nothing to do with them.

0:07:28.315,0:07:32.165
We just provided this new tool,[br]a new way to look at the world,

0:07:32.165,0:07:36.683
and then people find other interesting,[br]new and creative ways of using it.

0:07:37.735,0:07:39.620
But we didn't stop there.

0:07:40.943,0:07:44.597
This tool not only allows us[br]to look at the world in a new way,

0:07:44.597,0:07:47.034
it also redefines what we can do

0:07:47.034,0:07:50.232
and pushes the limits[br]of what we can do with our cameras.

0:07:50.232,0:07:52.611
So as scientists, we started wondering,

0:07:52.611,0:07:56.299
what other types of physical phenomena[br]produce tiny motions

0:07:56.299,0:07:59.212
that we could now use[br]our cameras to measure?

0:07:59.212,0:08:02.635
And one such phenomenon[br]that we focused on recently is sound.

0:08:03.664,0:08:05.963
Sound, as we all know,[br]is basically changes

0:08:05.963,0:08:08.134
in air pressure[br]that travel through the air.

0:08:08.134,0:08:11.857
Those pressure waves hit objects[br]and they create small vibrations in them,

0:08:11.857,0:08:14.519
which is how we hear[br]and how we record sound.

0:08:14.519,0:08:18.294
But it turns out that sound[br]also produces visual motions.

0:08:18.579,0:08:21.303
Those are motions[br]that are not visible to us

0:08:21.303,0:08:24.229
but are visible to a camera[br]with the right processing.

0:08:24.229,0:08:26.045
So here are two examples.

0:08:26.045,0:08:29.374
This is me demonstrating[br]my great singing skills.

0:08:30.845,0:08:33.603
(Singing)

0:08:33.603,0:08:34.710
(Laughter)

0:08:34.710,0:08:37.706
And I took a high-speed video[br]of my throat while I was humming.

0:08:37.706,0:08:39.355
Again, if you stare at that video,

0:08:39.355,0:08:41.387
there's not too much[br]you'll be able to see,

0:08:41.387,0:08:45.793
but once we magnify the motions 100 times,[br]we can see all the motions and ripples

0:08:45.793,0:08:49.103
in the neck that are involved[br]in producing the sound.

0:08:49.103,0:08:51.528
That signal is there in that video.

0:08:51.528,0:08:54.228
We also know that singers[br]can break a wine glass

0:08:54.228,0:08:56.274
if they hit the correct note.

0:08:56.274,0:08:58.325
So here, we're going to play a note

0:08:58.325,0:09:00.849
that's in the resonance frequency[br]of that glass

0:09:00.849,0:09:03.125
through a loudspeaker that's next to it.

0:09:03.125,0:09:07.568
Once we play that note[br]and magnify the motions 250 times,

0:09:07.568,0:09:10.789
we can very clearly see[br]how the glass vibrates

0:09:10.789,0:09:13.623
and resonates in response to the sound.

0:09:14.132,0:09:16.545
It's not something you're used to seeing[br]every day.

0:09:16.545,0:09:19.408
And we actually have the demo[br]right outside set up,

0:09:19.408,0:09:21.300
so I encourage you to stop by,

0:09:21.300,0:09:24.347
and just play with it yourself,[br]you can actually see it live.

0:09:24.608,0:09:27.768
But this made us think.[br]It gave us this crazy idea.

0:09:28.078,0:09:32.865
Can we actually invert this process[br]and recover sound from video

0:09:33.454,0:09:37.581
by analyzing the tiny vibrations[br]that sound waves create in objects,

0:09:37.581,0:09:41.898
and essentially convert those[br]back into the sounds that produced them.

0:09:42.548,0:09:46.472
In this way, we can turn[br]everyday objects into microphones.

0:09:47.958,0:09:49.595
So that's exactly what we did.

0:09:49.595,0:09:52.462
So here's an empty bag of chips[br]that was lying on a table,

0:09:52.462,0:09:55.234
and we're going to turn that bag of chips[br]into a microphone

0:09:55.234,0:09:57.145
by filming it with a video camera

0:09:57.145,0:10:00.914
and analyzing the tiny motions[br]that sound waves create in it.

0:10:01.479,0:10:04.242
So here's the sound[br]that we played in the room.

0:10:04.242,0:10:07.634
(Music: "Mary Had a Little Lamb")

0:10:12.476,0:10:15.426
And this is a high-speed video[br]we recorded of that bag of chips.

0:10:15.426,0:10:16.528
Again it's playing.

0:10:16.528,0:10:19.886
There's no chance you'll be able[br]to see anything going on in that video

0:10:19.886,0:10:21.000
just by looking at it,

0:10:21.000,0:10:23.962
but here's the sound we were able[br]to recover just by analyzing

0:10:23.962,0:10:26.273
the tiny motions in that video.

0:10:27.127,0:10:30.494
(Music: "Mary Had a Little Lamb")

0:10:44.607,0:10:46.458
I call it -- Thank you.

0:10:46.458,0:10:49.328
(Applause)

0:10:53.834,0:10:56.140
I call it the visual microphone.

0:10:56.140,0:10:59.251
We actually extract audio signals[br]from video signals.

0:10:59.251,0:11:02.435
And just to give you a sense[br]of the scale of the motions here,

0:11:02.435,0:11:06.696
a pretty loud sound will cause[br]that bag of chips

0:11:06.696,0:11:09.266
to move less than a micrometer.

0:11:09.807,0:11:12.485
That's one thousandth of a millimeter.

0:11:12.485,0:11:16.179
That's how tiny the motions are[br]that we are now able to pull out

0:11:16.179,0:11:19.282
just by observing how light[br]bounces off objects

0:11:19.282,0:11:21.704
and gets recorded by our cameras.

0:11:22.208,0:11:25.358
We can recover sounds[br]from other objects, like plants.

0:11:25.986,0:11:29.183
(Music: "Mary Had a Little Lamb")

0:11:34.153,0:11:36.456
And we can recover speech as well.

0:11:36.456,0:11:38.817
So here's a person speaking in a room.

0:11:38.817,0:11:43.632
Voice: Mary had a little lamb[br]whose fleece was white as snow,

0:11:43.632,0:11:47.570
and everywhere that Mary went,[br]that lamb was sure to go.

0:11:48.722,0:11:51.486
Michael Rubinstein: And here's[br]that speech again recovered

0:11:51.486,0:11:54.220
just from this video[br]of that same bag of chips.

0:11:54.220,0:11:59.211
Voice: Mary had a little lamb[br]whose fleece was white as snow,

0:11:59.211,0:12:03.731
and everywhere that Mary went,[br]that lamb was sure to go.

0:12:04.352,0:12:06.907
MR: We used "Mary Had a Little Lamb"

0:12:06.907,0:12:09.252
because those are said to be[br]the first words

0:12:09.252,0:12:13.053
that Thomas Edison spoke[br]into his phonograph in 1877.

0:12:13.053,0:12:16.565
It was one of the first[br]sound recording devices in history.

0:12:16.565,0:12:19.842
It basically directed the sounds[br]onto a diaphragm

0:12:19.842,0:12:24.270
that vibrated a needle that essentially[br]engraved the sound on tinfoil

0:12:24.270,0:12:26.565
that was wrapped around the cylinder.

0:12:26.565,0:12:29.679
Here's a demonstration of recording

0:12:29.679,0:12:32.446
and replaying sound[br]with Edison's phonograph.

0:12:33.549,0:12:36.487
(Video) Voice: Testing, testing,[br]one two three.

0:12:36.487,0:12:39.654
Mary had a little lamb[br]whose fleece was white as snow,

0:12:39.654,0:12:43.491
and everywhere that Mary went,[br]the lamb was sure to go.

0:12:43.491,0:12:46.014
Testing, testing, one two three.

0:12:46.014,0:12:50.103
Mary had a little lamb[br]whose fleece was white as snow,

0:12:50.103,0:12:54.235
and everywhere that Mary went,[br]the lamb was sure to go.

0:12:55.719,0:12:59.081
MR: And now, 137 years later,

0:13:00.334,0:13:03.492
we're able to get sound[br]in pretty much similar quality

0:13:03.492,0:13:07.559
but by just watching objects[br]vibrate to sound with cameras,

0:13:07.853,0:13:09.952
and we can even do that when the camera

0:13:09.952,0:13:13.631
is 15 feet away from the object,[br]behind soundproof glass.

0:13:14.178,0:13:17.475
So this is the sound that we were able[br]to recover in that case.

0:13:17.475,0:13:22.282
Voice: Mary had a little lamb[br]whose fleece was white as snow,

0:13:22.282,0:13:26.993
and everywhere that Mary went,[br]the lamb was sure to go.

0:13:28.111,0:13:31.711
MR: And of course, surveillance is[br]the first application that comes to mind.

0:13:31.711,0:13:33.993
(Laughter)

0:13:33.993,0:13:38.055
But it might actually be useful[br]for other things as well.

0:13:38.095,0:13:41.196
Maybe in the future,[br]we'll be able to use it, for example,

0:13:41.196,0:13:43.557
to recover sound across space,

0:13:43.557,0:13:46.569
because sound can't travel[br]in space, but light can.

0:13:47.166,0:13:49.525
We've only just begun exploring

0:13:49.525,0:13:52.509
other possible uses[br]for this new technology.

0:13:52.509,0:13:55.288
It lets us see physical processes[br]that we know are there

0:13:55.288,0:13:59.575
but that we've never been able[br]to see with our own eyes until now.

0:14:00.677,0:14:01.917
This is our team.

0:14:01.917,0:14:04.767
Everything I showed you today[br]is a result of a collaboration

0:14:04.767,0:14:06.864
with this great group[br]of people you see here,

0:14:06.864,0:14:10.484
and I encourage you and welcome you[br]to check out our website,

0:14:10.484,0:14:12.017
try it out yourself,

0:14:12.017,0:14:15.263
and join us in exploring[br]this world of tiny motions.

0:14:15.263,0:14:16.700
Thank you.

0:14:16.700,0:14:18.726
(Applause)