WEBVTT

00:00:15.088 --> 00:00:21.142
So over the past few centuries,
microscopes have revolutionized our world.

00:00:23.204 --> 00:00:27.563
They revealed to us a tiny world
of objects, life and structures

00:00:27.563 --> 00:00:30.567
that are too small for us
to see with our naked eyes.

00:00:30.567 --> 00:00:34.064
They are a tremendous contribution
to science and technology.

00:00:34.064 --> 00:00:37.839
Today I'd like to introduce you
to a new type of microscope,

00:00:37.839 --> 00:00:40.230
a microscope for changes.

00:00:40.230 --> 00:00:43.438
It doesn't use optics
like a regular microscope

00:00:43.438 --> 00:00:45.350
to make small objects bigger,

00:00:45.350 --> 00:00:49.673
but instead it uses a video camera
and image processing

00:00:49.673 --> 00:00:54.387
to reveal to us the tiniest motions
and color changes in objects and people,

00:00:54.526 --> 00:00:58.368
changes that are impossible
for us to see with our naked eyes.

00:00:58.906 --> 00:01:02.970
And it lets us look at our world
in a completely new way.

00:01:03.295 --> 00:01:05.715
So what do I mean by color changes?

00:01:07.093 --> 00:01:09.896
Our skin, for example,
changes its color very slightly

00:01:09.896 --> 00:01:12.211
when the blood flows under it.

00:01:12.211 --> 00:01:14.443
That change is incredibly subtle,

00:01:14.443 --> 00:01:16.872
which is why, when you look
at other people,

00:01:16.872 --> 00:01:19.171
when you look at the person
sitting next to you,

00:01:19.171 --> 00:01:21.920
you don't see their skin
or their face changing color.

00:01:21.920 --> 00:01:27.150
When we look at this video of Steve here,
it appears to us like a static picture,

00:01:27.849 --> 00:01:31.292
but once we look at this video
through our new, special microscope,

00:01:31.292 --> 00:01:34.550
suddenly we see
a completely different image.

00:01:35.276 --> 00:01:39.200
What you see here are small changes
in the color of Steve's skin,

00:01:39.200 --> 00:01:43.087
magnified 100 times
so that they become visible.

00:01:43.539 --> 00:01:46.259
We can actually see a human pulse.

00:01:46.554 --> 00:01:49.729
We can see how fast
Steve's heart is beating,

00:01:49.729 --> 00:01:53.994
but we can also see the actual way
that the blood flows in his face.

00:01:55.152 --> 00:01:58.055
And we can do that not just
to visualize the pulse,

00:01:58.055 --> 00:02:01.448
but also to actually recover
our heart rates,

00:02:01.448 --> 00:02:03.510
and measure our heart rates.

00:02:03.510 --> 00:02:07.692
And we can do it with regular cameras
and without touching the patients.

00:02:07.692 --> 00:02:12.768
So here you see the pulse and heart rate
we extracted from a neonatal baby

00:02:12.768 --> 00:02:16.250
from a video we took
with a regular DSLR camera,

00:02:16.250 --> 00:02:18.364
and the heart rate measurement we get

00:02:18.364 --> 00:02:22.588
is as accurate as the one you'd get
with a standard monitor in a hospital.

00:02:23.384 --> 00:02:26.457
And it doesn't even have to be
a video we recorded.

00:02:26.457 --> 00:02:29.391
We can do it essentially
with other videos as well.

00:02:29.391 --> 00:02:32.565
So I just took a short clip
from "Batman Begins" here

00:02:32.565 --> 00:02:35.226
just to show Christian Bale's pulse.

00:02:35.226 --> 00:02:37.282
(Laughter)

00:02:37.282 --> 00:02:39.417
And you know, presumably
he's wearing makeup,

00:02:39.417 --> 00:02:41.389
the lighting here is kind of challenging,

00:02:41.389 --> 00:02:44.392
but still, just from the video,
we're able to extract his pulse

00:02:44.392 --> 00:02:46.224
and show it quite well.

00:02:46.224 --> 00:02:47.995
So how do we do all that?

00:02:47.995 --> 00:02:52.315
We basically analyze the changes
in the light that are recorded

00:02:52.315 --> 00:02:54.928
at every pixel in the video over time,

00:02:54.928 --> 00:02:56.648
and then we crank up those changes.

00:02:56.648 --> 00:02:59.494
We make them bigger
so that we can see them.

00:02:59.494 --> 00:03:01.576
The tricky part is that those signals,

00:03:01.576 --> 00:03:04.359
those changes that we're after,
are extremely subtle,

00:03:04.359 --> 00:03:07.353
so we have to be very careful
when you try to separate them

00:03:07.353 --> 00:03:10.240
from noise that always exists in videos.

00:03:10.240 --> 00:03:13.682
So we use some clever
image processing techniques

00:03:13.682 --> 00:03:17.736
to get a very accurate measurement
of the color at each pixel in the video,

00:03:17.736 --> 00:03:20.569
and then the way
the color changes over time,

00:03:20.569 --> 00:03:23.227
and then we amplify those changes.

00:03:23.227 --> 00:03:27.081
We make them bigger to create those types
of enhanced videos, or magnified videos,

00:03:27.081 --> 00:03:29.552
that actually show us those changes.

00:03:32.007 --> 00:03:36.227
But it turns out we can do that
not just to show tiny changes in color,

00:03:36.227 --> 00:03:38.379
but also tiny motions,

00:03:38.379 --> 00:03:42.064
and that's because the light
that gets recorded in our cameras

00:03:42.064 --> 00:03:45.189
will change not only if the color
of the object changes,

00:03:45.189 --> 00:03:47.305
but also if the object moves.

00:03:47.905 --> 00:03:52.543
So this is my daughter
when she was about two months old.

00:03:56.157 --> 00:03:59.397
It's a video I recorded
about three years ago.

00:03:59.397 --> 00:04:02.807
And as new parents, we all want
to make sure our babies are healthy,

00:04:02.807 --> 00:04:05.373
that they're breathing,
that they're alive, of course.

00:04:05.373 --> 00:04:07.465
So I too got one of those baby monitors

00:04:07.465 --> 00:04:10.072
so that I could see my daughter
when she was asleep.

00:04:10.072 --> 00:04:13.590
And this is pretty much what you'll see
with a standard baby monitor.

00:04:13.590 --> 00:04:15.686
You can see the baby's sleeping,

00:04:15.686 --> 00:04:17.729
but there's not too much information
there.

00:04:17.729 --> 00:04:19.516
There's not too much we can see.

00:04:19.516 --> 00:04:22.358
Wouldn't it be better,
or more informative, or more useful,

00:04:22.358 --> 00:04:25.261
if instead we could look
at the view like this.

00:04:25.261 --> 00:04:30.310
So here I took the motions
and I magnified them 30 times,

00:04:31.217 --> 00:04:33.708
and then I could clearly see
that my daughter

00:04:33.708 --> 00:04:35.428
was indeed alive and breathing.

00:04:35.428 --> 00:04:37.565
(Laughter)

00:04:38.092 --> 00:04:39.891
Here is a side-by-side comparison.

00:04:39.891 --> 00:04:42.440
So again, in the source video,
in the original video,

00:04:42.440 --> 00:04:44.260
there's not too much we can see,

00:04:44.260 --> 00:04:48.212
but once we magnify the motions,
the breathing becomes much more visible.

00:04:48.212 --> 00:04:50.801
And it turns out,
there's a lot of phenomena

00:04:50.801 --> 00:04:54.474
we can reveal and magnify
with our new motion microscope.

00:04:54.474 --> 00:04:58.909
We can see how our veins and arteries
are pulsing in our bodies.

00:04:59.752 --> 00:05:02.560
We can see that our eyes
are constantly moving

00:05:02.560 --> 00:05:04.776
in this wobbly motion.

00:05:04.776 --> 00:05:06.354
And that's actually my eye,

00:05:06.354 --> 00:05:09.414
and again this video was taken
right after my daughter was born,

00:05:09.414 --> 00:05:13.103
so you can see I wasn't getting
too much sleep. (Laughter)

00:05:13.539 --> 00:05:16.403
Even when a person is sitting still,

00:05:16.403 --> 00:05:18.997
there's a lot of information
we can extract

00:05:18.997 --> 00:05:21.989
about their breathing patterns,
small facial expressions.

00:05:22.672 --> 00:05:24.623
Maybe we could use those motions

00:05:24.623 --> 00:05:27.488
to tell us something about
our thoughts or our emotions.

00:05:29.003 --> 00:05:32.385
We can also magnify
small mechanical movements,

00:05:32.385 --> 00:05:34.337
like vibrations in engines,

00:05:34.337 --> 00:05:38.017
that can help engineers detect
and diagnose machinery problems,

00:05:40.130 --> 00:05:45.547
or see how our buildings and structures
sway in the wind and react to forces.

00:05:45.547 --> 00:05:50.333
Those are all things that our society
knows how to measure in various ways,

00:05:50.333 --> 00:05:52.875
but measuring those motions is one thing,

00:05:52.875 --> 00:05:55.479
and actually seeing those motions
as they happen

00:05:55.479 --> 00:05:57.614
is a whole different thing.

00:05:58.450 --> 00:06:02.021
And ever since we discovered
this new technology,

00:06:02.021 --> 00:06:03.723
we made our code available online

00:06:03.723 --> 00:06:06.269
so that others could use
and experiment with it.

00:06:08.005 --> 00:06:09.809
It's very simple to use.

00:06:09.809 --> 00:06:11.853
It can work on your own videos.

00:06:11.853 --> 00:06:15.357
Our collaborators at Quantum Research
even created this nice website

00:06:15.357 --> 00:06:18.036
where you can upload your videos
and process them online,

00:06:18.036 --> 00:06:21.603
so even if you don't have any experience
in computer science or programming,

00:06:21.603 --> 00:06:24.509
you can still very easily experiment
with this new microscope.

00:06:24.509 --> 00:06:26.941
And I'd like to show you
just a couple of examples

00:06:26.941 --> 00:06:28.919
of what others have done with it.

00:06:32.363 --> 00:06:37.259
So this video was made
by a YouTube user called Tamez85.

00:06:37.259 --> 00:06:38.807
I don't know who that user is,

00:06:38.807 --> 00:06:41.105
but he, or she, used our code

00:06:41.105 --> 00:06:43.910
to magnify small belly movements
during pregnancy.

00:06:44.933 --> 00:06:46.420
It's kind of creepy.

00:06:46.420 --> 00:06:48.818
(Laughter)

00:06:48.818 --> 00:06:52.782
People have used it to magnify
pulsing veins in their hands.

00:06:53.532 --> 00:06:56.699
And you know it's not real science
unless you use guinea pigs,

00:06:58.037 --> 00:07:00.584
and apparently this guinea pig
is called Tiffany,

00:07:00.584 --> 00:07:04.007
and this YouTube user claims
it is the first rodent on Earth

00:07:04.007 --> 00:07:05.780
that was motion-magnified.

00:07:06.604 --> 00:07:08.811
You can also do some art with it.

00:07:08.811 --> 00:07:12.123
So this video was sent to me
by a design student at Yale.

00:07:12.123 --> 00:07:14.516
She wanted to see
if there's any difference

00:07:14.516 --> 00:07:16.072
in the way her classmates move.

00:07:16.072 --> 00:07:20.361
She made them all stand still,
and then magnified their motions.

00:07:20.361 --> 00:07:23.457
It's like seeing still pictures
come to life.

00:07:23.714 --> 00:07:26.077
And the nice thing with all those examples

00:07:26.077 --> 00:07:28.315
is that we had nothing to do with them.

00:07:28.315 --> 00:07:32.165
We just provided this new tool,
a new way to look at the world,

00:07:32.165 --> 00:07:36.683
and then people find other interesting,
new and creative ways of using it.

00:07:37.735 --> 00:07:39.620
But we didn't stop there.

00:07:40.943 --> 00:07:44.597
This tool not only allows us
to look at the world in a new way,

00:07:44.597 --> 00:07:47.034
it also redefines what we can do

00:07:47.034 --> 00:07:50.232
and pushes the limits
of what we can do with our cameras.

00:07:50.232 --> 00:07:52.611
So as scientists, we started wondering,

00:07:52.611 --> 00:07:56.299
what other types of physical phenomena
produce tiny motions

00:07:56.299 --> 00:07:59.212
that we could now use
our cameras to measure?

00:07:59.212 --> 00:08:02.635
And one such phenomenon
that we focused on recently is sound.

00:08:03.664 --> 00:08:05.963
Sound, as we all know,
is basically changes

00:08:05.963 --> 00:08:08.134
in air pressure
that travel through the air.

00:08:08.134 --> 00:08:11.857
Those pressure waves hit objects
and they create small vibrations in them,

00:08:11.857 --> 00:08:14.519
which is how we hear
and how we record sound.

00:08:14.519 --> 00:08:18.294
But it turns out that sound
also produces visual motions.

00:08:18.579 --> 00:08:21.303
Those are motions
that are not visible to us

00:08:21.303 --> 00:08:24.229
but are visible to a camera
with the right processing.

00:08:24.229 --> 00:08:26.045
So here are two examples.

00:08:26.045 --> 00:08:29.374
This is me demonstrating
my great singing skills.

00:08:30.845 --> 00:08:33.603
(Singing)

00:08:33.603 --> 00:08:34.710
(Laughter)

00:08:34.710 --> 00:08:37.706
And I took a high-speed video
of my throat while I was humming.

00:08:37.706 --> 00:08:39.355
Again, if you stare at that video,

00:08:39.355 --> 00:08:41.387
there's not too much
you'll be able to see,

00:08:41.387 --> 00:08:45.793
but once we magnify the motions 100 times,
we can see all the motions and ripples

00:08:45.793 --> 00:08:49.103
in the neck that are involved
in producing the sound.

00:08:49.103 --> 00:08:51.528
That signal is there in that video.

00:08:51.528 --> 00:08:54.228
We also know that singers
can break a wine glass

00:08:54.228 --> 00:08:56.274
if they hit the correct note.

00:08:56.274 --> 00:08:58.325
So here, we're going to play a note

00:08:58.325 --> 00:09:00.849
that's in the resonance frequency
of that glass

00:09:00.849 --> 00:09:03.125
through a loudspeaker that's next to it.

00:09:03.125 --> 00:09:07.568
Once we play that note
and magnify the motions 250 times,

00:09:07.568 --> 00:09:10.789
we can very clearly see
how the glass vibrates

00:09:10.789 --> 00:09:13.623
and resonates in response to the sound.

00:09:14.132 --> 00:09:16.545
It's not something you're used to seeing
every day.

00:09:16.545 --> 00:09:19.408
And we actually have the demo
right outside set up,

00:09:19.408 --> 00:09:21.300
so I encourage you to stop by,

00:09:21.300 --> 00:09:24.347
and just play with it yourself,
you can actually see it live.

00:09:24.608 --> 00:09:27.768
But this made us think.
It gave us this crazy idea.

00:09:28.078 --> 00:09:32.865
Can we actually invert this process
and recover sound from video

00:09:33.454 --> 00:09:37.581
by analyzing the tiny vibrations
that sound waves create in objects,

00:09:37.581 --> 00:09:41.898
and essentially convert those
back into the sounds that produced them.

00:09:42.548 --> 00:09:46.472
In this way, we can turn
everyday objects into microphones.

00:09:47.958 --> 00:09:49.595
So that's exactly what we did.

00:09:49.595 --> 00:09:52.462
So here's an empty bag of chips
that was lying on a table,

00:09:52.462 --> 00:09:55.234
and we're going to turn that bag of chips
into a microphone

00:09:55.234 --> 00:09:57.145
by filming it with a video camera

00:09:57.145 --> 00:10:00.914
and analyzing the tiny motions
that sound waves create in it.

00:10:01.479 --> 00:10:04.242
So here's the sound
that we played in the room.

00:10:04.242 --> 00:10:07.634
(Music: "Mary Had a Little Lamb")

00:10:12.476 --> 00:10:15.426
And this is a high-speed video
we recorded of that bag of chips.

00:10:15.426 --> 00:10:16.528
Again it's playing.

00:10:16.528 --> 00:10:19.886
There's no chance you'll be able
to see anything going on in that video

00:10:19.886 --> 00:10:21.000
just by looking at it,

00:10:21.000 --> 00:10:23.962
but here's the sound we were able
to recover just by analyzing

00:10:23.962 --> 00:10:26.273
the tiny motions in that video.

00:10:27.127 --> 00:10:30.494
(Music: "Mary Had a Little Lamb")

00:10:44.607 --> 00:10:46.458
I call it -- Thank you.

00:10:46.458 --> 00:10:49.328
(Applause)

00:10:53.834 --> 00:10:56.140
I call it the visual microphone.

00:10:56.140 --> 00:10:59.251
We actually extract audio signals
from video signals.

00:10:59.251 --> 00:11:02.435
And just to give you a sense
of the scale of the motions here,

00:11:02.435 --> 00:11:06.696
a pretty loud sound will cause
that bag of chips

00:11:06.696 --> 00:11:09.266
to move less than a micrometer.

00:11:09.807 --> 00:11:12.485
That's one thousandth of a millimeter.

00:11:12.485 --> 00:11:16.179
That's how tiny the motions are
that we are now able to pull out

00:11:16.179 --> 00:11:19.282
just by observing how light
bounces off objects

00:11:19.282 --> 00:11:21.704
and gets recorded by our cameras.

00:11:22.208 --> 00:11:25.358
We can recover sounds
from other objects, like plants.

00:11:25.986 --> 00:11:29.183
(Music: "Mary Had a Little Lamb")

00:11:34.153 --> 00:11:36.456
And we can recover speech as well.

00:11:36.456 --> 00:11:38.817
So here's a person speaking in a room.

00:11:38.817 --> 00:11:43.632
Voice: Mary had a little lamb
whose fleece was white as snow,

00:11:43.632 --> 00:11:47.570
and everywhere that Mary went,
that lamb was sure to go.

00:11:48.722 --> 00:11:51.486
Michael Rubinstein: And here's
that speech again recovered

00:11:51.486 --> 00:11:54.220
just from this video
of that same bag of chips.

00:11:54.220 --> 00:11:59.211
Voice: Mary had a little lamb
whose fleece was white as snow,

00:11:59.211 --> 00:12:03.731
and everywhere that Mary went,
that lamb was sure to go.

00:12:04.352 --> 00:12:06.907
MR: We used "Mary Had a Little Lamb"

00:12:06.907 --> 00:12:09.252
because those are said to be
the first words

00:12:09.252 --> 00:12:13.053
that Thomas Edison spoke
into his phonograph in 1877.

00:12:13.053 --> 00:12:16.565
It was one of the first
sound recording devices in history.

00:12:16.565 --> 00:12:19.842
It basically directed the sounds
onto a diaphragm

00:12:19.842 --> 00:12:24.270
that vibrated a needle that essentially
engraved the sound on tinfoil

00:12:24.270 --> 00:12:26.565
that was wrapped around the cylinder.

00:12:26.565 --> 00:12:29.679
Here's a demonstration of recording

00:12:29.679 --> 00:12:32.446
and replaying sound
with Edison's phonograph.

00:12:33.549 --> 00:12:36.487
(Video) Voice: Testing, testing,
one two three.

00:12:36.487 --> 00:12:39.654
Mary had a little lamb
whose fleece was white as snow,

00:12:39.654 --> 00:12:43.491
and everywhere that Mary went,
the lamb was sure to go.

00:12:43.491 --> 00:12:46.014
Testing, testing, one two three.

00:12:46.014 --> 00:12:50.103
Mary had a little lamb
whose fleece was white as snow,

00:12:50.103 --> 00:12:54.235
and everywhere that Mary went,
the lamb was sure to go.

00:12:55.719 --> 00:12:59.081
MR: And now, 137 years later,

00:13:00.334 --> 00:13:03.492
we're able to get sound
in pretty much similar quality

00:13:03.492 --> 00:13:07.559
but by just watching objects
vibrate to sound with cameras,

00:13:07.853 --> 00:13:09.952
and we can even do that when the camera

00:13:09.952 --> 00:13:13.631
is 15 feet away from the object,
behind soundproof glass.

00:13:14.178 --> 00:13:17.475
So this is the sound that we were able
to recover in that case.

00:13:17.475 --> 00:13:22.282
Voice: Mary had a little lamb
whose fleece was white as snow,

00:13:22.282 --> 00:13:26.993
and everywhere that Mary went,
the lamb was sure to go.

00:13:28.111 --> 00:13:31.711
MR: And of course, surveillance is
the first application that comes to mind.

00:13:31.711 --> 00:13:33.993
(Laughter)

00:13:33.993 --> 00:13:38.055
But it might actually be useful
for other things as well.

00:13:38.095 --> 00:13:41.196
Maybe in the future,
we'll be able to use it, for example,

00:13:41.196 --> 00:13:43.557
to recover sound across space,

00:13:43.557 --> 00:13:46.569
because sound can't travel
in space, but light can.

00:13:47.166 --> 00:13:49.525
We've only just begun exploring

00:13:49.525 --> 00:13:52.509
other possible uses
for this new technology.

00:13:52.509 --> 00:13:55.288
It lets us see physical processes
that we know are there

00:13:55.288 --> 00:13:59.575
but that we've never been able
to see with our own eyes until now.

00:14:00.677 --> 00:14:01.917
This is our team.

00:14:01.917 --> 00:14:04.767
Everything I showed you today
is a result of a collaboration

00:14:04.767 --> 00:14:06.864
with this great group
of people you see here,

00:14:06.864 --> 00:14:10.484
and I encourage you and welcome you
to check out our website,

00:14:10.484 --> 00:14:12.017
try it out yourself,

00:14:12.017 --> 00:14:15.263
and join us in exploring
this world of tiny motions.

00:14:15.263 --> 00:14:16.700
Thank you.

00:14:16.700 --> 00:14:18.726
(Applause)