-
John DeCrey: Welcome to week 10.
-
We're going to introduce and
talk a little bit more about
-
third party modules and
-
what that means and
how to install it.
-
Then we've got
several code longs
-
coming up that I
think will be fun.
-
I'd like to introduce my guest.
-
This is Jerick.
-
We've worked together
for several years now.
-
When we first meet, was it 2014?
-
Jerik: Yeah. That sounds right.
-
John DeCrey: Anyway Jerick lives
-
and breathes in Python
and he's done a lot
-
of his own personal
Python scripts
-
that I'll let him talk about
and some of them are funny.
-
Then he's an avid motorcyclist,
-
and he'll talk a little
bit about that too.
-
We'll say, how's the saying go,
-
was it bad decisions
leads to good stories?
-
Jerik: Yes.
-
John DeCrey: Unfortunately
I have a lot of stories.
-
Anyway go ahead, Jerick.
-
Jerik: Yes. Just to
introduce myself,
-
I've been working in
the software industry
-
for almost 12 years.
-
I've mostly been in the QA side
-
of things that's testing
and finding bugs.
-
A lot of my job though has
been creating automated tests.
-
A lot of that has been
through Python 2.
-
Our coat along exercises
is going to go
-
through some of the things
I've learned doing that.
-
I have some slides
to go through at
-
the beginning though,
let me just start this.
-
This is third party Python
modules. I'll just read this.
-
Modular programming refers
to the process of breaking
-
a large unwieldy programming
tasks into separate,
-
smaller, more manageable
sub tasks or modules.
-
Individual modules can then be
-
coubbled together like
building blocks to create
-
a larger application.
Just as an example.
-
You guys are probably
familiar with this when
-
you do these import
statements like this time.
-
This is a module that's
built into Python.
-
It's a way that you
can bring this into
-
your program and utilize
-
everything that's been built
out around this module.
-
It's just a way of not
-
having to redo work that
someone else has already done.
-
John DeCrey: We have done
imports throughout the course.
-
The from is probably
-
a little bit different
than anyone has seen it.
-
Can you talk about
that a little bit?
-
Jerik: Yeah. Let me
go back to that.
-
These firm statements are
-
similar to the
import statements,
-
but it's digging more
into the module itself.
-
John DeCrey: Submitting a
specific item from the import.
-
Jerik: Yeah. Exactly.
The advantages
-
of modular programming are,
-
that it's more simple, it's
maintainable and reusable.
-
When you import
that time module,
-
you don't have to do any
maintenance on that.
-
The people who run the
Python project are going to
-
be adding methods to that
or fixing something.
-
If we changed our time format,
-
they would be the ones going
-
and making sure
that still worked
-
with all the existing
Python programs out there.
-
Python, that time module
is one that's built in.
-
There's a bunch that
Python has built in.
-
Then there's also a bunch
of third party modules.
-
These are examples of ones
that are built into Python.
-
It's just a Python
modules index.
-
Then these are some common
third party modules.
-
They're guaranteed,
-
not every one of them
isn't listed here
-
because I would guess there's
hundreds of thousands.
-
The thing I love about Python
-
is any problem that
you're looking to solve,
-
someone on the
Internet has probably
-
solved it and created
a module for it.
-
You can just go and
import that module,
-
and get up and
-
running a lot faster than
having to do it yourself.
-
John DeCrey: Chances are high.
-
There's a module
out there for it.
-
The saying there's an app for
-
that in Python, there's
a module for that.
-
Jerik: Exactly. You guys
-
have probably already
done this too.
-
John DeCrey: We actually
have not used pip yet.
-
I'm trying to think,
-
but I don't think that we have.
-
As far as exiting installing.
-
Jerik: To install a
third party module.
-
The syntax is pip install
and then the package name.
-
I'll show an example of that
-
later on when we do the
code along exercises.
-
John DeCrey: We type that
in at the Python console.
-
Jerik: Now I want to just
show some examples of
-
different modules that I've used
-
like throughout work and
personal projects and stuff.
-
This is only three
lines of code,
-
but it's going to do
something really cool.
-
This is a module
called pytesseract.
-
What it is it's image,
-
it's a OCR engine.
-
You can get a picture.
-
This is just a
picture of a receipt.
-
You can get all the
text data from it,
-
which is crazy when
-
three lines of code,
and it can do that.
-
The reason I needed this
is I worked at a company
-
where we would scrape
the web for obituaries.
-
Then we sold that
the obituaries to
-
a well known Utah based
ancestry website.
-
There's this website.
-
John DeCrey: What company
you're talking about?
-
Jerik: Yeah. [LAUGHTER] There's
-
this obituary website in Germany
-
that for some reason
on their web page
-
instead of pasting the
obituary like as text they had
-
generated like a PNG
or a picture file
-
of the obituary and posted
that on the website.
-
We had our scraper go
and get those images,
-
and it would use pytesseract
to get the text out of
-
those images, to
get the obituaries.
-
But I can show this running.
-
We have this picture.
-
John DeCrey: That's a PNG file.
-
Jerik: All this is going
to do is open that
-
and get the text from
it and printed out.
-
John DeCrey: We're invoking
the image to string function,
-
and then passing in the
image to that function.
-
Jerik: Yeah, so this
image-to-string function is
-
something that's built into
this pytesseract module.
-
John DeCrey: Perfect.
-
Jerik: So running.
-
John DeCrey: Well, look at
that. Yeah, that's cool.
-
That was a fuzzy picture too.
-
It wasn't like, crisp, clean.
-
Jerik: Yeah, it actually
does pretty good.
-
I was playing around
with other images but.
-
John DeCrey: Yeah.
It did a good job.
-
Jerik: To me, it was just crazy.
-
Like that huge task of
getting obituary data
-
from picture files
on the Internet
-
was reduced to basically
three lines of code.
-
John DeCrey: Yeah, seriously.
-
Jerik: It did not take me
hours upon hours to do that.
-
[LAUGHTER] [OVERLAPPING]
-
John DeCrey: There's
already, stuff out there,
-
but if you wanted to design
your own, like for example,
-
you take a picture of receipts
and have it parse out,
-
saving it to your own data.
-
There's just lots of.
-
Jerik: Yeah, the sky's
the limit. [LAUGHTER]
-
John DeCrey: Of course
stuff you can do with that.
-
Actually, if I can
talk a little bit
-
about that for a minute.
-
We did a project at
the Uview Hospital.
-
The hospital uses Epic,
-
which is a huge EMR system,
Electronic Medical Records.
-
Every now and then,
like nurses and
-
the medical personnel that use
-
it will get an error message.
-
Sometimes the error, the screen,
-
it's like it's a message box.
-
They may not be sure what
the recovery is or what
-
the procedure is or process
-
is when you encounter
certain errors.
-
[NOISE] We developed a thing
-
where the medical personnel
can take a picture from
-
the screen of that error and
-
it translates it
to text and does
-
a look-up from a database,
-
finds the error code,
-
the error number, and stuff,
-
and then brings up the
documentation right
-
there on the handheld.
-
Jerik: That's awesome.
-
John DeCrey: Just give more
information on the error
-
and what best practices moving
forward from that error.
-
Lots of really great
opportunities.
-
Like you said,
-
the sky is the limit there.
-
Jerik: Here's another example.
-
Where is it? This is another
thing from that same job.
-
John DeCrey: Other
funeral homes?
-
Jerik: Yeah. [LAUGHTER] We
wanted to make sure that we had
-
all the funeral
home websites that
-
has the obituary data
in the US and Canada.
-
This isn't the script itself,
-
but the same idea.
-
A script that would
go on Google Maps,
-
search for a funeral home in
-
every zip code and
-
return a list of all
the funeral homes.
-
And then I would
go and add it to
-
a master list of
-
that would check for like
duplicates and stuff.
-
John DeCrey: Let's
talk about that.
-
Just those couple lie to
code and what it's doing.
-
We got on line four,
-
so you got request and
get and Google URL,
-
along with an appended
search string,
-
which is how Google
searches work.
-
Line five, however,
looks very cryptic.
-
I know what that is. I
do my best to avoid it.
-
[LAUGHTER]
-
Jerik: I probably knew
what this meant at
-
one point too. But
I don't anymore.
-
John DeCrey: This is called
regular expressions.
-
Whoever invented the syntax
-
for regular [LAUGHTER]
expressions,
-
I don't know, should maybe
-
go find something
else to do instead.
-
[LAUGHTER]
-
Jerik: I agree.
-
John DeCrey: I
suppose if you spend
-
enough time in it,
it becomes memory.
-
But anytime you
have something that
-
you have to have a guide and
a cheat sheet and stuff,
-
and ledgers and
whatever you look up,
-
I would say that it failed.
-
That's regular expressions on
-
Line 5 is what that
stuff is in. Go ahead.
-
Jerik: This is the
third-party module
-
that I'm using in this one.
-
This request, it
basically just gets
-
the entire HTML
from this request,
-
and then this rejects
URLs out of the HTML.
-
John DeCrey: Just
for clarification,
-
that's just shorthand for
-
regular expressions
when you hear
-
the rejects, same thing.
-
Jerik: Then Google
obviously has some of
-
their own URLs in there,
-
so I filter those out.
-
I'll run this, you can see
-
what [OVERLAPPING]
it prints out.
-
These are all what
-
shows up when I search
funeral homes in my zip code.
-
[LAUGHTER]
-
John DeCrey: That's right.
Then you specified a zip code?
-
Jerik: Yeah. When
I ran this script,
-
like for my job at that company,
-
I think I just passed it in
-
all the zip codes and just
let it run overnight.
-
[LAUGHTER] Then in the morning
had a list of like 20,000.
-
John DeCrey: In Line 4,
-
I just noticed that you're
-
converting your zip
code to a string.
-
Why wouldn't you just wrap
it inside of a string?
-
Is there a particular reason?
-
Jerik: I probably copied
this from my real script and
-
I was probably passing in
-
a variable of [OVERLAPPING]
the zip codes there.
-
I was just lazy
-
when I brought it over
to this example one.
-
[LAUGHTER]
-
John DeCrey: It's pretty cool
that you can do that too.
-
When that returns, it looks like
-
the URLs that on Line 5,
-
all that the results are
coming into that URL's,
-
that looks like a
list. Is that right?
-
Jerik: Yeah.
-
John DeCrey: Okay.
-
Jerik: I can even do this.
-
John DeCrey: Oh, I see.
Then in your fore loop,
-
you're going through that list,
-
and then also making sure
-
that anything that's
in the bad URLs is
-
not in the URL's
list. That's cool.
-
Jerik: I have a
breakpoint right here.
-
Now, this res
variable will have,
-
this text is going to be
the entire HTML from that,
-
I don't know, there's a way
to copy the whole thing.
-
But anyway, it's like
-
a big jumbling of HTML
that gets returned.
-
John DeCrey: Very cool.
-
Jerik: This next one is a
personal project that is dumb,
-
but awesome too because
-
it's been [OVERLAPPING]
paying out.
-
[LAUGHTER] I call this
one the Being Rewarder.
-
Being the search engine
has this dumb program
-
where it's called Being Rewards.
-
It gives you these points
-
every day for just
searching and being.
-
I think you can do
like 20 searches
-
on desktop and then like 15
-
on mobile being and
you get points for it.
-
I've made this script that
just searches being every day.
-
John DeCrey: Then you can redeem
the points for goods, right?
-
Jerik: Yeah. You can get
gift cards and stuff.
-
I use it for like
Xbox game pass on PC,
-
which is like a game service.
-
John DeCrey: What is
that typically files?
-
Jerik: I have no
idea. [LAUGHTER] I
-
bet it's like 20 bucks a
month or 15 bucks a month,
-
probably. Not too expensive.
-
John DeCrey: I have to say
more like 50 bucks a year.
-
But if you're saying
25 [LAUGHTER]
-
a month, that's
pretty expensive.
-
Jerik: This script
has been working
-
for like six years now.
-
[OVERLAPPING] I thought it would
-
have been shut down forever.
-
John DeCrey: You've
never paid for it.
-
Because you get the
being rewarded.
-
Jerik: Yeah, exactly.
-
[LAUGHTER] I'll talk through
the code a little bit.
-
These functions, actually,
-
I'll run through it, and
then I'll go through it.
-
What this is going to do
is just going to open,
-
being in desktop mode,
-
search it three times,
-
and then open it in mobile mode,
-
and search it three
times. We'll run that.
-
These are just random words.
-
John DeGrey: From
your generator there.
-
Jerik: I haven't thought
about this before.
-
This is actually
-
a good example to bring up.
-
I have these generated words,
-
but there was also another
third party module
-
that was for just
entering in real.
-
But from the dictionary words.
-
I was using that for a while,
-
but it had to get it from
-
their online database
or whatever and
-
sometimes it didn't work
and so I went back to this.
-
John DeGrey: I see.
-
Jerik: That's just
something you have to
-
deal with with third
party modules.
-
Sometimes they don't
work the best, I guess.
-
John DeGrey: Yeah, we saw it run
-
three times for the
desktop and three times on
-
the mobile and you just
-
run this automatically
daily, right?
-
Jerik: Yeah.
-
John DeGrey: It's awesome.
-
Jerik: I call this the KSL
deal finder. It's a script.
-
When you search for
something you want,
-
like dirt bikes for example,
-
you have your list
of search results.
-
The top ad is going
to be the newest.
-
This script gets the top
ad and then it keeps
-
checking for new ads to show up.
-
If a new ad is
posted on my script,
-
at least I send an email
-
to myself that
there was a new ad.
-
It's a way to be the
first person to message,
-
the person who posts
what they're selling.
-
John DeGrey: Yeah, especially
if it's a great deal
-
and I love the story.
-
If you wouldn't mind telling us
-
the great deal you got on
one of your motorcycles.
-
Jerik: I was in the market
for a new dirt bike
-
and I had this script
running and I was up in
-
Heber and was that like
-
eight o'clock at night
and I was looking at
-
a bike and didn't want it and
-
then eight o'clock I get a
email from the deal finders.
-
John DeGrey: Eight
o'clock at night?
-
Jerik: Yes. Of this
incredible deal.
-
Like crazy deal.
-
But it was down in Grand
Junction, Colorado.
-
John DeGrey: You're out
of where? You're at?
-
Jerik: Yeah, I'm in Orem.
-
I think it was like a
three or four hour drive.
-
But anyways, I'm
like, ding it now.
-
I have to drive to Grand
Junction, Colorado tonight.
-
Yeah, I called the guy up.
-
I'm like, I can drive down
-
now but I won't be
there till like
-
midnight or 01:00 AM
and he said that was
-
fine and so I start driving down
-
there and it was a blizzard
-
in Spanish Fort Canyon
of all places, too.
-
I hate that road, but
I almost turned back.
-
John DeGrey: Oh, is that right.
-
Jerik: Probably like
three or four times?
-
Yeah. I even pulled over once.
-
John DeGrey: Oh, really?
-
Jerik: Yeah, I almost just
because the deal was so good,
-
I thought it had to
have been a scam.
-
John DeGrey: Oh, yeah,
that's through the whole way
-
down you're thinking,
please don't be a scam.
-
Jerik: But yeah, I ended
up picking up the bike.
-
Drove back. Yeah, I still
have that dirt bike.
-
I could still sell it for
-
1,000 more than I bought it for.
-
John DeGrey: Wow.
-
Jerik: It was one was that
access data so over six years.
-
John DeGrey: Wow, so
because of your script,
-
you got an alert to
-
the ad immediately as
soon as it became on it.
-
Jerik: Yeah, so I was
definitely the first caller.
-
But I've had like
buddies use this script
-
too and find deals on stuff too.
-
John DeGrey: Then you still
went to work that same day?
-
Jerik: Oh, yeah I got back
-
at 04:00 AM and I still went to
-
work and there's absolutely
no way I could do that now.
-
I think just getting over 30,
-
I'm retired from doing
stuff like that.
-
John DeGrey: Until the next
great deal comes along.
-
Jerik: Oh yeah, probably.
-
John DeGrey: By the way,
-
for the listeners here,
-
actually two things for anybody
-
that's out of state
not familiar with KSL,
-
that KSL is a local
station here in Utah
-
and they have online
classified listing.
-
It is probably one
of the most common
-
and popular in the state.
-
In fact, other people in other
states use it frequently.
-
It's not as famous as eBay.
-
Well, it may be
as an auction too
-
but coniine ads and stuff.
-
We're actually
going to do one of
-
our labs using the
KSL deal finder.
-
Then just want to do a caution
-
because any time that
-
this is also referred to
as like screen scraping,
-
anytime that you're doing that,
-
you want to make sure that
you don't flag yourself as
-
being a bot that could
-
potentially get banned
from a service.
-
Jerik, I think
-
didn't you get banned
at one time from KSL?
-
Jerik: Yeah, they banned my IP.
-
John DeGrey: Yeah, so they
block your IP address.
-
You can't use it anymore?
-
I think we have the timer.
-
This goes into a timer setting,
-
loops and there's
a sleep function
-
that we can use to sleep for
-
so many seconds and
so he's got it set to
-
15 which is I'm assuming
is that 15 seconds?
-
Jerik: Yeah.
-
John DeGrey: That's
probably well adequate.
-
The thing is, you
don't want to flag
-
yourself and lower that
down to maybe even
-
like 10 or five seconds
because chances are,
-
a lot of these systems
including KSL,
-
they do have bot detectors and
-
you'll definitely get
yourself put on the radar.
-
That's just a word of caution
when using these scripts.
-
Take it away Jarik.
-
Jerik: I don't have any
examples but there's
-
a couple of other projects that
-
I did that I want to talk about.
-
There's one I called
the info wall.
-
It was me and my buddies.
-
We did like a Bluetooth
speaker start up thing that
-
we sold it on Amazon
-
and we wanted a pretty display
-
that showed how many
sales we got that day.
-
I made a little
script that would go
-
and look at our Amazon portal
-
and just get the
sales for that day.
-
Now that I think about
it, there's probably
-
an API I could have looked
into to get that data,
-
but this is what I'm
used to so I did that.
-
John DeGrey: It's fun
to do your own anyway
-
then you have more control of
-
getting exactly what you want.
-
Jerik: Unfortunately,
the info wall
-
didn't have very good numbers.
-
We didn't sell very
many speakers,
-
so that was retired.
-
Another project,
-
this is probably the
dumbest project I've done.
-
I called it NBA bet buddy.
-
I used to bet on
NBA games a lot.
-
I tried an experiment
where I had made
-
a script that every morning
-
it would go and get the
Vegas odds of games
-
and then compare it against
-
expert picks for
games that night.
-
Then anytime there was
a mismatch between
-
the odds and the expert picks,
-
it would send those to me
-
because those are
supposedly supposed to
-
be like good ones to bet on.
-
End of the story, they were not
-
so that project is retired too.
-
John DeGrey: Still
a lot of risk?
-
Jerik: Yes. Vegas knows how
to make their money still.
-
John DeGrey: That's funny.
-
Jerik: All right. We're good
-
to start the code-along
exercises now?
-
John DeGrey: Yeah,
let's jump into it.
-
What's our first code-along.
-
Jerik: This one is going
to be a simple one.
-
I'll show you what
this is going to do.
-
John DeGrey: We're going
to do this ski report.
-
Jerik: Yeah. What this
script is going to do,
-
it's going to go
to this web page,
-
SkiUtah.com snow report.
-
This is a good day to do it.
-
This is October 23.
-
We just had some snow.
-
John DeGrey: Actually got
some snow last 24 hours.
-
Jerik: We got some
data to extract.
-
This is what our
script, it's just going
-
to get for snowbird,
-
the 24 hour snowfall,
-
48 and then the base
-
and since this is
the first snowfall,
-
these are going to be the same.
-
But that is all right.
-
Let me make a new file.
-
John DeGrey: You're just
creating a new game pip file?
-
Jerik: Yeah. Then let's see.
-
This is where we're going to use
-
the pip install syntax
-
to install the
modules that we need.
-
The first one we need
is called selenium,
-
so you just type pip install
selenium into your terminal.
-
John DeGrey: Most
people, I don't know
-
if anyone's using
the IDE you're in,
-
most of us is in PyCharm.
-
If you open up the
Python console,
-
which is in PyCharm,
-
down at the bottom you'll see
-
a terminal and there
will be Python console.
-
You can also do it
from the terminal,
-
but you still have
to be in Python.
-
But you go and there's
a PIP, install.
-
Jerik: Yeah, then this
-
will say that I already
have it installed.
-
But if you don't,
it will install it.
-
John DeGrey: See how
do you spell that?
-
Jerik: S-E-L-E-N-I-U-M.
-
John DeGrey: Okay.
-
John DeCrey: Mine
says the same thing.
-
Already set aside.
-
Jerik: So Selenium.
That's the module that
-
you're using to hook
to the web page,
-
can navigate web pages with
it like click on buttons
-
or Enter and search
-
bars like it did for
the being rewarder,
-
or you can also get
text from it too,
-
so that's what we're going
to use for this script.
-
We do need another module,
-
this one's called
WebDriverManager,
-
so you do pip, install
web driver-manager.
-
John DeCrey: Web driver
and then hyphen,
-
is there a space?
-
Jerik: No space,
just dash manager.
-
John DeCrey: Dash manager.
-
Jerik: So when I was
getting this lesson
-
ready yesterday, I noticed that.
-
When I was running the scripts,
-
I had to add one more package,
-
I don't know why,
but just in case,
-
I might as well install it,
-
it's called packaging,
-
so you do pip install packaging,
-
and now we should be good to go.
-
John DeCrey: Install packaging?
-
Jerik: Yeah.
-
John DeCrey: Requirement
already satisfied. Cool.
-
Jerik: Then for each of
these Selenium scripts,
-
I'm just going to copy
-
some stuff that gets
everything set up.
-
I don't know if I
should leave it on
-
the screen for a while, John.
-
I'm good to keep going.
-
John DeCrey: Actually,
if you wouldn't mind
-
just bumping your
font up a little bit,
-
I've been using 18.
-
Jerik: How is that?
-
John DeCrey: There you
go. That looks great.
-
What are the instructions?
-
Instructions are to just
-
copy what you got
there on the screen?
-
Jerik: You can
pause now and copy.
-
John DeCrey: People
watching this,
-
you can just pause the video,
-
go ahead and enter
all that stuff
-
in your script and
then go ahead, Jack.
-
Do you want to explain?
-
Jerik: Yes, so I'll just
explain some of these.
-
This driver object it's
-
basically the browser that
-
we're going to be
interacting with,
-
so you can pass it in
a bunch of arguments,
-
one of them is headless,
-
what headless does
is it makes it
-
so it doesn't bring
up the browser.
-
Makes the script run faster,
-
because it times have to
load images and stuff.
-
John DeCrey: You
are still running,
-
but you don't see it.
-
It's running as a
back end process,
-
so there's no user interfaces.
-
Jerik: When I'm
building scripts,
-
I'll usually comment
this out so I can see
-
what's going on as it runs.
-
Then this, I just had to add
-
because I don't know if
it's chrome or Selenium,
-
just logs, or these two lines I
-
guess make it so it doesn't spit
-
out logs that look scary.
-
This is all setting
up this driver,
-
which is the web driver
-
or the browser that we're
going to interact with.
-
John DeCrey: The driver
is what that's basically
-
what's driving the
screen scraping.
-
That's what's loading
the web browser thing,
-
Jerik: To navigate to a
web page using Selenium,
-
you do driver.get,
-
and then in this case
-
we're going to go
to this web page,
-
and then there's better
ways to do this,
-
but this makes it a lot simpler.
-
I'm just going to
put a sleep there,
-
so this is going to
go to that web page
-
and sleep for five seconds,
-
this sleep is just
going to make sure that
-
everything's loaded before
we try and do stuff with it.
-
John DeCrey: Then
five means we're
-
sleeping it for five seconds.
-
Jerik: The first thing
we want to get is
-
this 24 hour snowfall.
-
To do that we can set
it as a variable,
-
so I'll just do our 24 equals,
-
and then you do everything
through the driver.
-
driver.find_element.
-
All these things on the page,
-
they're called elements,
-
and then with Selenium,
-
there's different
ways you can find
-
or you can hook to the elements,
-
CSS selectors is a
really popular one,
-
but I've used different
ones like x path,
-
I've had to use a few times,
-
it just depends on
what works are not,
-
so we'll do CSS sector.
-
Then just a quick overview
on CSS selectors,
-
so this is the page source,
-
over on the right, you can
click this little button,
-
it's called the element picker.
-
John DeCrey: How did
you tell [OVERLAPPING]?
-
Jerik: Sorry.
-
John DeCrey: How
you even got there?
-
Jerik: So when you
have chrome open,
-
you can press F12 and
we'll bring this up,
-
it's called the DevTools.
-
John DeCrey: And for mac users?
-
Jerik: I think it's F12
on Mac 2, isn't it?
-
John DeCrey: There's
a menu option too
-
from the Chrome menu.
-
Jerik: Inspect, I think.
-
John DeCrey: Inspect.
-
Jerik: Excuse me. So
this element picker.
-
John DeCrey: So you
click that little icon
-
in the far left?
-
Jerik: Yeah.
-
John DeCrey: That's cool.
Then you can hover over
-
the web page and it goes
right into the elements?
-
Jerik: Yeah. For this exercise,
-
I'm going to show you
a really easy way to
-
get the selector for
whatever element you want.
-
You click on what you
-
want and it highlights
it here in the source,
-
and then you can right
click on this and do copy,
-
select that and that
copies the CSS selector.
-
You can just paste
that in there.
-
John DeCrey: That's really cool.
-
Jerik: So if we left it
like this and ran it,
-
this hour 24 variable would
be the Selenium element,
-
but we need to get the
text from that element,
-
so to do that,
-
we're going to use.getattribute,
-
and then here you do inner HTML.
-
John DeCrey: And by
the way, for anybody
-
that might be having problems
-
getting that whole
CSS selector stuff,
-
I do have it posted
in the assignment
-
in Canvas that you can copy.
-
I encourage you to try to follow
-
Jerrek getting it live
from the website,
-
but just as a backup plan,
-
you can get it from
the Canvas page too.
-
Jerik: Next we want
48 hour total.
-
This is going to
be really similar
-
just a different selector,
-
so driver.find_element
by CSS selector,
-
and I'm just going to
add the get attribute,
-
inner HTML here,
-
then do the same thing to get
-
that selector, click on that,
-
right click, copy selector,
-
and put that in there.
-
John DeCrey: Do it
once for the 24 hour
-
and then twice for the 48.
-
Jerik: Then, same
thing for the base.
-
Now this should have those
values there, the 19.
-
Now we just want to print
it out so we can make
-
a report variable and
build out a report.
-
Let's see, 24-hour snowfall.
-
Then the base,
-
and then we can print
out the report.
-
John DeCrey: Just to
explain like line 25
-
using catenation and like
the slash being there.
-
We have talked about that.
-
Another kind of cool away we
could do it would be using
-
the string tripolation and
-
then the triple quotes
rather to get the literal.
-
Anyway, that's all that's doing.
-
Jerik: I'm going
to try this now.
-
I am running it, this
-
is going to be a boring
one because it's
-
just bringing up this web page.
-
I noticed even though
that box is there,
-
the page in the
background is still
-
loaded everything.
It doesn't matter.
-
John DeCrey: That's cool.
-
Jerik: If we did need to click
on things for our script,
-
I bet we would have had
to get rid of that.
-
This anticipated
opening dates thing.
-
That's just things
that you have to deal
-
with writing scripts like this.
-
That is that script all done.
-
John DeCrey: I was just
-
going to say you
can change it to
-
headless so we can see
the difference there.
-
Jerik: Yeah, it's just
not going to bring up
-
the browser window this time
-
and it should just
post the data there.
-
John DeCrey: There we go.
-
This is
-
one of the card
along assignments.
-
If you get it running
and you're good with
-
this before we move on,
-
you can submit your
script in the canvas,
-
and then, we can move
on to the next one.
-
Jerik: The next one. Where is
-
the deals page? That was weird.
-
This one is going to go
to the Amazon Deal of
-
the Day page and this
script is going to
-
get these deal titles along
with the link to the deal,
-
and it's going to
just print them out,
-
or we're going to save them
-
as a dictionary and
then print them out.
-
Same thing with this one.
-
You can copy it over
from the last script.
-
This is just stuff to get
everything set up and
-
then we're going to
-
do driver.get to
go to that page.
-
Instead of using this URL,
-
I'm just going to use this.
-
This seems like it will
work in the future better.
-
It's Amazon.com/deal.
-
Then same thing.
-
We're going to let it sleep for
-
five seconds to make sure
everything is loaded.
-
This script is going
to be a little more
-
complicated and here's
the thought process
-
to why we need to
do it this way.
-
For this one or for
the last script,
-
we just got a
single element from
-
Selenium using that
fine-by CSS thing.
-
But this we're going to want
to get every single one
-
of these titles and links.
-
I kind of see here in the
source how these are set up.
-
They're each in their
own little div.
-
We're going to find a selector
that gets all the divs.
-
If I click here and then
look in the source code,
-
you can hover over these,
-
and what I'm looking for now
is something that is unique,
-
that I can hook to
with the CSS selector.
-
Sorry, there's a lot here.
I want to make sure I
-
find the right one.
-
I'm going to find a
selector that finds
-
this div that's highlighted,
-
and then we can
-
drill down and get the
title and links from that.
-
We'll make a variable
called deal divs,
-
be similar to the last one.
-
This time though is really
-
important and in
the last classes,
-
a few people always miss this.
-
You want to make sure
this one is plural.
-
Let's find elements.
-
John DeCrey: Element with
an S. Easy to get confused.
-
If at the end
-
have some issues it's
not running as expected,
-
go back to this line,
-
to the line that you
have, and make sure
-
that it's elements
with S.element.
-
Jerik: Going back to this CSS,
-
I can search for this class.
-
I'll show you
another cool thing.
-
When you have Chrome
dev tools open,
-
you can press Control
F that brings up
-
this search bar that you
can search for selectors.
-
I use this a lot to
-
test selectors to make sure
I'm getting the right thing.
-
I'm just going to type
in the selector I
-
am thinking I want to use here.
-
These selectors will be
posted to, right, John?
-
John DeCrey: We can post
it. Your mouse cursor is
-
kind of covering what
you're typing there.
-
Jerik: This isn't too
important to the lessons.
-
Just in case you're
interested on
-
like my thought process
of finding selectors.
-
Then it's that deal
grid item module.
-
This star equals is
searching for a wild card,
-
so it's going to do any
class containing this.
-
We can see that it's
-
pining these divs,
which is good.
-
Then this found them all.
-
I'm going to go one
more down though.
-
To do that, let's do read of n,
-
and then div then it's
going down one more level.
-
The reason I like this
search thing to test
-
selectors is now I
can go and make sure.
-
John DeCrey: Narrow
it down and confirm
-
that you got the right one.
-
Jerik: Now when this runs,
-
instead of finding one of these,
-
this is setting all of
these to that variable.
-
So that is that.
-
Now we'll make an empty list for
-
the titles and an empty
list for the links too.
-
Then we can iterate through
-
the list of those
div elements and get
-
out what we need and
append it to these lists.
-
So to do that, we'll just do
for deal-div in deal_div.
-
John DeCrey: Just pay attention
-
on one that is without the S,
-
the one right after the for
-
loop is your variable
in the fourth in
-
that loop just to not
get confused there.
-
Jerik: We'll go deal titles.
-
I wanted to make
that [inaudible].
-
Deal titles, append.
-
This deal_div now when it gets
to this point is going to
-
be a single div.
-
So the first time it
iterates through it,
-
it's going to be this first one.
-
I'm going to expand
this because we want
-
to get the title.
-
Let me do this.
-
This text right here.
-
These smart DIY and home tools,
-
that's what we're trying to get.
-
This class looks like
something we could hook to.
-
We could do another wild card
to search so we don't have
-
to get this random.
-
We'll do deal_div.find element.
-
This is going to be single
-
now because we find in
the single element now.
-
John DeCrey: Yes.
Make sure it's find
-
element, not elements.
-
Jerik: Yes.
-
Then okay,
-
so from what was that first one?
-
We found this one and then
we drill down to that.
-
So from there we need
to go down and a tag,
-
so we need to go to this a tag.
-
So to do that, we'll
do a.a-text-normal.
-
That will bring us
down into this class.
-
Then we need to go down one
-
more into this deal
content module thing.
-
We can do a wildcard
search on that just
-
to make sure we're
getting the right thing.
-
Then before that last
closing parenthese do.text,
-
this will get the text from it.
-
This.text is actually
doing the same thing
-
as the get.attribute interHTML.
-
For some reason, text would
-
not work on this website though.
-
I have no idea why.
-
I will do a breakpoint and
-
this hour 24 would have
the text in there,
-
but for some reason when I would
-
do hour 24, that text
it wouldn't work.
-
I have no idea why.
-
Then now
-
we just need to
get the link deal,
-
links start, append,
-
and then using
-
that same deal_div
find_element, singular.
-
Then this one is right here.
-
So we just need to
drill down once
-
from that original div.
-
Then this one we're
-
going to do get attribute.
-
The attribute we want is this
HF that has the link to it.
-
That's that. This is
-
going to be a
dictionary in the end.
-
Title with links equals.
-
This dictionary zip thing is
just something I found that,
-
where you can take two lists and
-
this merges them
into a dictionary,
-
passing the deal_titles
and deal_links,
-
and then we can print the
title with the links.
-
John DeCrey: So ultimately that
-
becomes a dictionary title?
-
Jerik: Yep. This
title with links is
-
a dictionary that
has both of those.
-
There it is.
-
John DeCrey: Hey, look at that.
-
Jerik: Sweet.
-
John DeCrey: See, can
I get the price too?
-
Was that in there?
-
Jerik: They used to have
like one item per price,
-
but now they have these
-
are going to be a bunch
of deets in there.
-
John DeCrey: Fifteen
percent off.
-
Jerik: Yes.
-
John DeCrey: Interesting.
-
Jerik: Yes, there
is that script.
-
Should we go ahead and
move on to the KSL one?
-
John DeCrey: Let me talk
about this for a minute.
-
There's actually
business models that
-
are around this thing.
-
There's companies
that have that screen
-
scrap Amazon's site and
showing the trends of
-
pricing over time because
sometimes you can look at
-
certain products that
might be cheaper
-
in some time part of
the year than others.
-
Where you can maybe make a wise,
-
maybe a smarter decision
on when to buy something,
-
especially on the cost.
-
But even like business
opportunities are there as well.
-
I do want to also caution
a little bit with ethics
-
because with knowledge is
-
power and it comes
responsibility.
-
There is the ethics of things,
-
especially screen scraping and
-
other people's content
and stuff like that,
-
so just something
else to be aware
-
and consider on
certain projects.
-
But I think that's all I
have to say about that.
-
Actually at one time I did have
-
some code fragment to
a pen to this to take
-
all the findings and put it
into a vocal database like
-
SQL-lite but I think
we'll skip that for now.
-
But we could easily extend
this script and save
-
it to a database or even to
a CSV file or something.
-
That's pretty much for all the
scripts that we're giving.
-
We could easily save off.
-
You want to move on
to the next one?
-
Jerik: Yes.
-
John DeCrey: Go ahead and if you
-
want to submit this one into
-
canvas in the Amazon I
believe it's just called,
-
Amazon Web Scrape Lab,
-
so you can submit that into
that assignment in Canvas and
-
then we'll go to the next one.
-
Jerik: This one is
-
the one my testing I
-
found that furniture gets
posted the most often.
-
[NOISE] What this is
-
going to do is it's going
to go to this page,
-
like we're searching
for furniture to buy,
-
it's going to get
this first listing
-
then it's going to save that off
-
and check again
every 15 seconds.
-
Once this changes, then
we know a new ad has been
-
posted and it's going to tell us
-
the newest ad that was listed.
-
One thing to note on this is
-
these first three ads are
like sponsored spots.
-
They were posted days ago,
-
so when we do our CSS selectors,
-
we're going to find
this fourth one.
-
But anyway, so do
-
the new file and then same
thing with the other scripts,
-
stuff is just a set up.
-
Then [NOISE] I'm going to
-
put this setting
section at the top.
-
These are things that you
might want to change.
-
If you want to actually
use this as a deal finder,
-
you can put in
-
a different search criteria
-
or you can have it
checked more often.
-
In my experience, 15 seconds
between checking is a lot.
-
I think you'll get blacklisted
-
pretty quick if you did that.
-
When I was running
-
this deal finder for
-
its intended purpose to
actually try and find deals,
-
I would run it through
-
a BPN2 just so I didn't
get my IP banned again.
-
John DeCrey: What's
the recommendation
-
, maybe every minute.
-
Jerik: I think I was
doing every two minutes
-
and you could even go
-
bigger than that depending on
what you're searching for.
-
If you're searching for
something really niche,
-
you could even do like once
every 30 minutes probably?
-
John DeCrey: Yes.
-
Jerik: It just
depends on how much.
-
John DeCrey: I agree
to, o every 15 seconds
-
is pretty excessive.
-
I probably avoid anything
other than 30 seconds.
-
Jerik: Yes.
-
John DeCrey: Even every
minute would be adequate.
-
I was just thinking you could
use a random generator.
-
Random number generator.
-
Jerik: Yes. That's
actually a thing.
-
When I was working at
that job where we were
-
scraping the funeral
home websites,
-
we did like a bunch of
other things we were
-
scraping for too but it was
-
always like a cat and
mouse game between us
-
and the people not wanting
us to scrape their stuff.
-
But that was one of
the things we did
-
was add random timeouts,
-
so it seemed more like a human
was browsing their site.
-
John DeCrey: Jarik and
I used to work together
-
in digital forensics and
-
specifically mobile but we were
-
always looking for and
working with hackers that
-
have workarounds on
mobile security to
-
get to the data for criminal
investigations and whatnot.
-
But same thing,
-
it was always a cat and mouse.
-
The manufacturers
would always close
-
the security holes and then
-
the hackers would find new ones,
-
and the forensic industry
-
would expose and put
it in their product.
-
[LAUGHTER]
-
Jerik: Just for this exercise,
-
I'm going to put 15 seconds
-
here just so it
doesn't take too long.
-
This furniture section gets
posted too all the time.
-
I'll be surprised if it doesn't
-
find one in the
first 15 seconds.
-
John DeCrey: Yes. [NOISE]
-
Jerik: Now I'm going to make
a function that's going to
-
get the first listings info.
-
This listings info, call it,
-
get_first_listing_info
and this is
-
where we'll have our
first driver.get.
-
This is going to go to
that classified link.
-
Then like the other scripts,
-
we'll let it sleep
for five seconds.
-
Now we want to get the link
-
driver.find element by CSS.
-
I think on this one,
-
let's see if we want the link.
-
This is what we want.
-
Let me see if I can get it by
this item info title link.
-
Except we are going
to want it from
-
this fourth listing so
we can't do it by that.
-
[NOISE]
-
Let's see, we got
search results.
-
We'll want to go from
there to this Div,
-
to that class.
-
I'm looking for when it goes to
-
this first ad and then we
can drill down from there.
-
These will be posted
somewhere too, right John?
-
This will be a long one.
-
John DeCrey: If you want
to post in the chat
-
then I'll put it in the Canvas.
-
Jerik: Okay.
-
I'll send it to you
-
after the recording is done.
-
This can be a div to
a section to a div,
-
to another div and by the way,
-
this is going from
-
the search result to
the div section div.
-
Then here I had to do a div
-
and then this nth child,
-
this is saying,
get the first one.
-
That's just saying, get
this first div after
-
this div so this is getting
-
that and then we can do
-
another nth child here to
get the fourth section,
-
which is the third that we want.
-
After that the sectioned
-
nth child the fourth one.
-
Then from there we go
-
to another div,
-
the 2dh div, and
-
then h_2 div and the
link, it's a long one.
-
It looks like there's a tag.
-
Then we'll do the
same thing we did on
-
the Amazon one we'll get
the attribute intrif.
-
John DeCrey: Okay,.
-
Jerik: I hope I typed
that all in right,
-
but we'll see, then the title.
-
Then this is going to be
-
similar from Oh,
-
wait, this is really similar.
This is the same thing.
-
Because here's the
title of this brand
-
new velvet ivory couch.
-
It's the same selector.
-
We're just getting the
text from it this time.
-
John DeCrey: I don't know
if I want a velvet couch.
-
Jerik: I know. It
look pretty though.
-
That's one of those
couches that it's
-
just a show couch. You
don't actually use it.
-
John DeCrey: Yeah.
-
Jerik: This function will return
-
the link in the title.
-
John DeCrey: When you do
two returns like that,
-
it's returning as a
tapple, is that right?
-
Jerik: Yeah. Now,
-
we can call this function
and save it to listing info.
-
Then we're going to
save off the link,
-
so we can check it
again in 15 seconds.
-
John DeCrey: That's our Delta.
-
Jerik: Yep.
-
John DeCrey: Sort of thing.
-
Jerik: The first listing link,
-
temp equals listening info
-
and it's the first thing
in there, the link.
-
The title is going to
be the second thing.
-
Scripts like this to
you, I like to print
-
out like what's happening
-
if it's going to be
running for a while.
-
John DeCrey: That is for sure.
-
Jerik: We'll print this out
-
just to log what's going on.
-
First, listing title.
-
It's going to be the title,
-
and then we'll print
out the link to you.
-
I'm going to add
-
a counter so we can print out
how many times this is run.
-
Then now, I'm going to
make a while loop and this
-
is where it's just going to
run until it finds a new ad,
-
so it'd be, while true
-
that means it's
just going to run
-
forever until I
tell it to break,
-
we'll increase the
check count and
-
then we'll sleep for our
time we specified up here.
-
John DeCrey: Okay.
-
Jerik: You print out
another log message.
-
This will print out
that it's checking
-
and then we'll get
a new listing info.
-
That is, we'll get
it from the get
-
first listing info
function we made.
-
Then first list link
-
can be listing info[0].
-
John DeCrey: So info[0] was
the title or the or no?
-
Jerik: Zero is the link
and then 1 is the title.
-
John DeCrey: Okay,.
-
Jerik: One, and then
-
this is where we'll check to see
-
if it's different or not.
-
If it's different we
know it's a new ad so if
-
first listing link temp
-
does not equal
first listing link,
-
this is when we know
it's different.
-
John DeCrey: Lets
checked in our Delta?
-
Jerik: Yep.
-
John DeCrey: There's the new ad.
-
Jerik: Man, this break
will stop the program.
-
John DeCrey: Okay.
-
Jerik: We'll see if this works.
-
John DeCrey: There's different
options that you can put in here.
-
If you were to
actually use this for
-
your own self and
looking for something,
-
and once it found something
that matches your criteria,
-
you can have it email,
-
send an email to you and
-
you probably even do a text
message, couldn't you?
-
Jerik: Probably,
there's probably
-
a module for that, to be honest.
-
I get email.
-
It was pretty easy
to set up to email.
-
There's SMTP module that I used.
-
John DeCrey: What's the error
that we got there on the bottom?
-
Jerik: That was that stuff.
-
I'm not sure what's
the logging that
-
whether it's selenium or chrome.
-
John DeCrey: We can
ignore that stuff?
-
Jerik: Yeah.
-
John DeCrey: Okay.
-
Jerik: I need to.
-
John DeCrey: Objection.
-
Jerik: A copy of that.
-
John DeCrey: This is
attempt of number one.
-
In 15 seconds we'll check
to see if there's a new ad.
-
Number two, has there been
new ads being posted?
-
Jerik: One just got posted.
-
John DeCrey: Sweet.
-
Jerik: You can go
cop yourself as
-
Freeman Style grandfather clock.
-
John DeCrey: Let's
look at that URL,.
-
Jerik: Nice.
-
John DeCrey: Brand new ad that
we just found within seconds.
-
Jerik: Yep.
-
John DeCrey: Very cool.
-
Jerik: That is it.
-
John DeCrey: Then we can
change it to headless,
-
browsers are not showing
while your script is running.
-
Jerik: That is all I have.
-
I'll turn it back to you.
Usually, ask questions,
-
but back to you John.
-
John DeCrey: Many thanks.
This concludes that lab.
-
If you would submit this into
-
the canvas that's called KSL,
-
it's just called KSL Lab.
-
Many thanks. Jerkally,
-
I appreciate all
the time that you
-
have spend on this and always
-
enjoyable talking
with you and hearing
-
your stories so I
really appreciate it.
-
Jerik: No problem.
Thanks for having me.
