Web Crawler for Scripts
reddit / selfhosted

----------------------------------------------------------------------
rfcity2 · 2023-11-02 01:24

I run the marketing for a website and don't have a direct connection
with the web team, and we run into the problem where the web team does
a  website change and removes certain scripts we implemented - without
notification. Is there a way to run a "script monitor" that checks
external webpages to see if a specific script is loaded? And if not,
to  send an email?

We will always know the script names and webpage URLs. Most pages
will have between 3-5 scripts. the scripts may either load directly
from  page source, or from a secondary js file that references it. It
would  be best to run the script hourly.

I'm happy to use some javascript or bash scripts with a cron job, but
am also tinkering with Scrappy.

Does anyone have experience on this and some ideas on which path to
start with?

----------------------------------------------------------------------
aft_punk · 2023-11-02 03:47

I’m a bit confused about the ask… are you wanting to find out if an
html file loads a certain js script?

----------------------------------------------------------------------
bladepen · 2023-11-02 05:13

Looking at it from another angle, you need to talk to the web
development team and get them to build something into their workflow
that ensures that your scripts are on the website - either they do not
remove them, they push them out along with their code, there is a
separate push of your scripts after their code, etc. Ownership,
maintenance, testing and support should be clearly understood by all
parties.

----------------------------------------------------------------------
RedditSlayer2020 · 2023-11-02 05:47

Easiest way is to connect to the developers. Your request really
sounds very fishy to me tbh

----------------------------------------------------------------------
lilolalu · 2023-11-02 08:32

You can let the scripts send heartbeat and monitor this with your
favourite monitoring app. Or use something like

https://github.com/dgtlmoon/changedetection.io

to poll for website changes.

----------------------------------------------------------------------
rfcity2 · 2023-11-02 10:18

Yep, that's it.

----------------------------------------------------------------------
rfcity2 · 2023-11-02 10:21

That is the route we've been trying, but the teams are different
companies and unfortunately they are not the best communicators.
it's not always that they remove the scripts, but they update the
template and the new template doesn't have the script. It's a lot of
them forgetting to make sure the scripts don't get dropped.