[HN Gopher] ChangeDetection, monitor any website change
___________________________________________________________________
ChangeDetection, monitor any website change
Author : serhack_
Score : 219 points
Date : 2023-09-01 08:27 UTC (14 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| shortrounddev2 wrote:
| A few years ago when I was a Junior engineer, I worked for a
| company which would provide us with free tickets to any sports
| team in our city. I'm not a sports fan, but my stepdad loves
| hockey. When the tickets went up, they got reserved pretty fast.
|
| I noticed that the site we used to reserve the tickets had a
| predictable slug like: `{company}-hockey-tickets-2016-season`,
| and `{company}-hockey-tickets-2015-season`. The sites also didn't
| change between each refresh.
|
| So, when it was nearly time for the 2017 season to start, I wrote
| a script which hashed a GET request of the site at
| `{company}-hockey-tickets-2017-season`. If the hash changed, it
| would send me a text message using twilio, and I would know to
| immediately get on a machine and reserve the tickets we wanted.
| After a couple false positives, the page eventually went up
| before it was announced and I reserved the tickets I wanted.
|
| Unfortunately, the office manager who did these things told me I
| couldn't jump the gun and un-reserved my tickets. After the
| official announcement went up, the spots I had originally
| reserved quickly went to someone else
| solardev wrote:
| It seems like bad company policy to let a bunch of techies
| fight over a limited resource via web requests. It's like
| network inspector gladiators. Maybe that was the point? Lol
| mhb wrote:
| Why is the hash of the request better than comparing strings?
| pc86 wrote:
| Content on the page with slug `{company}-hockey-
| tickets-2016-season` changed, not the slug itself.
| rented_mule wrote:
| I assume they meant they were hashing the response rather
| than hashing the request. Then you don't have to keep the
| entire response around for comparison, only its hash.
| HardlyCurious wrote:
| Yeah, that office manager probably knew who would be getting
| the tickets in advance.
| xenocratus wrote:
| And that's when I'd make that script and the knowledge behind
| it available publicly and advertise it on social media :)
| alach11 wrote:
| Working hard: writing a script to detect when the website
| goes live.
|
| Working smart: bring the office manager a pastry, ask them
| about their day, and drop hints about how excited you are for
| the tickets this year.
| nkko wrote:
| Engineer vs Non-Engineer approach.
| deedree wrote:
| You mean social engineering.
| wkat4242 wrote:
| Interesting! I've been making some scripts privately for specific
| tasks, like an Nvidia video card I wanted to buy. However many
| sites are really hostile to scraping these days. I'll give it a
| try,.
| weinzierl wrote:
| camelcamelcamel is a _really_ old service, but it 's still
| around and still works well for product price changes.
| hdivider wrote:
| Good stuff. Niche idea for ya: consider government contractors.
| Gov sites tend to be atrocious. Manual change tracking can mean
| you find a $1M RFP weeks earlier, which increases chance of
| winning. (Consider mda.mil. There's no email signup. What they
| consider a 'newsletter' is in fact a list of PDFs hosted on their
| site.)
|
| And that's federal gov contracting. State/local is even more of a
| Wild West. So much so that many, maany companies just never have
| the bandwidth to even find RFPs and navigate those ancient sites.
| solardev wrote:
| Funny how much better government might get if we just let their
| employees post on Craigslist
| MoOmer wrote:
| While that may be true if contractors/competitors for contracts
| were generally unaware of the contract until posting, that's
| usually not the reality. A lot of government contracting advice
| from experienced contractors is that if you're discovering it
| via a public source (i.e. the website) you're too late.
| hdivider wrote:
| This I am afraid is not true either, though. I know many
| dual-use companies who discovered an RFP (not just on SAM),
| _late_ -- i.e. with a proposal deathmarch -- and got through.
| MoOmer wrote:
| My understanding is that those scenarios are the exception,
| and finding obscure contracts that have gone undiscovered
| isn't really an untapped fount of free, missed,
| opportunity.
|
| That said, aggregators like bonfire etc. can get you pretty
| far if that's your strategy!
| hdivider wrote:
| Yeah it's probably true that such things are the
| exception. But for a startup, a chance at e.g. a $256k
| SBIR Phase I and then $1.9M Phase II and the modern
| follow-on options in e.g. the Army (called the CATALYST
| program and other things) -- this one exception turns the
| company into something real.
|
| So because the amounts are large (compared to commercial,
| in terms of when a customer first buys from a small
| company), those cases really make a huge difference.
| MoOmer wrote:
| Oh yeah, grants are great, but a completely different
| beast!
| Mo3 wrote:
| Kudos, this is a very feature rich solution. I like it. 8.99
| seems a little bit much, but I guess it's probably the sweet spot
| on the curve.
| johnnyworker wrote:
| For Windows, there is the OG https://www.aignes.com/
| serhack_ wrote:
| You can install it for free on your machine. I think it's a
| decent price, considering that they offer you to check a
| website even with a ratio of seconds.
| _flux wrote:
| I've used https://urlwatch.readthedocs.io but this definitely
| seems much easier to use--though possibly not quite as powerful
| regarding page filtering. But at least ChangeDetection supports
| jq which is already quite a nice feature in that department.
| kierenj wrote:
| First time I've noticed a GH repo having so much SEO in the About
| section! Not sure how I feel about it..
| serhack_ wrote:
| I actually found it via the about section so I guess it helped
| haha
| realPubkey wrote:
| They should add an url into the description text. This will
| render as backlink any there are many pages out there which
| replicate github-topic pages which will then give your site
| many links for free.
| mnstngr wrote:
| It's actually refreshing to see end-user documentation and
| answering the basic questions of "what can this do" and "why
| should I care" in the readme. Most projects jump straight into
| how to build it locally, with not even a screenshot of what
| they've created.
| Lalabadie wrote:
| Right? First time I run into a README file on Github and I
| think "Oh, this text is for search engines, not readers".
| serhack_ wrote:
| I'm OP. I'm not the developer of the project, but I think it
| deserved a post on HN because it's pretty an useful tool to track
| website changes. Setup can be done in a few minutes and the lead
| developer is always looking for solid feedbacks.
| TekMol wrote:
| It cannot be a Show HN then:
|
| https://news.ycombinator.com/showhn.html
|
| Please change the title.
| serhack_ wrote:
| Uops, sorry
| champagnepapi wrote:
| How does this handle A/B tests that sites are running?
| masukomi wrote:
| wouldn't that depend entirely on which group its request got
| put in, which would be something it would have no control over?
| abbaselmas wrote:
| my favorite one is visualping.io
| mchenier wrote:
| https://visualping.io does a nice similar job and is free for
| moderate personal use.
| cloudyporpoise wrote:
| Curious how this handles sites behind CloudFlare using bot
| detection.
| acheong08 wrote:
| I see selenium and playwright in the dependencies. I would've
| just gone with tls-client
| _boffin_ wrote:
| It's not too difficult to get past cloudflare using puppeteer.
| Being normal is the key.
| mcstempel wrote:
| Yeah, +1. Even vanilla puppeteer is pretty successful against
| Cloudflare
| accidbuddy wrote:
| Anyone has been utilized this tool to monitoring Amazon Prices
| (with login)?
| stewx wrote:
| Use CamelCamelCamel
| accidbuddy wrote:
| As far as I know, doesn't work to Amazon.br (Brazil). Thanks
| by the reply.
| robk wrote:
| Have you tried keepa?
| Aulig wrote:
| I've been self-hosting this for a year or so and it's pretty
| neat.
|
| It happens quite often that I need to wait for something which
| doesn't have a builtin alert.
|
| Currently I'm for example using it to get an alert when a flutter
| package has been updated on pub.dev
| sccxy wrote:
| I have used it for some time now. It has been very useful for me.
|
| Some use cases for me:
|
| * price changes
|
| * calendar changes for sport events
|
| * document changes for local gov
|
| * new firmware releases
|
| * terms of conditions changes
| ephbit wrote:
| Here's an addon for Firefox that also checks websites in regular
| intervals to detect changes:
|
| https://github.com/WaldiPL/webpageScanner
|
| Not affiliated with it and I don't know if it still works,
| haven't used it lately. When I used it, it worked to my
| satisfaction.
___________________________________________________________________
(page generated 2023-09-01 23:01 UTC)