https://github.com/ggerganov/hnreplies Skip to content Sign up * Why GitHub? + Features + Mobile + Actions + Codespaces + Packages + Security + Code review + Issues + Integrations + GitHub Sponsors + Customer stories * Team * Enterprise * Explore + Explore GitHub + Learn and contribute + Topics + Collections + Trending + Learning Lab + Open source guides + Connect with others + The ReadME Project + Events + Community forum + GitHub Education + GitHub Stars program * Marketplace * Pricing + Plans + Compare plans + Contact Sales + Education [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} ggerganov / hnreplies Public * Notifications * Fork 0 * Star 8 * Scrape Hacker News replies GPL-3.0 License 8 stars 0 forks Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Wiki * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Wiki * Security * Insights master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 1 branch 0 tags Code Latest commit @ggerganov ggerganov Update README.md ... 6984319 Jan 20, 2022 Update README.md 6984319 Git stats * 9 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time public Update comments + minor cpp refactoring Jan 20, 2022 .gitignore Add Makefile Jan 19, 2022 LICENSE Create LICENSE Jan 20, 2022 Makefile Add Makefile Jan 19, 2022 README.md Update README.md Jan 20, 2022 json.h Create README.md Jan 20, 2022 main.cpp Update comments + minor cpp refactoring Jan 20, 2022 poll.sh Update comments + minor cpp refactoring Jan 20, 2022 View code HNReplies Motivation How it works? How to use? ./public/get.php ./ public/index.html README.md HNReplies A service for scraping the latest replies to HN users. Sample API endpoint: https://hnreplies.ggerganov.com/get.php # # Get latest replies to "dang" # $ curl "https://hnreplies.ggerganov.com/get.php?u=dang&print=pretty" { "replies": [ { "by": "dncornholio", "id": 30008834, "parent": 30008561, "text": "Agreed! Next time I will gather more arguments.", "time": 1642686636, "type": "comment" }, { "by": "MaxBarraclough", "id": 30008321, "parent": 30008042, "text": "Got it, thanks.", "time": 1642683577, "type": "comment" }, ... ] } Motivation Currently the Hacker News forum lacks the functionality to notify its users when another user has responded to their comment. In some cases this can cause a user to miss a reply in a discussion in which they had participated recently. Some users consider this to be a feature rather than a deficiency. In any case, this is a simple attempt to offer such functionality in the form of a 3rd party service. The HNReplies service tracks all comments posted on HN in realtime by querying the official HN API. The comments are then grouped by their parent's username by storing them as files in respective directories on the disk. The collected data can then be used to efficiently query for the latest replies to a certain user. The service is very basic and can be easily self-hosted on a low-end machine. It depends only on curl. Although I implemented this mostly for educational purposes, maybe it can potentially find some useful applications. For example, you can write a script that sends an e-mail notification every time someone responds to you on HN. All it takes is to query the HNReplies service with your username every ~5 minutes and check for new replies. How it works? The straightforward implementation is 20-30 lines of Bash script (./ poll.sh): * We poll the HN API every 30 seconds to get all items that have been updated recently * We then iterate the items and look for comments (i.e. Stories, Polls, etc. are ignored) * For every comment, we query the HN API about its parent item in order to get the username that they are replying to * We store the updated item in a folder named as the parent username This process produces the following directory structure: data/ +-- asciimov/ | +-- 29015505 +-- AshamedCaptain/ | +-- 30010021 | +-- 30011553 +-- bobcostas55/ | +-- 30007698 +-- CobrastanJorji/ | +-- 30012581 +-- dack/ | +-- 30013307 +-- daveaiello/ | +-- 30012555 | +-- 30013087 | +-- 30013242 | +-- 30013325 +-- dberhane/ | +-- 30007270 +-- deadcoder0904/ | +-- 29996499 ... Each leaf is a text file with the JSON result of querying the HN API about the corresponding item Id. For example, the file ./data/mayoff/ 2921983 contains the JSON obtained from the HN API query: https://hacker-news.firebaseio.com/v0/item/2921983.json Sample result: $ cat data/mayoff/2921983 { "by" : "norvig", "id" : 2921983, "kids" : [ 2922097, 2922429, 2924562, 2922709, 2922573, 2922140, 2922141 ], "parent" : 2921506, "text" : "Aw shucks, guys ... you make me blush with your compliments.

Tell you what, Ill make a deal: I'll keep writing if you keep reading. K?", "time" : 1314211127, "type" : "comment" } Running the service continuously will accumulate new HN data and organize it in the described way. The initial Bash implementation turned out to be a bit inefficient for a low-end machine, so I also added a C++ implementation that performs much better. The source code is in the ./main.cpp file and it essentially does the same thing as the Bash script using libcurl. To run it: git clone https://github.com/ggerganov/hnreplies cd hnreplies make ./poll The service performs about ~200 requests / minute to the HN API. The actual value depends on the activity on HN. How to use? Sample usages are given in the ./public folder of this repo. * ./public/get.php This is a simple PHP endpoint that uses the accumulated data by the HNReplies service to return the latest replies to a user. I've hosted an example on https://hnreplies.ggerganov.com/ and you can use it as follows: # # Get latest replies to "dang" # $ curl "https://hnreplies.ggerganov.com/get.php?u=dang&print=pretty" Or directly in the browser: https://hnreplies.ggerganov.com/get.php?u=dang&print=pretty The returned replies are orderer chronologically. You can modify the PHP script to provide some other data that you might want. * ./public/index.html A basic web page that sends an XHR request to the PHP endpoint and displays the results in HTML. For example, checkout the following URL: https://hnreplies.ggerganov.com/index.html?u=dang About Scrape Hacker News replies Resources Readme License GPL-3.0 License Stars 8 stars Watchers 1 watching Forks 0 forks Releases No releases published Packages 0 No packages published Languages * C++ 71.7% * Shell 14.3% * HTML 8.1% * PHP 5.5% * Makefile 0.4% * (c) 2022 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.