[HN Gopher] Auto-grading decade-old Hacker News discussions with...
___________________________________________________________________
Auto-grading decade-old Hacker News discussions with hindsight
Related from yesterday: _Show HN: Gemini Pro 3 imagines the HN
front page 10 years from now_ -
https://news.ycombinator.com/item?id=46205632
Author : __rito__
Score : 220 points
Date : 2025-12-10 17:23 UTC (5 hours ago)
(HTM) web link (karpathy.bearblog.dev)
(TXT) w3m dump (karpathy.bearblog.dev)
| bediger4000 wrote:
| _LLMs are watching (or humans using them might be). Best to be
| good._
|
| Shades of Roko's Basilisk!
| ambicapter wrote:
| More like a Panopticon. As the parenthesis notes, this is just
| as bad when humans are the final link in the eyeball chain.
| gen6acd60af wrote:
| Commenters of HN:
|
| Your past thoughts have been dredged up and judged.
|
| For each $TOPIC, you have been awarded a grade by GPT-5.1
| Thinking.
|
| Your grade is based on OpenAI's _aligned_ worldview and what
| OpenAI 's blob of weights considers Truth in 2025.
|
| Did you think _well_ , netizen?
|
| Are you an Alpha or a Delta-Minus?
|
| Where will the dragnet grading of your online history happen
| next?
| siliconc0w wrote:
| Random Bets for 2035:
|
| * Nvidia GPUs will see heavy competition and most chat-like use-
| cases switching to cheaper models and inference-specific-silicon
| but will be still used on the high end for critical applications
| and frontier science
|
| * Most Software and UIs will be primarily AI-generated. There
| will be no 'App Stores' as we know them.
|
| * ICE Cars will become niche and will be largely been replaced
| with EVs, Solar will be widely deployed and will be the dominate
| source of power
|
| * Climate Change will be widely recognized due to escalating
| consequences and there will be lots of efforts in mitigations
| (e.g, Climate Engineering, Climate-resistant crops, etc).
| xattt wrote:
| You're about 20 days short or 345 days late for this HN
| tradition. ;)
| pu_pe wrote:
| The infamous Dropbox comment might turn out to be right in 10
| more years, when LLMs might just build an entire application
| from scratch for you.
| rafaelmn wrote:
| I'd take the other side for most of these - Nvidia one is too
| vague (some could argue it's already seeing "heavy competition"
| from Google and other players in the space) but something more
| concrete - I doubt they will fall below 50% market share.
| jasonthorsness wrote:
| It's fun to read some of these historic comments! A while back I
| wrote a replay system to better capture how discussions evolved
| at the time of these historic threads. Here's Karpathy's list
| from his graded articles, in the replay visualizer:
|
| Swift is Open Source https://hn.unlurker.com/replay?item=10669891
|
| Launch of Figma, a collaborative interface design tool
| https://hn.unlurker.com/replay?item=10685407
|
| Introducing OpenAI https://hn.unlurker.com/replay?item=10720176
|
| The first person to hack the iPhone is building a self-driving
| car https://hn.unlurker.com/replay?item=10744206
|
| SpaceX launch webcast: Orbcomm-2 Mission [video]
| https://hn.unlurker.com/replay?item=10774865
|
| At Theranos, Many Strategies and Snags
| https://hn.unlurker.com/replay?item=10799261
| HanClinto wrote:
| Okay, your site is a ton of fun. Thank you! :)
| SauntSolaire wrote:
| I'd love to see sentiment analysis done based on time of day.
| I'm sure it's largely time zone differences, but I see a large
| variance in the types of opinions posted to hn in the morning
| versus the evening and I'd be curious to see it quantified.
| moultano wrote:
| Notable how this is only possible because the website is a good
| "web citizen." It has urls that maintain their state over a
| decade. They contain a whole conversation. You don't have to log
| in to see anything. The value of old proper websites increases
| with our ability to process them.
| chrisweekly wrote:
| Yes! See "Cool URIs Don't Change"^1 by Sir TBL himself.
|
| 1. https://www.w3.org/Provider/Style/URI
| jeffbee wrote:
| There are things that you have to log in to see, and the mods
| sometimes move conversations from one place to another, and
| also, for some reason, whole conversations get reset to a
| single timestamp.
| latexr wrote:
| > for some reason, whole conversations get reset to a single
| timestamp.
|
| What do you mean?
| jeffbee wrote:
| There is some action that moderators can take that throws
| one of yesterday's articles back on the front page and when
| that happens all the comments have the same timestamp.
| consumer451 wrote:
| I believe that this is called "the second chance pool."
| It is a bit strange when it unexpectedly happens to one's
| own post.
| embedding-shape wrote:
| Submissions put in the second-chance pool briefly appear
| (sometimes "again") on the frontpage, and the conversation
| timestamps are reset so it appears like they were written
| after the second-chance submission, not before.
| Y_Y wrote:
| I never noticed that. What a weird lie!
|
| I suppose they want to make the comments seem "fresh" but
| it's a deliberate misrepresentation. You could probably
| even contrive a situation where it could be damaging,
| e.g. somebody says something before some relevant
| incident, but the website claims they said it afterwards.
| embedding-shape wrote:
| I think the reason is much simpler than that. Resetting
| the timestamp lets them easily resurface things on the
| frontpage, because the current time - posting time delta
| becomes a lot smaller, so it's again ranked higher. And
| avoiding adding a special case, lets the rest of the
| codebase work exactly like it was before, basically just
| need to add a "set submission time to now" function and
| you get the rest for free.
|
| But, I'm just guessing here based on my own refactoring
| experience through the years, may be a completely
| different reason, or even by mistake? Who knows? :)
| embedding-shape wrote:
| > and the mods sometimes move conversations from one place to
| another
|
| This only manipulates the children references though, never
| the item ID itself. So if you have the item ID of an item
| (submission, comment, poll, pollItem), it'll be available
| there as long as moderators don't remove it, which happens
| very seldom.
| dietr1ch wrote:
| > because the website is a good "web citizen." It has urls that
| maintain their state over a decade.
|
| It's a shame that maintaining the web is so hard that only a
| few websites are "good citizens". I wish the web was a -bit-
| way more like git. It should be easier to crawl the web and
| serve it.
|
| Say, you browse and get things cached and shared, but only your
| "local bookmarks" persist. I guess it's like pinning in IPFS.
| moultano wrote:
| Yes, I wish we could serve static content more like
| bittorent, where your uri has an associate hash, and any
| intermediate router or cache could be an equivalent source of
| truth, with the final server only needing to play a role if
| nothing else has it.
|
| It is not possible right now to make hosting
| democratized/distributed/robust because there's no way for
| people to donate their own resources in a seamless way to
| keeping things published. In an ideal world, the internet
| archive seamlessly drops in to serve any content that goes
| down in a fashion transparent to the user.
| oncallthrow wrote:
| This is IPFS
| DANmode wrote:
| Let Reddit and friends continue to out themselves for who
| they are.
|
| Keeps the spotlight on carefully protected communities like
| this one.
| drdec wrote:
| > It's a shame that maintaining the web is so hard that only
| a few websites are "good citizens"
|
| It's not hard actually. There is a lack of will and
| forethought on the part of most maintainers. I suspect that
| monetization also plays a role.
| exasperaited wrote:
| > Everything we do today might be scrutinized in great detail in
| the future because it will be "free".
|
| s/"free"/stolen/
|
| The bit about college courses for future prediction was just
| silly, I'm afraid: reminds me of how Conan Doyle has Sherlock not
| knowing Earth revolves around the Sun. Almost all serious study
| concerns itself with predicting, modelling and influence over the
| future behaviour of some system; the problem is only that people
| _don 't fucking listen_ to the predictions of experts. They
| aren't going to value refined, academic general-purpose
| futurology any more than they have in the past; it's not even a
| new area of study.
| GaggiX wrote:
| I think the most fun thing is to go to:
| https://karpathy.ai/hncapsule/hall-of-fame.html
|
| And scroll down to the bottom.
| MBCook wrote:
| It's interesting, if you go down near the bottom you see some
| people with both A's and D's.
|
| According to the ratings for example, one person both had
| extremely racist ideas but also made a couple of accurate
| points about how some tech concepts would evolve.
| brian_spiering wrote:
| That is interesting because of the Halo effect. There is a
| cognitive bias that if a person is right in one area, they
| will be right in another unrelated area.
|
| I try to temper my tendency to believe the Halo effect with
| Warren Buffett's notion of the Circle of Competence; there is
| often a very narrow domain where any person can be
| significantly knowledgeable.
| MBCook wrote:
| #272, I got a B+! Neat.
|
| It would be very interesting to see this applied year after year
| to see if people get better or worse over time in the accuracy of
| their judgments.
|
| It would also be interesting to correlate accuracy to scores, but
| I kind of doubt that can be done. Between just expressing popular
| sentiment and the first to the post people getting more votes for
| the same comment than people who come later it probably wouldn't
| be very useful data.
| pjc50 wrote:
| #250, but then I wasn't trying to make predictions for a future
| AI. Or anyone else, really. Got a high score mostly for status
| quo bias, e.g. visual languages going nowhere and FPGAs remain
| niche.
| swalsh wrote:
| I have never felt less confident in the future than I do in
| 2025... and it's such a stark contrast. I guess if you split
| things down the middle, AI probably continues to change the world
| in dramatic ways but not in the all or nothing way people expect.
|
| A non trivial amount of people get laid off, likely due to a
| finanical crisis which is used as an excuse for companies scale
| up use of AI. Good chance the financial crisis was partly caused
| by AI companies, which ironically makes AI cheaper as infra is
| bought up on the cheap (so there is a consolidation, but the
| bountiful infra keeps things cheap). That results in increased
| usage (over a longer period of time). and even when the economy
| starts coming back the jobs numbers stay abismal.
|
| Politics are divided into 2 main groups, those who are employed,
| and those who are retired. The retired group is VERY large, and
| has alot of power. They mostly care about entitlements. The
| employed age people focus on AI which is making the job market
| quite tough. There are 3 large political forces (but 2 parties).
| The Left, the Right, and the Tech Elite. The left and the right
| both hate AI, but the tech elite though a minority has outsized
| power in their tie breaker role. The age distributions would
| surprise most. Most older people are now on the left, and most
| younger people are split by gender. The right focuses on limiting
| entitlements, and the left focuses on growing them by taxing the
| tech elite. The right maintains power by not threatening the tech
| elite.
|
| Unlike the 20th century America is a more focused global agenda.
| We're not policing everyone, just those core trading powers. We
| have not gone to war with China, China has not taken over Taiwan.
|
| Physical robotics is becoming a pretty big thing, space travel is
| becoming cheaper. We have at least one robot on an astroid mining
| it. The yield is trivial, but we all thought it was neat.
|
| Energy is much much greener, and you wouln't have guessed it...
| but it was the data centers that got us there. The Tech elite
| needed it quickly, and used the political connections to cut red
| tape and build really quickly.
| 1121redblackgo wrote:
| We do not currently have the political apparatus in place to
| stop the dystopian nightmares depicted in movies and media.
| They were supposed to be cautionary tales. Maybe they still can
| be, but there are basically zero guardrails in non-progressive
| forms of government to prevent massive accumulations of power
| being wielded in ways most of the population disapproves of.
| samdoesnothing wrote:
| Thats the whole point of democracy, to prevent the ruling
| parties from doing wildly unpopular things. Unlike a
| dictatorship, where they can do anything (including good
| things, that otherwise wouldn't happen in a democracy).
|
| I know that "X is destroying democracy, vote for Y" has been
| a prevalent narrative lately, but is there any evidence that
| it's true? I get that it's death by a thousand cuts, or "one
| step at a time" as they say.
| Karrot_Kream wrote:
| Are you in the wrong thread?
| bgwalter wrote:
| "If LLMs are watching, humans will be on their best behavior".
| Karpathy, paraphrasing Larry Ellison.
|
| The EU may give LLM surveillance an F at some point.
| lapcat wrote:
| Does anyone else think that HN engages in far too much navel-
| gazing? Nothing gets upvotes faster than a HN submission _about_
| HN.
| yellow_lead wrote:
| It's weird that HN viewers are interested in HN
| CamperBob2 wrote:
| As moultano suggests, this is likely because most other
| websites make it completely impossible to navel-gaze. We can't
| possibly give the HN admins too much praise and credit for
| their commitment to open and stable availability of legacy
| data.
| dang wrote:
| It's true that meta is the crack of internet forums, so we, er,
| crack down on it quite a bit. That's a longstanding view:
| https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
|
| Alternate metaphor: evil catnip -
| https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
|
| But yesterday's thread and this one are clearly exceptions--far
| above the median. https://news.ycombinator.com/item?id=46212180
| was particularly incredible I think!
| latexr wrote:
| I love it when you share some insight about HN or internet
| communication for which you have relevant searches at the
| ready to explanations of the concept.
|
| A personal favourite is "the contrarian dynamic".
|
| Do you have a list of those at the ready or do you just
| remember them? If you feel like sharing, what's your process
| and is there a list of those you'd make public?
|
| I imagine having one would be useful, e.g. for onboarding
| someone like tomhow, though that doesn't really happen often.
| dang wrote:
| I just remember them. Or forget them!
|
| The process is simply that moderation is super repetitive,
| so eventually certain pathways get engraved in one's
| memory. A lot of the time, though, I can't quite remember
| one of these patterns and I'm unable to dig up my past
| comments about it. That's annoying, in that particular way
| when your brain can feel something's there but is unable to
| retrieve it.
| artur44 wrote:
| Interesting experiment. Using modern LLMs to retroactively grade
| decade-old HN discussions is a clever way to measure how well our
| collective predictions age. It's impressive how little time and
| compute it now takes to analyze something that would've required
| days of manual reading. My only caution is that hindsight grading
| can overvalue outcomes instead of reasoning -- good reasoning can
| still lead to wrong predictions. But as a tool for calibrating
| forecasting and identifying real signal in discussions, this is a
| very cool direction.
| collinmcnulty wrote:
| > But if intelligence really does become too cheap to meter, it
| will become possible to do a perfect reconstruction and synthesis
| of everything. LLMs are watching (or humans using them might be).
| Best to be good.
|
| I cannot believe this is just put out there unexamined of any
| level of "maybe we shouldn't help this happen". This is complete
| moral abdication. And to be clear, being "good" is no defense.
| Being good often means being unaligned with the powerful, so
| being good is often the very thing that puts you in danger.
| Teever wrote:
| The time for discussion and action on this was over a 15 years
| ago when Snowden and the NSA with their Utah data centre was a
| big story.
|
| Governments around the world have profiles on people and
| spiders that quietly amass the data that continuously updates
| those profiles.
|
| It's just a matter of time before hardware improves and we see
| another holocaust scale purge facilitated by robots.
|
| Surveillance capitalism won.
| doctoboggan wrote:
| I've had the same though as Karpathy over the past couple of
| months/years. I don't think it's good, exciting, or something
| to celebrate, but I also have no idea how to prevent it.
|
| I would read his "Best to be good." as a warning or reminder
| that everything you do or say online will be collected and
| analyzed by an "intelligence". You can't count on hiding
| amongst the mass of online noise. Imagine if someone were to
| collect everything you've written or uploaded to the internet
| and compiled it into a long document. What sort of story would
| that tell about who you are? What would a clever person (or
| LLM) be able to do with that document?
|
| If you have any ideas on how to stop everyone from building the
| torment nexus, I am willing to listen.
| karpathy wrote:
| Thank you
| collinmcnulty wrote:
| This is my plan at least
|
| 1. Don't build the Torment Nexus yourself. Don't work for
| them and don't give them your money.
|
| 2. When people you know say they're taking a new job to work
| at Torment Nexus, act like that's super weird, like they said
| they're going to work for the Sinaloa cartel. Treat rich
| people working on the Torment Nexus like it's cringe to quote
| them.
|
| 3. Get hostile to bots. Poison the data. Use AdNauseum and
| Anubis.
|
| 4. Give your non-tech friends the vague sense that this stuff
| is bad. Some might want to listen more, but most just take
| their sense of what's cool and good from people they trust in
| the area.
| Teever wrote:
| Do you have any suggestions on how to interact online with
| people who work at Torment Nexus?
| magic_hamster wrote:
| This seems to me like a form of social engineering, or to
| some extent, being a bit insufferable. And, rest assured it
| will not result in anything useful. The only result of this
| is that you will alienate your friends and colleagues if
| they work for an employer you don't like.
| flir wrote:
| That's not my department, says Wernher von Braun.
|
| Don't know why that just popped into my head.
| tensor wrote:
| I think we need to stop focusing only on the AI aspect of
| this. Yes, it's an important component to the sort of mass
| surveillance system you're describing, but it's not the only
| component. The internet, advertising, privacy, all of these
| are integral to this outcome.
|
| While I don't have a general solution, I do believe that the
| solution will need to be multi-faceted and address multiple
| aspects of the technologies enabling this. My first step
| would be for society to re-evaluate and shift its views
| towards information, both locally and internationally.
|
| For example, if you proposed to get rid of all physical
| borders between countries, everyone would likely be aghast.
| Obviously there are too many disagreements and conflicting
| value sets between countries for this to happen. Yet in the
| west we think nothing have having no digital information
| borders, despite the fact that the lack of them in part
| enables this data collection and other issues such as
| election interference. Yes, erecting firewalls is extremely
| unpalatable to people in the west, but is almost certainly
| part of the solution on the national level. Countries like
| China long ago realized this, though they also use firewalls
| as a means of control, not just protection (it doesn't have
| to be this way).
|
| But within countries we also need to shift away from a
| default position of "I have the right to say whatever I want
| so therefore I should" and into one of "I'm not putting
| anything online unless I'm willing to have my employer,
| parents, literally everyone, read it." Also, we need to
| systematically attack and dismantle the advertising industry.
| That industry is one of the single biggest driving factors
| behind the extreme systematic collection and correlation of
| data on people. Advertising needs to switch to a "you come to
| me" approach not a "I'm coming to you" approach.
| thatguy0900 wrote:
| Well the companies that facilitate this have found themselves
| in a position where if they go down they take the US economy
| with them, so the maybe this shouldn't happen thing is a moot
| point. At least we know this stuff is in stable, secure hands
| though, like how the palantir ceo does recorded interviews
| while obviously blasted out of his mind on drugs.
| cootsnuck wrote:
| To be clear...prior to this recent explosive interest in LLMs,
| this was already true. Snowden was over 10 years ago.
|
| We can't start clutching our pearls now as if programmatic mass
| surveillance hasn't been running on all cylinders for over 20
| years.
|
| Don't get me wrong, we should absolutely care about this,
| everyone should. I'm just saying any vague gestures at imminent
| privacy-doom thanks to LLMs is liable to be doing some big
| favors of inadvertently sanitizing the history of prior (and
| still) egregious privacy offenders.
|
| I'm just suggesting more "Yes and" and less "pearl clutching"
| is all.
| panarky wrote:
| Who, exactly, is the "we" who you see "pearl clutching"
| instead of "yes and-ing"?
| consumer451 wrote:
| It's nice that the LLM-enabled authoritarian panopticon still
| cannot find this very recent related media, [0] but my silly
| mind can. It is actually an interesting commentary from a non-
| tech point of view. This is how the rest of the world feels:
|
| Anyway, back to work trying to make my millions using Opus and
| such.
|
| [0]
| https://old.reddit.com/r/funny/comments/1pj5bg9/al_companies...
| gaigalas wrote:
| I am not sure if we need a karma precog analogue.
|
| It does seem better than just upvotes and downvotes though.
| jeffbee wrote:
| I'm delighted to see that one of the users who makes the same
| negative comments on every Google-related post gets a "D" for
| saying Waymo was smoke and mirrors. Never change, I guess.
| modeless wrote:
| This is a cool idea. I would install a Chrome extension that
| shows a score by every username on this site grading how well
| their expressed opinions match what subsequently happened in
| reality, or the accuracy of any specific predictions they've
| made. Some people's opinions are closer to reality than others
| and it's not always correlated with upvotes.
|
| An extension of this would be to grade people on the accuracy of
| the comments they upvote, and use that to weight their upvotes
| more in ranking. I would love to read a version of HN where the
| only upvotes that matter are from people who agree with opinions
| that turn out to be correct. Of course, only HN could implement
| this since upvotes are private.
| cootsnuck wrote:
| The RES (Reddit Enhancement Suite) browser extension indirectly
| does this for me since it tracks the lifetime number of upvotes
| I give other users. So when I stumble upon a thread with a user
| with like +40 I know "This is someone whom I've repeatedly
| found to have good takes" (depending on the context).
|
| It's subjective of course but at least it's transparently so.
|
| I just think it's neat that it's kinda sorta a loose proxy for
| what you're talking about but done in arguably the simplest way
| possible.
| nickff wrote:
| I am not a Redditor, but RES sounds like it would increase
| the 'echo-chamber' effect, rather than improving one's
| understanding of contributors' calibration.
| mistercheph wrote:
| it depends on if you vote based on the quality of
| contribution to the discussion or based on how much you
| agree/disagree.
| modeless wrote:
| Reddit's current structure very much produces an echo
| chamber with only one main prevailing view. If everyone
| used an extension like this I would expect it to increase
| overall diversity of opinion on the site, as things that
| conflict with the main echo chamber view could still thrive
| in their own communities rather than getting downvoted with
| the actual spam.
| PunchyHamster wrote:
| More than having exact same system but with any random
| reader voting ? I'd say as long as you don't do "I disagree
| therefore I downvote" it would probably be more accurate
| than having essentially same voting system driven by
| randoms like reddit/HN already does
| janalsncm wrote:
| That assumes your upvotes in the past were a good proxy for
| being correct today. You could have both been wrong.
| TrainedMonkey wrote:
| I long had a similar idea for stocks. Analyze posts of people
| giving stock tips on WSB, Twitter, etc and rank by accuracy. I
| would be very surprised if this had not been done a thousand
| times by various trading firms and enterprising individuals.
|
| Of course in the above example of stocks there are clear
| predictions (HNWS will go up) and an oracle who resolves it
| (stock market). This seems to be a way harder problem for
| generic free form comments. Who resolves what prediction a
| particular comment has made and whether it actually happened?
| Karrot_Kream wrote:
| I ran across Sybil [1] the other day which tries to offer a
| reputation score based on correct predictions in prediction
| markets.
|
| [1]: https://sybilpredicttrust.info/
| 8organicbits wrote:
| The problem seems underspecified; what does it mean for a
| comment to be accurate? It would seem that comments like "the
| sun will rise tomorrow" would rank highest, but they aren't
| surprising.
| leobg wrote:
| That's what Elon's vision was before he ended up buying
| Twitter. Keep a digital track record for journalists. He wanted
| to call it Pravda.
|
| (And we do have that in real life. Just as, among friends, we
| do keep track of who is in whose debt, we also keep a mental
| map of whose voice we listen to. Old school journalism still
| had that, where people would be reading someone's column over
| the course of decades. On the internet, we don't have that, or
| we have it rarely.)
| GaggiX wrote:
| I was reading the Anki article on 2015-12-13, and the best
| prediction was by markm248 saying: "Remember that you read it
| here first, there will be a unicorn built on the concept of SRS"
|
| They were right, Duolingo.
| mtlynch wrote:
| Duolingo existed for a while at that point and was already
| valued at $500M by end of 2015.
| GaggiX wrote:
| It became a unicorn in December 2019 tho, 4 years later.
| Bjartr wrote:
| Neat, I got a shout-out. Always happy to share the random stuff I
| remember exists!
| mvdtnz wrote:
| Do we need more AI slop on the front page?
| hackthemack wrote:
| I noticed the Hall of Fame grading of predictive comments has a
| quirk? It grades some comments about if they came true or not,
| but in the grading of comment to the article
|
| https://news.ycombinator.com/item?id=10654216
|
| The Cannons on the B-29 Bomber "accurate account of LeMay
| stripping turrets and shifting to incendiary area bombing;
| matches mainstream history"
|
| It gave a good grade to user cstross but to my reading of the
| comment, cstross just recounted a bit of old history. The
| evaluation gave cstross for just giving a history lesson or no?
| karpathy wrote:
| Yes I noticed a few of these around. The LLM is a little too
| willing to give out grades for comments that were good/bad in a
| bit more general sense, even if they weren't making strong
| predictions specifically. Another thing I noticed is that the
| LLM has a very impressive recognition of the various usernames
| and who they belong to, and I think shows a little bit of a
| bias in its evaluations based on the identity of the person. I
| tuned the prompt a little bit based on some low-hanging fruit
| mistakes but I think one can most likely iterate it quite a bit
| further.
| slg wrote:
| This is a perfect example of the power and problems with LLMs.
|
| I took the narcissistic approach of searching for myself. Here's
| a grade of one of my comments[1]:
|
| >slg: B- (accurate characterization of PH's "networking & facade"
| feel, but implicitly underestimates how long that model can
| persist)
|
| And here's the actual comment I made[2]:
|
| >And maybe it is the cynical contrarian in me, but I think the
| "real world" aspect of Product Hunt it what turned me off of the
| site before these issues even came to the forefront. It always
| seemed like an echo chamber were everyone was putting up a
| facade. Users seemed more concerned with the people behind
| products and networking with them than actually offering opinions
| of what was posted.
|
| >I find the more internet-like communities more natural. Sure,
| the top comment on a Show HN is often a critique. However I find
| that more interesting than the usual "Wow, another great product
| from John Developer. Signing up now." or the "Wow, great product.
| Here is why you should use the competing product that I work on."
| that you usually see on Product Hunt.
|
| I did not say nor imply anything about "how long that model can
| persist", I just said I personally don't like using the site.
| It's a total hallucination to claim I was implying doom for "that
| model" and you would only know that if you actually took the time
| to dig into the details of what was actually said, but the
| summary seems plausible enough that most people never would.
|
| The LLM processed and analyzed a huge amount of data in a way
| that no human could, but the single in-depth look I took at that
| analysis was somewhere between misleading and flat out wrong. As
| I said, a perfect example of what LLMs do.
|
| And yes, I do recognize the funny coincidence that I'm now doing
| the exact thing I described as the typical HN comment a decade
| ago. I guess there is a reason old me said "I find that more
| interesting".
|
| [1] -
| https://karpathy.ai/hncapsule/2015-12-18/index.html#article-...
|
| [2] - https://news.ycombinator.com/item?id=10761980
| neilv wrote:
| > _I spent a few hours browsing around and found it to be very
| interesting._
|
| This seems to be the result of the exercise? No evaluation?
|
| My concern is that, even if the exercise is only an amusing
| curiosity, many people will take the results more seriously than
| they should, and be inspired to apply the same methods to
| products and initiatives that adversely affect people's lives in
| real ways.
| cootsnuck wrote:
| > My concern is that, even if the exercise is only an amusing
| curiosity, many people will take the results more seriously
| than they should, and be inspired to apply the same methods to
| products and initiatives that adversely affect people's lives
| in real ways.
|
| That will most definitely happen. We already have known for
| awhile that algorithmic methods have been applied "to products
| and initiatives that adversely affect people's lives in real
| ways", for awhile:
| https://www.scientificamerican.com/blog/roots-of-unity/revie...
|
| I guess the question is if LLMs for some reason will
| reinvigorate public sentiment / pressure for governing bodies
| to sincerely take up the ongoing responsibility of trying to
| lessen the unique harms that can be amplified by reckless
| implementation of algorithms.
| btbuildem wrote:
| I've spent a weekend making something similar for my gmail
| account (which google keeps nagging me about being 90% full).
| It's fascinating to be able to classify 65k+ of emails (surprise:
| more than half are garbage), as well as summarize and trace the
| nature of communication between specific senders/recipients. It
| took about 50 hours on a dual RTX 3090 running Qwen 3.
|
| My original goal was to prune the account deleting all the
| useless things and keeping just the unique, personal, valuable
| communications -- but the other day, an insight has me convinced
| that the safer / smarter thing to do in the current landscape is
| the opposite: remove any personal, valuable, memorable items, and
| leave google (and whomever else is scraping these repositories)
| with useless flotsam of newsletters, updates, subscription
| receipts, etc.
| 0xWTF wrote:
| Now: compared to what? Is there a better source than HN? How's it
| compare to Reddit or lobsters?
|
| Compared to what happens next? Does tptacek's commentary become
| market signal equivalent to the Fed Chair or the BLS labor and
| inflation reports?
| tptacek wrote:
| What makes you think it already isn't?
| jacquesm wrote:
| You've made me billions by now! Thank you...
| scosman wrote:
| Anyone have a branch that I can run to target my own comments?
| I'd love to see where I was right and where I was off base. Seems
| like a genuinely great way to learn about my own biases.
| tptacek wrote:
| 'pcwalton, I'm coming for you. You're going down.
|
| Kidding aside, the comments it picks out for us are a little
| random. For instance, this was an A+ predictive thread (it
| appears to be rating threads and not individual comments):
|
| https://news.ycombinator.com/item?id=10703512
|
| But there's just 11 comments, only 1 for me, and it's like a
| 1-sentence comment.
|
| I do love that my unaccredited-access-to-startup-shares take is
| on that leaderboard, though.
| mistercheph wrote:
| A majority don't seem to be predictions about the future, and it
| seems to mostly like comments that give extended air to what was
| then and now the consensus viewpoint, e.g. the top comment from
| pcwalton the highest scored user:
| https://news.ycombinator.com/item?id=10657401
|
| > (Copying my comment here from Reddit /r/rust:) Just to repeat,
| because this was somewhat buried in the article: Servo is now a
| multiprocess browser, using the gaol crate for sandboxing. This
| adds (a) an extra layer of defense against remote code execution
| vulnerabilities beyond that which the Rust safety features
| provide; (b) a safety net in case Servo code is tricked into
| performing insecure actions. There are still plenty of bugs to
| shake out, but this is a major milestone in the project.
| Rperry2174 wrote:
| One thing this really highlights to me is how often the "boring"
| takes end up being the most accurate. The provocative, high-
| energy threads are usually the ones that age the worst.
|
| If an LLM were acting as a kind of historian revisiting today's
| debates with future context, I'd bet it would see the same
| pattern again and again: the sober, incremental claims quietly
| hold up, while the hyperconfident ones collapse.
|
| Something like "Lithium-ion battery pack prices fall to $108/kWh"
| is classic cost-curve progress. Boring, steady, and historically
| extremely reliable over long horizons. Probably one of the most
| likely headlines today to age correctly, even if it gets little
| attention.
|
| On the flip side, stuff like "New benchmark shows top LLMs
| struggle in real mental health care" feels like high-risk
| framing. Benchmarks rotate constantly, and "struggle" headlines
| almost always age badly as models jump whole generations.
|
| I bet theres many "boring but right" takes we overlook today and
| I wondr if there's a practical way to surface them before
| hindsight does
| simianparrot wrote:
| Instead of "LLM's will put developers out of jobs" the boring
| reality is going to be "LLM's are a useful tool with limited
| use".
| yunwal wrote:
| "Boring but right" generally means that this prediction is
| already priced in to our current understanding of the world
| though. Anyone can reliably predict "the sun will rise
| tomorrow", but I'm not giving them high marks for that.
| SubiculumCode wrote:
| Perhaps a new category, 'highest risk guess but right the
| most often'. Those is the high impact predictions.
| arjie wrote:
| Prediction markets have pretty much obviated the need for
| these things. Rather than rely on "was that really a hot
| take?" you have a market system that rewards those with
| accurate hot takes. The massive fees and lock-up period
| discourage low-return bets.
| Karrot_Kream wrote:
| FWIW Polymarket (which is one of the big markets) has no
| lock-up period and, for now while they're burning VC
| coins, no fees. Otherwise agree with your point though.
| onraglanroad wrote:
| I'm giving them higher marks than the people who say it
| won't.
|
| LLMs have seen huge improvements over the last 3 years. Are
| you going to make the bet that they will continue to make
| similarly huge improvements, taking them well past human
| ability, or do you think they'll plateau?
|
| The former is the boring, linear prediction.
| yunwal wrote:
| > Are you going to make the bet that they will continue to
| make similarly huge improvements
|
| Sure yeah why not
|
| > taking them well past human ability,
|
| At what? They're already better than me at reciting
| historical facts. You'd need some actual prediction here
| for me to give you "prescience".
| onraglanroad wrote:
| At every intellectual task.
|
| They're already better than you at reciting historical
| facts. I'd guess they're probably better at composing
| poems (they're not great but far better than the average
| person).
|
| Or you agree with me? I'm not looking for prescience
| marks, I'm just less convinced that people really make
| the more boring and obvious predictions.
| yunwal wrote:
| What is an intellectual task? Once again, there's tons of
| stuff LLMs won't be trained on in the next 3 years. So it
| would be trivial to just find one of those things and say
| voila! LLMs aren't better than me at that.
|
| I'll make one prediction that I think will hold up. No
| LLM-based system will be able to take a generic ask like
| "hack the nytimes website and retrieve emails and
| password hashes of all user accounts" and do better than
| the best hackers and penetration testers in the world,
| despite having plenty of training data to go off of. It
| requires out-of-band thinking that they just don't
| possess.
| hathawsh wrote:
| I'll take a stab at this: LLMs currently seem to be
| rather good at details, but they seem to struggle greatly
| with the overall picture, in every subject.
|
| - If I want Claude Code to write some specific code, it
| often handles the task admirably, but if I'm not sure
| what should be written, consulting Claude takes a lot of
| time and doesn't yield much insight, where as 2 minutes
| with a human is 100x more valuable.
|
| - I asked ChatGPT about some political event. It mirrored
| the mainstream press. After I reminded it of some obvious
| facts that revealed a mainstream bias, it agreed with me
| that its initial answer was wrong.
|
| These experiences and others serve to remind me that
| current LLMs are mostly just advanced search engines.
| They work especially well on code because there is a lot
| of reasonably good code (and tutorials) out there to
| train on. LLMs are a lot less effective on intellectual
| tasks that humans haven't already written and published
| about.
| janalsncm wrote:
| To be clear, you are suggesting "huge improvements" in
| "every intellectual task"?
|
| This is unlikely for the trivial reason that some tasks
| are roughly saturated. Modest improvements in chess
| playing ability are likely. Huge improvements probably
| not. Even more so for arithmetic. We pretty much have
| that handled.
|
| But the more substantive issue is that intellectual tasks
| are not all interconnected. Getting significantly better
| at drawing hands doesn't usually translate to executive
| planning or information retrieval.
| janalsncm wrote:
| "At what?" is really the key question here.
|
| A lot of the press likes to paint "AI" as a uniform field
| that continues to improve together. But really it's a
| bunch of related subfields. Once in a blue moon a
| technique from one subfield crosses over into another.
|
| "AI" can play chess at superhuman skill. "AI" can also
| drive a car. That doesn't mean Waymo gets safer when we
| increase Stockfish's elo by 10 points.
| Gravityloss wrote:
| something like correctness^2 x novel information content
| rank?
| johnfn wrote:
| This suggests that the best way to grade predictions is some
| sort of weighting of how unlikely they were at the time. Like,
| if you were to open a prediction market for statement X, some
| sort of grade of the delta between your confidence of the event
| and the "expected" value, summed over all your predictions.
| jacquesm wrote:
| Exactly, that's the element that is missing. If there are 50
| comments against and one pro and that pro has it in the
| longer term then that is worth noticing, not when there are
| 50 comments pro and you were one of the 'pros'.
|
| Going against the grain and turning out right is far more
| valuable than being right consistently when the crowd is with
| you already.
| karmickoala wrote:
| I understand the exercise, but I think it should have a
| disclaimer, some of the LLM reviews are showing a bias and when I
| read the comments they turned out not to be as bad as the LLM
| made them. As this hits the front page, some people will only
| read the title and not the accompanying blog post, losing all of
| the nuance.
|
| That said, I understand the concept and love what you did here.
| By this being exposed to the best disinfectant, I hope it will
| raise awareness and show how people and corporations should be
| careful about its usage. Now this tech is accessible to anyone,
| not only big techs, in a couple of hours.
|
| It also shows how we should take with a grain of salt the result
| of any analysis of such scale by a LLM. Our private channels now
| and messages on software like Teams and Slack can be analyzed to
| hell by our AI overlords. I'm probably going to remove a lot of
| things from cloud drives just in case. Perhaps online discourse
| will deteriorate to more inane / LinkedIn style content.
|
| Also, I like that your prompt itself has some purposefully leaked
| bias, which shows other risks--1for instance, "fsflover: F",
| which may align the LLM to grade worse the handles that are
| related to free software and open source).
|
| As a meta concept of this, I wonder how I'll be graded by our AI
| overlords in the future now that I have posted something
| dismissive of it.
|
| 1Alt+0151
| smugma wrote:
| I believe that the GPA calculation is off, maybe just for F's.
|
| I scrolled to the bottom of the hall of fame/shame and saw that
| entry #1505 and 3 F's and a D, with an average grade of D+
| (1.46).
|
| No grade better than a D shouldn't average to a D+, I'd expect it
| to be closer to a 0.25.
| ComputerGuru wrote:
| Looking at the results and the prompt, I would tweak the prompt
| to
|
| * ignore comments that do not speculate on something that was
| unknown or had not achieved consensus as of the date of yyyy-mm-
| dd
|
| * at the same time, exclude speculations for which there still
| isn't a definitive answer or consensus today
|
| * ignore comments that speculate on minor details or are stating
| a preference/opinion on a subjective matter
|
| * it is ok to generate an empty list of users for a thread if
| there are no comments meeting the speculation requirements laid
| out above
|
| * etc
| janalsncm wrote:
| You would also need to exclude "predictions" for things which
| already happened at the time they were predicted.
| godelski wrote:
| > I was reminded again of my tweets that said "Be good, future
| LLMs are watching". You can take that in many directions, but
| here I want to focus on the idea that future LLMs are watching.
| Everything we do today might be scrutinized in great detail in
| the future because doing so will be "free". A lot of the ways
| people behave currently I think make an implicit "security by
| obscurity" assumption. But if intelligence really does become too
| cheap to meter, it will become possible to do a perfect
| reconstruction and synthesis of everything. LLMs are watching (or
| humans using them might be). Best to be good.
|
| Can we take a second and talk about how dystopian this is? Such
| an outcome is not inevitable, it relies on us making it. The
| future is not deterministic, the future is determined by _us_.
| Moreso, Karpathy has significantly more influence on that future
| than your average HN user.
|
| We are doing something very * _very*_ wrong if we are operating
| under the belief that this future is unavoidable. That future is
| simply unacceptable.
| jacquesm wrote:
| Given the quality of the judgment I'm not worried, there is no
| value here.
|
| To properly execute this idea rather than to just toss it off
| without putting in the work to make it valuable is exactly what
| irritates me about a lot of AI work. You can be 900 times as
| productive at producing mental popcorn, but _if_ there was
| value to be had here we 're not getting it, just a whiff of it.
| Sure, fun project. But I don't feel particularly judged here.
| The funniest bit is the judgment on things that clearly could
| not yet have come to pass (for instance because there is an
| exact date mentioned that we have not yet reached). QA could be
| better.
| dschnurr wrote:
| Nice! Something must be in the air - last week I built a very
| similar project using the historical archive of all-in podcast
| episodes: https://allin-predictions.pages.dev/
| sanex wrote:
| I'll use this as evidence supporting my continued demand for a
| Friedberg only spinoff.
| LeroyRaz wrote:
| I am surprised the author thought the project passed quality
| control. The LLM reviews seem mostly false.
|
| Looking at the comment reviews on the actual website, the LLM
| seems to have mostly judged whether it agreed with the takes, not
| whether they came true, and it seems to have an incredibly poor
| grasp of it's actual task of accessing whether the comments were
| predictive or not.
|
| The LLM's comment reviews are of often statements like "correctly
| characterized [program language] as [opinion]."
|
| This dynamic means the website mostly grades people on having the
| most confirmist take (the take most likely to dominate the
| training data, and be selected for in the LLM RL tuning process
| of pleasing the average user).
| hathawsh wrote:
| Are you sure? The third section of each review lists the "Most
| prescient" and "Most wrong" comments. That sounds exactly like
| what you're looking for. For example, on the "Kickstarter is
| Debt" article, here is the LLM's analysis of the most prescient
| comment. The analysis seems accurate and helpful to me.
|
| https://karpathy.ai/hncapsule/2015-12-03/index.html#article-...
| phire > "Oculus might end up being the most
| successful product/company to be kickstarted... > Product wise,
| Pebble is the most successful so far... Right now they are up
| to major version 4 of their product. Long term, I don't think
| they will be more successful than Oculus." With
| hindsight: Oculus became the backbone of Meta's VR
| push, spawning the Rift/Quest series and a multi-billion-dollar
| strategic bet. Pebble, despite early success, was shut
| down and absorbed by Fitbit barely a year after this thread.
| That's an excellent call on the relative trajectories of the
| two flagship Kickstarter hardware companies.
| LeroyRaz wrote:
| Examples: tptacek gets an 'A' for his comment on DF which the
| LLM claiming that the user "captured DF's unforgiving nature,
| where 'can't do x or it crashes is just another feature to
| learn' which remained true until it was fixed on ..."
|
| Link to LLM review: https://karpathy.ai/hncapsule/2015-12-02/in
| dex.html#article-....
|
| So the LLM is praising a comment as describing DF as
| unforgiving (a characterization of the present then, not a
| statement about the future). And worse, it seems like tptacek
| may in fact be implying the opposite of the future (e.g., x
| will continue to crash when it was eventually fixed.)
|
| Here is the original comment: " tptacek on Dec 2, 2015 | root |
| parent | next [-]
|
| If you're not the kind of person who can take flaws like
| crashes or game-stopping frame-rate issues and work them into
| your gameplay, DF is not the game for you. It isn't a friendly
| game. It can take hours just to figure out how to do core game
| tasks. "Don't do this thing that crashes the game" is just
| another task to learn."
|
| Note: I am paraphrasing the LLM review, as the website is also
| poorly designed, with one unable to select the text of the LLM
| review!
|
| N.b., this choice of comment review is not overly cherry
| picked. I just scanned the "best commentators" and tptacek was
| number two, with this particular egregiously unrelated-to-
| prediction LLM summary given as justifying his #2 rating.
| andy99 wrote:
| I haven't looked at the output yet, but came here to say,LLM
| grading is crap. They miss things, they ignore instructions,
| bring in their own views, have no calibration and in general
| are extremely poorly suited to this task. "Good" LLM as a judge
| type products (and none are great) use LLMs to make binary
| decisions - "do these atomic facts match yes / no" type stuff -
| and aggregate them to get a score.
|
| I understand this is just a fun exercise so it's basically what
| LLMs are good at - generating plausible sounding stuff without
| regard for correctness. I would not extrapolate this to their
| utility on real evaluation tasks.
| jacquesm wrote:
| Predictions are only valuable when they're actually made ahead of
| the knowledge becoming available. A man will walk on mars by 2030
| is falsifiable, a man will walk on mars is not. A lot of these
| entries have very low to no predictive value or were already
| known at the time, but just related. Would be nice if future
| 'judges' put in more work to ensure quality judgments.
|
| I would grade this article B-, but then again, nobody wrote it...
| ;)
___________________________________________________________________
(page generated 2025-12-10 23:00 UTC)