[HN Gopher] Mapping Hacker News to find who knows what in the HN...
___________________________________________________________________
Mapping Hacker News to find who knows what in the HN community
Author : robg
Score : 115 points
Date : 2024-07-25 15:04 UTC (8 hours ago)
(HTM) web link (blog.wilsonl.in)
(TXT) w3m dump (blog.wilsonl.in)
| robg wrote:
| As Wilson and I are working on this project, we'd love to hear
| your thoughts!
| ipnon wrote:
| The visualization of the space is excellent.
| thomasfromcdnjs wrote:
| It's very fun to play around with, there _seems_ to be value
| somewhere.
|
| I wonder if you could make a video of how an individual could
| use it effectively in a pragmatic sense e.g. networking,
| research
|
| (after more playing around I get the UX, very powerful indeed)
|
| (feedback: add some controls for filtering (dates etc),
| relevancy, color islands slightly, just general UX improvements
| and I would probably pay a few dollars a month for this)
|
| (feedback: when looking at a user, you have to zoom really far
| in before results start appearing. it should probably show
| labels/results if less than X amount of data points are in your
| viewport.)
|
| (feedback: Is there "Explore more users" section the more
| related users to the current profile? It would seem so, but not
| immediately clear. And if it is, none of my related users have
| me as a related user in their profile.
|
| ===
|
| This is really inspiring work, hats off!
| robg wrote:
| Thanks so much and for the great feedback!
| disqard wrote:
| Thanks for making and sharing!
|
| Could you share some info about how you generated the 2D space
| embedding/visualization?
| robg wrote:
| [delayed]
| JohnFen wrote:
| OK, I'm going to just be upfront here and admit that I'm an
| idiot. This is the third time I've seen this link come up on
| HN, and each time I've checked it out, but I can't really
| figure out how to interpret what I'm seeing there in a way that
| is useful.
|
| How can that map be used to determine who is knowledgeable
| about what? Looking up myself, I can't connect what I see with
| my own areas of knowledge.
|
| I think I need an ELI5 for this. Again, this isn't a criticism
| of the effort at all, it's a public admission of my own
| ignorance.
| djoldman wrote:
| Take everything a user says and encode it into some vector.
| That's "embedding."
|
| Then you encode a search query the same way and return the
| nearest N vectors/users. That's vector search.
|
| It seems they did the above for HN comments.
| JohnFen wrote:
| Right, I understand all that. I just don't understand how
| to interpret the resulting map (or how to formulate a
| search) to provide useful information, specifically "who
| knows what".
| robg wrote:
| Thank you for being so clear on where we are failing you.
| We're basically computing _your_ semantic space as
| compared to the whole community. The map shows the
| relevant instances in that space. As you browse your
| concepts the space changes and we link to the relevant
| comments from the associated threads. Who knows what is
| prioritizing the user and the semantics / voice of that
| user amidst the noise of the whole community speaking at
| once.
|
| We'd love to hear your feedback as you play with it more
| and we improve all of the above.
| jmuguy wrote:
| Looking at my own profile the words (topics i know about?) seem
| sort of generic. "issue", "company" - not sure what to make of
| that other than I assume those are words I use a lot in
| comments?
| robg wrote:
| We'll take a look at how to do better, thanks!
| toast0 wrote:
| I feel targeted when you call me a "system server user network
| number service code issue" ... not that it's inaccurate ;)
| robg wrote:
| Ha, exactly!
| wcedmisten wrote:
| Wow I really like the topographic visualization of the data,
| looks really slick!
| robg wrote:
| Thanks so much!
| causal wrote:
| This is really cool. Also a healthy reminder that anything we
| post publicly is likely to be analyzed. With a little analysis
| it's probably easy to know me better than I know myself.
| robg wrote:
| Maybe also who you'd tend to get along with and your unique
| place in the community?
| junon wrote:
| Would be interested in a study, unfeasible as it may be, on the
| social compatibility of people who have the strongest overlap
| between themselves and the next closest user.
|
| Cool visualization and analysis, really well made!
| robg wrote:
| Thanks so much! How do you see that study playing out in
| metrics that we can measure and examine?
| mattdesl wrote:
| For the "terrain contours" is there something specific you've
| done to make it feel more cartographic? Or is it basically just
| marching cubes / iso lines on some data points?
|
| Looks fantastic. Very cool project.
| robg wrote:
| [delayed]
| batch12 wrote:
| When I did this, I found it to also be an interesting way to
| fingerprint users and find alternate accounts. I was able to
| match an old account I used in a top 10 similarity match out of
| all users.
| rob wrote:
| https://hn2.wilsonl.in/user/rob
|
| Seems pretty accurate. I do love me some PHP and WordPress.
| robg wrote:
| [delayed]
| hu3 wrote:
| Searched for "dupe" and yes, it correctly pointed out the one
| user that takes it a hobby:
|
| https://hn2.wilsonl.in/search/dupe
| 0xbadcafebee wrote:
| HN and other forums perpetuate a fallacy (that I assume isn't
| well known because nobody has published a research paper on
| it)... but "the wisdom of the crowd" isn't.
|
| "Trusted voices" on HN are often not experts, yet espouse views
| which are not what the larger body of experts would consider
| correct. In addition, actual expert voices are drowned out by
| whatever the popular position is. HN also provides its own
| cultural bias, in that everything spoken on HN has to follow a
| rigorous cultural sieve set down by the guidelines, such that a
| "negative" view, even if correct, is considered either wrong or
| distasteful and buried. This is exacerbated by banning the use of
| humor to disarm controversial or heated comments. And this is the
| comments that HN does get; many opinions are never entered as
| comments here, so there exists a large knowledge gap. Then
| there's the "taboo" subjects like race, gender, religion,
| politics, social justice, etc which get buried for fear of
| controversy, so you're definitely not gonna find any expert
| opinions on those, as the stories just aren't there for
| discussion.
|
| The end result is that often experts go unheeded or even
| downvoted, popular shallow opinions get upvoted, and substantive
| commentary based on evidence and experience is frequently
| missing. The fact is that we have no idea who knows what, or
| what's true or right. We just have "popularity" according to the
| particular cultural quirks of this site. So you can definitely
| find out "who _thinks_ what ", and who is considered to be more
| trustworthy to a HNer, but it has absolutely nothing to do with
| objective truth or the body of real knowledge that exists outside
| the world of HN comments. This is an echo chamber, but it's not a
| chamber of experts. It just seems that way because occasionally
| you see a minor tech celebrity, and people talk with absolute
| authority regardless of if they have any.
|
| You want to find out who knows what? Look at their diplomas and
| careers. If they've done 20 years in a single field, probably
| they're an expert. If they have a degree (or multiple) in a
| field, probably they're an expert. If they spent half their life
| working on a single hobby, they're at least very knowledgeable in
| that field. But you can't determine that just by looking at who's
| talking about what or how many completely subjective "points"
| they get for what they say. Determining real knowledge requires
| analysis of specific criteria, filtered to get a higher quality
| result.
| saagarjha wrote:
| Humor is not banned on Hacker News. Bad attempts at it are
| penalized, though.
| robg wrote:
| IMHO if there's any take away from the last 20 years of the
| internet, it's how hard it is to have decent communities of
| people online, curated like a blooming garden of diversity.
| What are some online communities that you appreciate?
|
| I have a Ph.D. in neuroscience with exactly 20 years of
| experience. I talk about what I know and try to be curious
| about what I don't know. HN really helps me on both counts. How
| does it help you?
| wonger_ wrote:
| Personally, I like how HN focuses on content and discussions
| rather than individual users. If I wanted to follow experts, I'd
| probably curate a selection on a social network like Mastodon, or
| kludge together some RSS feeds.
|
| Also, I feel like this tool selects for active commenters, not
| for knowledgeable experts. Not to mention throwaway accounts.
|
| Still a cool project.
| robg wrote:
| Thanks! Those are fair points. We're thinking we could uplevel
| the social layer so you can connect with people of similar
| interests for deeper connections. In this way we compute not
| just your contributions but how they relate to others.
| photochemsyn wrote:
| > "Despite the intervening 16 years, we're amazed that social
| networks, even Hacker News, don't compute and display the trusted
| voices across topics. Instead of prioritizing pages based on
| content, social networks could prioritize the people behind the
| content."
|
| Fallacy: appeal to authority. Practically, just because someone
| generates great content on subject A doesn't mean their take on
| subject B is any better than random. A well-reasoned self-
| consistent argument informed by accurate data is far more valuble
| than 'trust this expert opinion because this expert can be
| trusted' approaches - although it may require more work on the
| part of the reader. Don't get lazy.
| robg wrote:
| We're trying to stay away from authority and just compute based
| on what you say then transparently show where that information
| came from. How do you think we can do better?
| simonw wrote:
| https://hn2.wilsonl.in/user/simonw includes "Risk of COVID from
| pianos" down at the bottom of the map. I'd love to know where
| that came from!
| wonger_ wrote:
| Appears to be a summary of this comment:
| https://news.ycombinator.com/item?id=30105274
|
| If you zoom and pan the red cursor over that map location/text,
| your source comment will appear underneath the map.
| robg wrote:
| Great call out! Would it help to make the points more
| interactive?
| DoreenMichele wrote:
| Just curious what the mechanism is for adding a bio to individual
| users. Your sample -- robg -- has the same bio on your stuff as
| on HN but others have stuff on HN and "No bio provided" on your
| thing.
| robg wrote:
| Right now if you add it to your HN profile we'll be able to
| include it.
| zamalek wrote:
| Some sentiment analysis might help? Apple is one of my keywords,
| but you'd pretty-much only get an "avoid" response from me (to a
| degree, I have recommended their phones to a _very_ specific
| demographic).
|
| Awesome project, fantastic UI.
| hu3 wrote:
| Same. I think a sentiment analysis could be helpfull.
|
| I find myself dragged to Apple discusions more than I would
| like to admit, but they are generally negative in the last
| years.
| robg wrote:
| It's a topic we keep coming back to, seems like we can do
| better than sentiment analysis to show the semantic space of
| the sentiment(s) and how those breakdown by users and groups.
| For instance assuming a large discussion set of apples,
| different types of apples, and other types of fruits, we'd
| discover the patterns among users. Have you seen folks doing
| that type of semantic sentiment analytics?
| kaycebasques wrote:
| Super cool! Happy to see that I show up under documentation-
| related stuff, considering how frequently I mention that I'm a
| technical writer in my comments. Also pretty excited to connect
| with the other people that show up related to docs / technical
| writing / etc.
| nadermx wrote:
| As opposed to me not showing up at all for things I comment on.
| Guess I'm not knowledgeable enough.
| layer8 wrote:
| The results are biased towards users with high post count. It
| doesn't matter how knowledgeable you are about a topic, only
| how much you comment on it.
| kaycebasques wrote:
| And using the correct keywords. SEO is dead, long live SEO!
|
| Also I imagine that my area is pretty niche (not many
| people talking about documentation on HN) so I may have
| less competition
| robg wrote:
| We'll take a look!
| pcthrowaway wrote:
| How does determine which comments are valuable ("trusted")
| without access to the vote count of comments?
| robg wrote:
| We're trying to stay away from karma and focus on what people
| are saying within the community, rank orders, overall
| semantics, and more. How do you think we can do better?
| dang wrote:
| Recent and related:
|
| _Show HN: Exploring HN by mapping and analyzing 40M posts and
| comments for fun_ - https://news.ycombinator.com/item?id=40307519
| - May 2024 (159 comments)
| robg wrote:
| Thanks dang! Amazing to me how much HN brought Wilson and I
| together after 16 years and across the world. We're hopeful
| about the future of trust on the internet exactly because of
| communities like HN.
| neilv wrote:
| > _Mapping Hacker News to find who knows what in the HN
| community_
|
| Would things like this create/increase incentives to game the
| metrics?
| gradyfps wrote:
| In my limited experience, people tend to gamify any numeric
| metric readily apparent to them.
|
| Anecdotally, total game time (in hours, usually) is used to
| convey experience in video games (WoW, CS:GO, PUBG). I've seen
| people create & run 3rd-party software to artificially inflate
| these types of metrics.
| neilv wrote:
| And some metrics aren't just a "game", but are gamed for very
| real benefit (e.g., gaining admission to a college, getting a
| promotion, being more employable) in ways that are
| _counterproductive_ to pretty much all other goals.
|
| So, when something like HN has goals, and you create, say,
| something akin to a contest and link it to real-world
| benefits (e.g., people expect a recruiter or hiring to use
| that "who knows what" info), then many people will modify
| their behavior, probably to the detriment of original goals.
| robg wrote:
| We're trying to stay away from karma for that reason and
| focus on what people are saying within the community to
| find trusted voices. How can we do better?
| neilv wrote:
| How can Web sites do better at producing valuable and
| trustworthy content, when what everyone cares about now
| comes down to SEO?
| asp_hornet wrote:
| I hate this. If i used my real name id have some pleb website on
| the internet making chump guesses at what i know based on what i
| was prepared to share. Thanks for reminding me to use anonymous
| burner accounts on everything app i use.
| robg wrote:
| Thanks for the feedback. How can we better support authenticity
| and trusted voices, where you fit in, and others like you?
| motohagiography wrote:
| Good God I am a bore. May track some people down in my niche
| interest areas though, thank you for this.
| SCUSKU wrote:
| Oi! Watch it! Our distance in embedding space is small. Then
| again, maybe I am a bore...
| mmastrac wrote:
| I found it a bit challenging to actually drill down into my own
| username, but it doesn't seem to offer much other than throwing a
| lot of dots all over the map. I'm trying to understand what the
| overall clusters might be, but most of them are just
| android/apple/google?
|
| https://hn2.wilsonl.in/user/mmastrac
| qdot76367 wrote:
| Searched for "Buttplugs".
|
| My name pops up.
|
| Hell yeah. A++++ completely accurate would use again.
| robg wrote:
| [delayed]
| gnicholas wrote:
| Interesting. I have noticed that my most-upvoted comments relate
| to legal questions, which I have relatively more expertise than
| most HNers (I used to be a lawyer). Although I'm not among the
| top commenters for law/legal/lawyer according to this tool, I
| definitely recognize some of the top names and can recall seeing
| their comments in legal threads. Pretty cool tool!
| dmurray wrote:
| Things I thought I had expertise in and have posted about
| disproportionately on HN: chess, Python, HFT, Fermi estimation,
| Ireland.
|
| Keywords the site actually associates with me: language, English,
| article, team, book. To be fair, at some zoom levels I do get
| "chess".
| rpmisms wrote:
| I wonder how high I am on the guns knowledge list?
___________________________________________________________________
(page generated 2024-07-25 23:05 UTC)