[HN Gopher] Show HN: My recommendation engine for Hacker News
___________________________________________________________________
Show HN: My recommendation engine for Hacker News
Hi! I'm Julien and I built a recommendation engine for Hacker News.
I feel like this website is a gold mine. Every day, I find some
very interesting stories about a topic. And sometimes, I want to
find other stories covering that same topic but I can't. Hacker
News has years of history of awesome discussion and ressources.
Unfortunately, I think HN Algolia isn't helpful in searching these
old threads. As a student, I want to learn a lot from this website.
This is why I created HN Recommend. Input a sentence or the URL of
an article, and get the most popular and similar posts from Hacker
News. About the technical details, I've computed the embeddings of
over 100,000 articles from HN and indexed it using Faiss. I made a
blog post for a deeper explanation. Source code:
https://github.com/julien040/hn-recommendation-api Article:
https://julienc.me/articles/Extract_embeddings_Hacker_News_a...
Project: https://hn-recommend.julienc.me
Author : julien040
Score : 155 points
Date : 2023-06-19 14:45 UTC (8 hours ago)
(HTM) web link (hn-recommend.julienc.me)
(TXT) w3m dump (hn-recommend.julienc.me)
| julien040 wrote:
| I just released a new update, thanks to everyone's feedback. Now,
| you can sort results by relevancy, age, or score using the
| select.
| lfkdev wrote:
| A time filter is needed
| akomtu wrote:
| HN needs a simple feature: a weekly digest view that shows the
| top 30 most commented posts (it should completely ignore flags
| and votes).
| ColinWright wrote:
| You mean like the one that's emailed to me every week?
|
| https://hackernewsletter.com
| dxbydt wrote:
| Pls sort by recency. Otherwise you see 13 year old articles most
| of them obsolete/irrelevant to the current situation.
| julien040 wrote:
| By sorting by recency, I was worried I would get less revelant
| results. Perhaps I should add a thresold to not have too old
| posts
| julien040 wrote:
| You can now sort by recency. I hope this helps.
| dxbydt wrote:
| Very fast turnaround. Kudos! Works very nicely now.
|
| Try this: https://hn-recommend.julienc.me/?q=sf%20crime
| With the Newest filter vs the Oldest filter. ( btw the
| default Relevance filter gives only tangentially relevant
| results for this query. Whereas the Newest & Oldest are on
| point. )
| rounakdatta wrote:
| Nit:
|
| > Resources to learn about distributed systems
|
| I thought Murat Buffalo's blog would come up at the top. That's a
| gold, and I'm confident that it was shared on HN as well (maybe a
| year or two back).
|
| Otherwise neat and useful!
| sogen wrote:
| A comment about search results: "design system" is related to
| design, "system design" relates to computing
|
| It seems search takes the two inputs as the same.
|
| Also, search doesn't seem to work when using just 1 word.
| julien040 wrote:
| Yes it's an issue. Sadly, I can't fix it. I'm using the closed
| source "text-embedding-ada-002" model from OpenAI.
|
| As I can see, the longer the input, the more accurate the
| results. Perhaps you can try something longer, like "What is a
| design system for UI?"
| dpe82 wrote:
| This is great. I often come across some HN post on a topic I am
| interested in and then want to go look at other posts in the same
| topic cluster to expand my exposure. This looks awesome for that.
|
| I don't know if it would be useful or even work, but is it
| possible to let the user adjust the vector distance threshold and
| then apply the other sorting parameters to the results? Eg. if I
| want to go broader, but then sort by high score or something so I
| see popular posts within an expanded (but still relevant)
| cluster?
| julien040 wrote:
| I thought about an algorithm with weight adjustable by the
| user. Now, the API returns a field with the distance between
| the post and the query (the square of the Euclidean distance).
| It's used by the interface to rank results by relevance.
|
| Perhaps I can compute a score for each story, where each field
| has a weight and rank the results using this score. For
| example, the score could be 0.2 x score + 0.1 x comments +
| 1/distance - timestamp/ 10^9. The stories with the highest rank
| would be shown first, and the weight (0.2, 0.1, 10^9) could be
| adjusted by the user, as some might prefer recency while others
| prefer popularity.
| lettergram wrote:
| Checkout https://askhn.ai
|
| The content is ranked by how people discuss the topics and who
| discusses them
|
| If you just do embeddings on posts you might miss relevant
| content. When people who have knowledge of AMD discuss intel
| and believe that content is relevant to AMD, the content will
| be ranked
| swyx wrote:
| i like the idea of this but wont remember it because my muscle
| memory is tuned to news.ycombinator.com. perhaps i can recommend
| a chrome extension instead of a website?
| julien040 wrote:
| Thank you for suggesting this.
|
| The API is already made and can be found at
| https://github.com/julien040/hn-recommendation-api. I don't
| think it would be too difficult to build a Chrome extension
| that fetches it.
| TechBro8615 wrote:
| An iOS share widget would be cool too. Since you support
| putting the input text in the URL, then maybe someone can
| make a Workflow for it and share it here.
| julien040 wrote:
| Are iOS share widgets using Apple Shortcuts? I wish to
| learn more about this technology, so it would be a pleasure
| to try building it.
| TechBro8615 wrote:
| Yeah, my mistake, the app is called "Shortcuts" now. I
| get confused because it was an app called Workflow that
| was acquired by Apple [0].
|
| You can use the app itself to make some surprisingly
| powerful shortcuts, and then share them in some kind of
| text based serialized form (don't remember the details).
| I'm sure there are also ways to make them
| programmatically, but I doubt it would be necessary for
| this use case.
|
| Seems like you basically want to extract the URL of the
| HN page, store it in a variable and then append that
| variable to the URL of your recommendation engine. There
| are probably more fancy variants of "extract text" that
| you could use, too - I'm not sure of the details.
|
| [0] https://en.wikipedia.org/wiki/Shortcuts_(app)
| julien040 wrote:
| Here is the v1 of the shortcut: https://www.icloud.com/sh
| ortcuts/a7e9d236b35342c5aed1d022801...
|
| For now, it only pushes the shared URL to the
| recommendation engine. If I have more time, I'll try to
| find a way to extract the URL from the HN page.
| 4hEn wrote:
| I search for a url I know was posted and it doesn't show it. It
| shows unrelated articles.
| julien040 wrote:
| The data is a few weeks old. Do you know when the URL was
| published?
| 4hEn wrote:
| It's 10 years old.
|
| This search query https://hn-
| recommend.julienc.me/?q=paul%20graham returns articles that
| are missing both words of the query
| julien040 wrote:
| The website features only stories with a score greater than
| 100 but I don't think that is the problem.
|
| Unlike HN Algolia, it doesn't match words; it uses
| embeddings so stories are matched by their similar meaning
| rather than similar words. To find it, you might try to be
| more specific, such as "Paul Graham Y Combinator <facts of
| the article>". I'm sorry HN Recommend doesn't match your
| use case
| moritzwarhier wrote:
| This is a joy to use and also ot fits very nicely with the other
| highly ranked post by Nielsen group. Kudos!
| fewald_net wrote:
| Great project. I learned about the faiss library. Out of
| curiousity, did you also try it with doc2vec?
| julien040 wrote:
| I didn't try Doc2Vec. I wanted a hosted solution because I
| wouldn't have been able to compute all this locally (more than
| 100,000 posts).
|
| If you tried it, did you have great results with? I may use it
| in future projects.
| passion__desire wrote:
| One feature I would like for an Recommender Systems to have is :
| explicit ability to jump in and out of filter bubbles or research
| rabbit holes. Another example would be, put yourself in the shoes
| of another, e.g. what content is liked by game developers
| generally. apart from general gamedev content, what do they like,
| where do they take inspiration from, etc.
|
| I remember there was a project built on instagram which allowed a
| person to view instagram as it looked like to a particular
| celebrity.
| julien040 wrote:
| I'm a bit divided on this feature. On one hand, I would like to
| have this feature; it would be awesome to see the
| recommendation of people from different jobs. On the other
| hand, I'm a bit concerned about privacy. The system must ensure
| that each group is big enough to avoid the leak of someone's
| recommendations. I don't want anyone to know exactly what I'm
| liking and what I'm watching.
|
| If I recall correctly, myCANAL (the French Netflix) used to
| have a similar feature. You could access the recommendations of
| personalities of the channel, but it was curated manually.
| benzible wrote:
| Hmm I tried searching "elixir" and found nothing related to the
| language. HN Algolia gives me exactly what I want. On what basis
| do you say it's "not helpful"?
| julien040 wrote:
| Yes the search doesn't work very well for one word. Try to
| input an url about elixir like this: https://hn-
| recommend.julienc.me/?q=https%3A%2F%2Fnews.ycombi...
|
| I may have used the incorrect term. HN Algolia is effective for
| searching for a particular story. However, I am unable to
| utilize it to find related posts on the same topic that do not
| contain the same words.
| danvayn wrote:
| hey Julien. I love the product but the search doesn't seem to be
| doing the best for me. For example, I looked up Tailwind and got
| plenty of results but none of them actually involved Tailwind.
|
| Maybe a tagging solution is the way? if you determine a set
| amount of popular keywords for a topic and filter around those,
| you can offer more relevant results. With some sort of public
| tagging system you can also have SEO friendly pages around tags
| and get people browsing stuff they wouldn't normally search for.
| julien040 wrote:
| At first, the website concept focused on getting posts similar
| to a URL. Querying with text didn't yield relevant results.
|
| Your solution appears better suited for this use case. Thank
| you.
___________________________________________________________________
(page generated 2023-06-19 23:00 UTC)