[HN Gopher] Embedding Archives: Millions of Wikipedia Article Em...
       ___________________________________________________________________
        
       Embedding Archives: Millions of Wikipedia Article Embeddings in
       Many Languages
        
       Author : zerojames
       Score  : 68 points
       Date   : 2023-04-20 18:15 UTC (4 hours ago)
        
 (HTM) web link (txt.cohere.com)
 (TXT) w3m dump (txt.cohere.com)
        
       | scrum-treats wrote:
       | [dead]
        
       | [deleted]
        
       | jerrygenser wrote:
       | Seems like it would lock in a user to cohere proprietary model in
       | order to embed new search terms to the same space?
       | 
       | Would love to see similar but embedded with a more open
       | representation model or even sent2vec
        
         | jayalammar wrote:
         | There's a lot you can do with the vectors themselves without
         | needing to embed any more text (e.g., clustering, exploration,
         | visualization after dimensionality reduction...etc). Here's a
         | previous embeddings exploration of top HN posts:
         | https://txt.cohere.com/combing-for-insight-in-10-000-hacker-...
         | A lot of that code can be used here as well.
         | 
         | If you want to query for a search term, you can use a trial API
         | key which is free to use for prototyping. The embedding model
         | itself is not open source, though. [co-author of the post here]
        
           | minimaxir wrote:
           | If that's the intent, IMO the release dataset should have
           | more metadata (e.g. paragraph heading, article taxonomy)
        
             | jayalammar wrote:
             | How would you add that data? As new columns you mean? Or
             | add the paragraph headings to the text of the paragraphs
             | before embedding them?
        
               | minimaxir wrote:
               | New columns.
               | 
               | For the headings, I mean the Wikipedia section headings
               | (which isn't always a paragraphs, my mistake).
               | 
               | In both cases the data can be used like to
               | classify/visualize Show HNs in your linked post.
        
         | minimaxir wrote:
         | Unfortunately yes, as the corresponding model is not also open-
         | source.
        
       | gaogao wrote:
       | In a few of the examples given, WikiData would also be searchable
       | enough to find the relevant articles to pull in. I've actually
       | been really curious if you could use recent advancements in ML to
       | augment the filling out of WikiData, since it's a bit sparse as
       | is.
        
       | kristianp wrote:
       | How recent is the Wikipedia snapshot used?
       | 
       | Edit, going to [1] the datasets are labelled 2022-12.
       | 
       | [1] https://huggingface.co/Cohere
        
       | txtai wrote:
       | If you want something similar to this that is fully downloadable
       | and open source, check this out -
       | https://huggingface.co/NeuML/txtai-wikipedia
       | 
       | It's an embeddings database of Wikipedia abstracts with page view
       | data integrated to enable filtering pages based on popularity in
       | addition to similarity.
        
       ___________________________________________________________________
       (page generated 2023-04-20 23:02 UTC)