[HN Gopher] The Magic of Embeddings
       ___________________________________________________________________
        
       The Magic of Embeddings
        
       Author : gk1
       Score  : 93 points
       Date   : 2023-06-24 01:26 UTC (21 hours ago)
        
 (HTM) web link (stack.convex.dev)
 (TXT) w3m dump (stack.convex.dev)
        
       | danjc wrote:
       | There are often comments on HN about how Google search doesn't
       | work as well as it used to.
       | 
       | It's worth noting that this is probably because this audience
       | wants keyword matching rather than semantic search which Google
       | switched to years back [1].
       | 
       | Curious to know whether embeddings have been a subsequent step in
       | this transition.
       | 
       | 1. https://blog.google/products/search/search-language-
       | understa...
        
         | spion wrote:
         | Google is pretty bad at semantic understanding too. Try
         | searching for "implementing stackoverflow in typescript". You
         | will get "implementing in typescript" results that come from
         | "stackoverflow". (Then for a good measure try the same in
         | ChatGPT)
        
         | marginalia_nu wrote:
         | My hunch is the problem is they are essentially doing both in
         | the same input field.
         | 
         | One of the hardest design problems with search is that the
         | affordances of a search field is largely a mystery. Mixing
         | paradigms does not help.
        
         | esafak wrote:
         | Embeddings are the linchpin of semantic search. The embeddings
         | are designed to encode semantic similarity, so the response is
         | formed by retrieving the nearest neighbors of the query, and
         | there are data structures that allow this to be done
         | efficiently.
        
       | Solvency wrote:
       | Is there ever a use case for wanting to compare NON-normalized
       | vectors using dot products or cosine similarity? It seems like
       | all use cases always involve first normalizing the values, which
       | makes sense, but wondering when/why you wouldn't want to do that.
        
         | esafak wrote:
         | The case in which semantic similarity is encoded by Euclidean
         | distance, and you want to perform similarity search (retrieve
         | the nearest neighbors).
        
           | Solvency wrote:
           | But in a model like the article examples with 1500+
           | dimensions doesnt Euclidean distance suffer the "curse of
           | dimensionality", making it unusable for similarity searches?
        
         | QuadmasterXLII wrote:
         | Non-normalized vectors are for when some vectors should match
         | less often in general, encoded by length. In particular,
         | vectors that shouldn't match anything can be encoded as a zero
         | vector. For example, the last layer of a classification convnet
         | is just dot product similarity, and is usually not normalized
        
         | jstx1 wrote:
         | Normalized dot product is equivalent to cosine similarity.
        
       | mitko wrote:
       | Any folks from Convex, do you have plans on adding vector indexes
       | to Convex databases?
        
         | james_cowling wrote:
         | Convex cofounder here. Initial focus has been low-latency OLTP
         | database workloads, which is pretty tied to the user-facing
         | end-to-end reactivity angle we're pushing. As a bonus feature
         | we also have built-in strongly consistent full text search, but
         | not vector search.
         | 
         | So far our answer for folks who want alternative storage/query
         | engines is to use our streaming Airbyte source connector or
         | write directly to Pinecone, Snowflake, etc. This should work
         | great for most devs.
         | 
         | There are likely always going to be some developers who want to
         | use a particular third party database in addition to Convex,
         | but we plan to expand built-in support for most workloads over
         | time so that Convex is a truly full-stack backend replacement.
        
         | jamwt wrote:
         | Hi there. Another Convex cofounder here.
         | 
         | SiteGuide (https://siteguide.ai/) was the first to do vector
         | embeddings with Convex, built by integrating Convex + Pinecone.
         | This combination has been an increasingly common pattern over
         | the last few months. So we put a template project to
         | demonstrate how this is usually done:
         | 
         | https://github.com/ianmacartney/embeddings-in-convex
         | 
         | We're strongly considering building in vector search a little
         | further down the road, but this is the recommended approach for
         | now.
        
       | dr_dshiv wrote:
       | Great article! Fun, clear, straightforward, good examples,
       | practical.
        
         | justanotheratom wrote:
         | These type of articles are dime a dozen on Twitter, not clear
         | why this one is on HN front page.
        
           | dr_dshiv wrote:
           | Sure, post some here. Would love that.
        
       ___________________________________________________________________
       (page generated 2023-06-24 23:01 UTC)