[HN Gopher] Turbopuffer: Fast search on object storage
___________________________________________________________________
Turbopuffer: Fast search on object storage
Author : Sirupsen
Score : 119 points
Date : 2024-07-09 14:48 UTC (8 hours ago)
(HTM) web link (turbopuffer.com)
(TXT) w3m dump (turbopuffer.com)
| softwaredoug wrote:
| Having worked with Simon he knows his sh*t. We talked a lot about
| what the ideal search stack would look when we worked together at
| Shopify on search (him more infra, me more ML+relevance). I
| discussed how I just want a thing in the cloud to provide my
| retrieval arms, let me express ranking in a fluent "py-data"
| first way, and get out of my way
|
| My ideal is that turbopuffer ultimately is like a Polars
| dataframe where all my ranking is expressed in my search API. I
| could just lazily express some lexical or embedding similarity,
| boost with various attributes like, maybe by recency, popularity,
| etc to get a first pass (again all just with dataframe math).
| Then compute features for a reranking model I run on my side -
| dataframe math - and it "just works" - runs all this as some kind
| of query execution DAG - and stays out of my way.
| cmcollier wrote:
| Unrelated to the core topic, I really enjoy the aesthetic of
| their website. Another similar one is from Fixie.ai (also,
| interestingly, one of their customers).
| itunpredictable wrote:
| This website rocks
| nsguy wrote:
| Yeah! fast, clean, cool, unique.
| bigbones wrote:
| Sounds like a source-unavailable version of Quickwit?
| https://quickwit.io/
| pushrax wrote:
| LSM tree storage engine vs time series storage engine, similar
| philosophy but different use cases
| vidar wrote:
| Can you compare to S3 Athena (ELI5)?
| CyberDildonics wrote:
| Sounds like a filesystem with attributes in a database.
| drodgers wrote:
| I love the object-storage-first approach; it seems like such a
| natural fit for the could.
| eknkc wrote:
| Is there a good general purpose solution where I can store a
| large read only database in s3 or something and do lookups
| directly on it?
|
| Duckdb can open parquet files over http and query them but I
| found it to trigger a lot of small requests reading bunch of
| places from the files. I mean a lot.
|
| I mostly need key / value lookups and could potentially store
| each key in a seperate object in s3 but for a couple hundred
| million objects.. It would be a lot more managable to have a
| single file and maybe a cacheable index.
| jiggawatts wrote:
| > trigger a lot of small requests reading bunch of places from
| the files. I mean a lot.
|
| That's... the whole point. That's how Parquet files are
| supposed to be used. They're an improvement over CSV or JSON
| because clients can read small subsets of them efficiently!
|
| For comparison, I've tried a few other client products that
| don't use Parquet files properly and just read the whole file
| every time, no matter how trivial the query is.
| imiric wrote:
| ClickHouse can also read from S3. I'm not sure how it compares
| to DuckDB re efficiency, but it worked fine for my simple use
| case.
| mjlxyz wrote:
| this is good
| bean_salad_123 wrote:
| thank
| omneity wrote:
| > In 2022, production-grade vector databases were relying on in-
| memory storage
|
| This is irking me. pg_vector has existed from before that,
| doesn't require in-memory storage and can definitely handle
| vector search for 100m+ documents in a decently performant
| manner. Did they have a particular requirement somewhere?
| jbellis wrote:
| Have you tried it? pgvector performance falls off a cliff once
| you can't cache in ram. Vector search isn't like "normal"
| workloads that follow a nice pareto distribution.
___________________________________________________________________
(page generated 2024-07-09 23:00 UTC)