[HN Gopher] Indexing 100M vectors in 20 minutes on PostgreSQL wi...
___________________________________________________________________
Indexing 100M vectors in 20 minutes on PostgreSQL with 12GB RAM
Author : gaocegege
Score : 74 points
Date : 2025-12-08 17:30 UTC (5 days ago)
(HTM) web link (blog.vectorchord.ai)
(TXT) w3m dump (blog.vectorchord.ai)
| duckbot3000 wrote:
| Kinda makes you wonder why you need cloud for anything besides
| remote encrypted backups if you can run all that on 12GB
| setr wrote:
| Because getting any hardware out of infra-team on premise is
| utterly miserable, across the board.
| lelanthran wrote:
| That's not the only alternative.
|
| Rent your VPS and add in extra volumes for like $10 per
| 100GB.
| Imustaskforhelp wrote:
| Funny thing but netcup has $10 per 1 TB
|
| Netcup is under-rated but there are also other providers
| too at lowendbox/lowendtalk and I am interested to try out
| hetzner too sometime.
| benjiro wrote:
| And if you want to go even cheaper, check out Hetzner
| their EX63 (go to custom) > 4x 7.68TB drives for like 140
| Euro.
|
| Not counting the fact that Netcup is raided (also Netcup
| is limited to 8TB on a VPS).
|
| That is like 4.7 Euro /TB. That is like 4$/TB. 6 Euro /
| TB in a raid 5 setup.
|
| I do not understand why they are not using this new
| pricing model on their older servers. There the best you
| can get is like 10 Euro /TB (for the single 15TB U.2).
| riku_iki wrote:
| what about failover story if server dies? PG failover setup is
| complicated, and cloud infra handles this for you.
| positron26 wrote:
| Do we mean managed or PG on K8s like CNPG? In all cases, I
| use the infra to simplify things like having disk redundancy
| and failover nodes, not because 12GB is interesting.
| riku_iki wrote:
| Primary managed PG, since you still need
| setup/maintenance/monitoring on your K8S own solution.
| logifail wrote:
| (Genuine question) What's your current plan for when your
| cloud provider goes offline? Do you have a failover story, or
| it a case of "wait for them to come back online"?
| riku_iki wrote:
| I have backups on different cloud provider, so I could
| bootstrap db if provider goes dark indefinitely.
|
| But realistically, I believe major clouds (google, aws)
| likely has more robust org and infra for recovery than I
| can built and maintain.
| tjwebbnorfolk wrote:
| What are you willing to pay for cloud-native failover?
|
| Not every use case requires 100% uptime
| riku_iki wrote:
| Sure, but those who require (99% of major businesses) are
| ready to pay.
| benjiro wrote:
| https://github.com/multigres/multigres ... when its complete.
| From the guy that made Vitess for Mysql.
|
| And yes, i agree, the PG failover setup (and especially
| dealing with a failure afterwards, to restore the ex-master
| is beyond infuriating).
|
| But its not pay 10x the amount, while eating easily 10x
| performance infuriating :)
| nwellinghoff wrote:
| Too bad aws does not support any of these other vector extensions
| in managed rds.
| ayende wrote:
| That suffer from a serious issue
|
| You must have the data upfront, you cannot build this in an
| incremental fashion
|
| There is also bo mention on how this would handle updates, and
| from the description, even if updates are possible, this will
| degrade over time, requiring new indexing batch
| esafak wrote:
| How does it compare with paradedb and lancedb?
___________________________________________________________________
(page generated 2025-12-13 23:01 UTC)