[HN Gopher] Healthchecks.io now uses self-hosted object storage
___________________________________________________________________
Healthchecks.io now uses self-hosted object storage
Author : zdw
Score : 185 points
Date : 2026-04-17 14:29 UTC (1 days ago)
(HTM) web link (blog.healthchecks.io)
(TXT) w3m dump (blog.healthchecks.io)
| _joel wrote:
| I'm sure it's a lot better now but everytime I see btrfs I get
| PTSD.
| poly2it wrote:
| Care to elaborate? I've heard good things about it, but am
| personally a ZFS user.
| metadat wrote:
| Years of serious corruption bugs.
| dundercoder wrote:
| Gluster was that for me
| sigio wrote:
| Yup, still get nightmares about glusterfs.... still have
| one customer running on it.
| dundercoder wrote:
| I heard it got better, but we ran into the BOTF (billions
| of tiny files) issue around 2016. (For a genealogy
| startup this was a serious issue)
| _joel wrote:
| Ah, another one! Yep, also same, before ceph days at
| least (although I've had my own, albeit self-inflicted,
| nightmare there too).
| uroni wrote:
| I'd worry about file create, write, then fsync performance with
| btrfs, but not about reliability or data-loss.
|
| But a quick grep across versitygw tells me they don't use
| Sync()/fsync, so not a problem... Any data loss occurring from
| that is obviously not btrfs fault.
| 060880 wrote:
| Same here. Had a production node running btrfs under heavy
| write load (lots of small files, frequent creates) and spent
| two days debugging what turned out to be filesystem-level
| corruption. Switched to ext4 and never looked back. The article
| doesn't mention what filesystem sits under Versitygw here,
| which seems like a pretty relevant omission for anyone thinking
| of replicating the setup.
| __turbobrew__ wrote:
| I hit a panic in btrfs using an ubuntu 24 LTS kernel. The
| trauma is still well and alive.
| stackskipton wrote:
| Same, and reasoning around inodes feels easy fixed by just
| upping inode per KB from 16k to 4k which is likely block size
| anyways.
| NelsonMinar wrote:
| I'm a little surprised it's not ZFS. Too difficult to add to
| their Linux environment? That's still a problem here in 2026.
| tobilg wrote:
| I don't get it, if it's running on the same (mentioning "local")
| machine, why does it even need the S3 API? Could just be plain IO
| on the local drive(s)
| esafak wrote:
| So you don't need to refactor your code?
| ryanjshaw wrote:
| And when/if you decide to head back to a 3rd party it
| requires no refactoring again.
| tobilg wrote:
| yeah, sure, those 5-10 different API calls would surely be a
| huge toll to refactor... I'd rather run an additional service
| to reimplement the S3 API mapping to my local drive /s
| zipy124 wrote:
| seperate machine I think given the quoted point at the end:
|
| > The costs have increased: renting an additional dedicated
| server costs more than storing ~100GB at a managed object
| storage service. But the improved performance and reliability
| are worth it.
| VHRanger wrote:
| The S3 API doesn't work like normal filesystem APIs.
|
| Part of it is that it follows the object storage model, and
| part of it is just to lock people into AWS once they start
| working with it.
| jen20 wrote:
| > part of it is just to lock people into AWS once they start
| working with it.
|
| This is some next-level conspiracy theory stuff. What exactly
| would the alternative have been in 2006? S3 is one of the
| most commonly implemented object storage APIs around, so if
| the goal is lock-in, they're really bad at it.
| daveguy wrote:
| > What exactly would the alternative have been in 2006?
|
| Well, WebDAV (Document Authoring and Versioning) had been
| around for 8 years when AWS decided they needed a custom
| API. And what service provider _wasn 't_ trying to lock you
| into a service by providing a custom API (especially pre-
| GPT) when one existed already? Assuming they made the
| choice for a business benefit doesn't require anything
| close to a conspiracy theory.
|
| And it worked as a moat until other companies and open
| source projects started cloning the API. See also:
| Microsoft.
| PunchyHamster wrote:
| WebDAV is ass tho. I don't remember a single positive
| experience with anything using it.
|
| And still need redundant backend giving it as API
| QuercusMax wrote:
| When I was in school, we had a SkunkDAV setup that
| department secretaries were supposed to use to update
| websites... supporting that was no fun at all. I'm not
| sure why it was so painful (was 25 years ago) but it left
| a bad taste in my mouth.
| debugnik wrote:
| WebDAV is kinda bad, and back then it was a big deal that
| corporate proxies wouldn't forward custom HTTP methods.
| You could barely trust PUT to work, let alone PROPFIND.
| orf wrote:
| When S3 launched the core API could be described with 4
| requests. It was (and still mostly is) super simple.
|
| Saying they should have used WebDAV instead shows a lack
| of knowledge on your end rather than theirs.
| tobilg wrote:
| I'm 100% aware of how S3 works. I was questioning why the S3
| API is needed when the service is using local storage.
| zdw wrote:
| Sometimes API compatibility is an important detail.
|
| I've worked at a few places where single-node K8s
| "clusters" were frequently used just because they wanted
| the same API everywhere.
| throw1234567891 wrote:
| What kind of vendor lock-in do you even talk about. Their API
| is public knowledge, AWS publishes the spec, there are
| multiple open source reference client implementations
| available on GitHub, there are multiple alternatives
| supporting the protocol, you can find writings from AWS
| people as high in hierarchy as Werner Vogels about internals.
| Maybe you could say that some s3 features with no alternative
| implementation in alternative products are a lock-in. I would
| consider it a ,,competitive advantage". YMMV.
| _joel wrote:
| Apart from all these other products that implement s3? MinIO,
| Ceph (RGW), Garage, SeaweedFS, Zenko CloudServer, OpenIO,
| LakeFS, Versity, Storj, Riak CS, JuiceFS, Rustfs, s3proxy.
| 0x457 wrote:
| Riak CS been dead for over a decade which makes me question
| the rest. Some of these also do not have the same behaviors
| when it comes to paths (MinIO is one of those IIRC).
|
| Also, none of them implement full S3 API and features.
| Datagenerator wrote:
| What does RadosGW miss?
| skrtskrt wrote:
| There's a difference between S3 API spec and what Amazon
| does with S3 - for isntance, the new CAS capabilities
| with Amazon are not part of the spec.
|
| Ceph certainly implements the full API spec, though it
| may lag behind some changes. It's mostly a question of
| engineering time available to the projects to keep up
| with changes.
| 0x457 wrote:
| > There's a difference between S3 API spec and what
| Amazon does with S3 - for isntance, the new CAS
| capabilities with Amazon are not part of the spec.
|
| Sure, but those are S3 APIs and features that provided by
| AWS. We not talking about S3 spec, we're talking about s3
| product.
| ovaistariq wrote:
| Add Tigris to the list as well please.
|
| We maintain a page that shows our compatibility with S3
| API. It's at https://www.tigrisdata.com/docs/api/s3/. The
| test runner is open source at
| https://github.com/tigrisdata-community/s3-api-compat-tests
| ovaistariq wrote:
| The API has sort of become a standard. There are many
| providers providing S3 API-compatible storage.
| orev wrote:
| If the app was written using the S3 API, it would be much
| faster/cheaper to migrate to a local system the provides the
| same API. Switching to local IO would mean (probably) rewriting
| a lot of code.
| 0x457 wrote:
| Surely "read object" and "write object" are not hard to
| migrate to local file system. You can also use Apache OpenDAL
| which provide the same interface to both.
| QuercusMax wrote:
| Yeah, unless you have the raw S3 API throughout your
| codebase you should be able to write a couple dozen lines
| of code (maximum) to introduce a shim that's trivial to
| replace with local file access. In fact, I've done this in
| most projects that work with S3 or similar APIs so I can
| test them locally without needing real S3!
| cuu508 wrote:
| (Author here) There are multiple web servers for redundancy (3
| currently), and each needs access to all objects.
| PunchyHamster wrote:
| with average object size of 8.5kB I'd honestly consider
| storing it as blobs in cloud DB, with maybe some small per-
| server cache in front
| ethan_smith wrote:
| The app was already built against the S3 API when it used cloud
| storage. Keeping that interface means the code doesn't change -
| you just point it at a local S3-compatible gateway instead of
| AWS/DO. Makes it trivial to switch back or move providers if
| needed.
| rconti wrote:
| Or a simple SAN
| arter45 wrote:
| Compatibility with existing code, middleware and so on.
| lsb wrote:
| Self Hosted object storage looks neat!
|
| For this project, where you have 120GB of customer data, and
| thirty requests a second for ~8k objects (0.25MB/s object reads),
| you'd seem to be able to 100x the throughput vertically scaling
| on one machine with a file system and an SSD and never thinking
| about object storage. Would love to see why the complexity
| jakewins wrote:
| The complexity for that is almost always for redundancy and for
| ease of deploys.
| cuu508 wrote:
| (Author here) that's more or less what I have right now - one
| machine with a file system and an SSD. S3 API on top is there
| to give multiple web servers shared access to the same storage.
| I could have used something else instead of S3 - say, NFS - but
| there was a feature request for S3 [1] and S3 has a big
| ecosystem around it already.
|
| [1] https://github.com/healthchecks/healthchecks/issues/609
| smjburton wrote:
| > In March 2026, I migrated to self-hosted object storage powered
| by Versity S3 Gateway.
|
| Thanks for sharing this, I wasn't even aware of Versity S3 from
| my searches and discussions here. I recently migrated my projects
| from MinIO to Garage, but this seems like another viable option
| to consider.
| zenoprax wrote:
| First time hearing about Versity for me too. I thought "S3
| Gateways" were an Amazon-only service rather than something
| mere mortals could set up.
|
| I've been trying to give some containers (LXC/D and OCI)
| unprivileged access to a network-accessible ZFS filesystem and
| this might be what I need. Managing UID/GID through bind-mounts
| from the host to the container (ie NFS on host) has been
| trickier than I was expecting.
| nyrikki wrote:
| Not really s3, and I haven't touched LXC in a long but this
| may help on the OCI side. I apologize if this is redundant to
| you.
|
| Remember that UID mapping on namespaces is just a facade with
| an offset and a range typically based on subuid[0] and
| subgid[1] today.
|
| In the container `cat /proc/self/uid_map` or by looking at
| the pid from the host `cat /proc/$PID/uid_map` you can tell
| what those offsets are. $ cat
| /proc/self/uid_map 0 1000 1
| 1 100000 65536
|
| Here you know that PID =0 in the container maps to PID 1000
| in the host, with a length of 1
|
| The Container PID offset of 1 maps to the host offset of
| 100000 for a length of 65536
|
| With subuid/subgid you can assign ranges to the user that is
| instantiating the container, in the flowing I have two users
| that launch containers. $ cat /etc/subuid
| debian:100000:65536 runner:165536:65536
| $ cat /etc/subgid debian:100000:65536
| runner:165536:65536
|
| Assuming you pass in the host UID/GID that is how you can
| configure a compatible user at the entrypoint.
|
| But note that only highly _trusted_ containers should ever
| really use host bind mounts, it is often much safer to use
| mount NFS internally.
|
| Host bind mounts of network filesystems, if that is what you
| are doing, is also fragile as far as dataloss goes. I am an
| object store fan, but just wanted to give you the above info
| as it seems hard for people to find.
|
| I would highly encourage you to look into the history of
| security problems with host bind mounts to see the wack-a-
| mole that is required with them to see if it fits in with
| your risk appitite. But if you choose to use them, setting up
| dedicated uid/gid mappings and setting the external host to
| the expected _effective ID_ of the container users is a
| better way than using Idmapped mounts etc...
|
| [0] https://man7.org/linux/man-pages/man5/subuid.5.html [1]
| https://man7.org/linux/man-pages/man5/subgid.5.html
| zenoprax wrote:
| I appreciate the detailed explanation of ID/GID mapping.
|
| > it is often much safer to use mount NFS internally
|
| This is the config I'm trying to move away from! I don't
| see how an unprivileged LXC with a bind mount is worse than
| a privileged container with NFS, FUSE, and nesting enabled
| (I need all of that if I can't aggregate on the host).
|
| NFS and CIFS within the container requires kernel-level
| access and therefore the LXC must be privileged. I'd rather
| have a single defined path.
|
| I tried to get around this using FUSE but it creates its
| own issues with snapshots/backups (fsfreeze).
|
| If my solutiom works for a regular LXC it will probably
| work for Podman.
| nyrikki wrote:
| I can't comment directly on LXC but LXC is very different
| from runc/crun/your-CRI here, not better or worse, just
| different.
|
| With podman, unfortunately we don't k8s Container Storage
| Interface (CSI), so you have to work with what you have.
|
| When I said:
|
| > it is often much safer to use mount NFS internally
|
| What is more correct, is having the container runtime or
| container manager mount them, not the user inside the
| container.
|
| But as you are trying to run unprivileged or at least
| with minimal privileges, which is all we can do with
| namespaces, you are cutting across the grain.
|
| I do use podman pods and containers, mostly for the ease
| of development, but on more traditional long lived hosts.
|
| I have a very real need to separate UIDs between co-
| hosted products, but don't need to actually run a VM for
| these specific use cases.
|
| So I have particular rootful tasks that have to be done
| as the user root root in ansible:
|
| 1) Install OS packages 2) Create service admin and daemon
| user 3) Assign subuid/subgids ranges to those user
| security domains as needed 4) For specific services add
| NFS data directories to /etc/fstab with the 'user' and
| 'noauto' flags
|
| In Podman I would then create podman
| volume create --driver local --opt type=nfs --opt
| device=192.168.1.84:/path/to/share --opt
| o=addr=192.168.1.84,.... podman run -d
| --name nfs_test -v nfs-shared:/opt
| docker.io/library/debian:latest
|
| Which if you don't have the fstab entry will give you:
| Error: mounting volume nfs-shared for container ...:
| mount.nfs: Operation not permitted for
| 192.168.1.84:/path/to/share on
| /home/user/.local/share/containers/storage/volumes/nfs-
| shared/_data
|
| That `_data` is one of the hints of the risk of _host_
| bind mounts, the risk is either having an inode that the
| host cares about or issues across containers etc...
|
| While imperfect, this is following the named volume
| pattern, which really just uses tells about it being in a
| container and doesn't expose the mount inode to the
| container.
|
| What does happen inside the container entry point is
| validating that the expected UID is reachable, adding a
| user with the right UID offset and switching to that
| user.
|
| A misconfigured host bind mount or leaking because you
| can't view who has access are the most common problems,
| and as containers run with elevated privileges _until you
| drop them_ they can get around those protections, even if
| they aren 't elevating to root in a rootless situation,
| they can still access the data of any running container
| with just a few trivial mistakes or new discovered
| vulnerabilities. $ capsh
| --decode=00000000800405fb 0x00000000800405fb=cap_chown,ca
| p_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,
| cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroo
| t,cap_setfcap
|
| While NFS is absolutely a whole new ball of wax with
| other issues, one nice thing is that (at least the
| servers I know of) don't even support the concept of user
| namespaces and UID mapping, which makes it fragile and
| dangerous if you start mapping uid/gid's in, but can be
| an advantage if you can simply isolate uid/gid ranges.
|
| IMHO it will be horses for courses and depend on your
| risk appetite as all options are least worst and there
| simply will be no best option, especially with OCI.
| crabmusket wrote:
| It sounds like you're not using Incus, but for those who are,
| it looks like there is a S3 compatible object storage API
| built in. I haven't used it for anything yet but it seems
| relevant:
|
| https://linuxcontainers.org/incus/docs/main/howto/storage_bu.
| ..
| iamcreasy wrote:
| Given the individual file size and total volume, I'd argue it
| make sense to use move to local only storage.
|
| On a separate note, what tool is the final benchmark screenshot
| form?
| cuu508 wrote:
| The graphs are from Netdata. I'm using it to monitor the
| servers, and also feeding it some application metrics via its
| statsd interface.
| gandreani wrote:
| Do you like Netdata? I'm looking into it. I'm curious if you
| use all the features or just a few.
| choilive wrote:
| Moved object storage from AWS to CloudFlare and have been pretty
| happy. No problems with performance so far. Bills were 90%
| cheaper too (free bandwidth)
| chasd00 wrote:
| > The costs have increased: renting an additional dedicated
| server costs more than storing ~100GB at a managed object storage
| service. But the improved performance and reliability are worth
| it.
|
| Were your users complaining about reliability and performance? If
| it cost more, adds more work (backup/restore management), and the
| users aren't happier then why make the change in the first place?
| encoderer wrote:
| Not the OP but I have some... similar experience. When you run
| a high availability service without a full ops team, reliable
| infrastructure is non-negotiable. Burn out has to be managed.
| kikoreis wrote:
| As a user for over a decade, just here to submit my appreciation.
| healthchecks.io is fantastic.
| bityard wrote:
| As someone who has dealt with wacky storage issues/designs, a lot
| of this "felt" strange to me. Btrfs? Rsync? Then I got to the
| bottom and saw that they were only handling about 100 GB of data!
| At that scale, nearly anything will work great and TFA was right
| to just pick the thing with the fewest knobs.
|
| At a previous job years ago, we had a service that was
| essentially a file server for something like 50TB of tiny files.
| We only backed it up once a week because just _walking_ the whole
| filesystem with something like `du` took more than a day. Yes, we
| should have simply thrown money at the problem and just bought
| the right solution from an enterprise storage vendor or dumped
| them all into S3. Unfortunately, these were not options. Blame
| management.
|
| A close second would have been to rearchitect dependent services
| to speak S3 instead of a bespoke REST-ish API, deploy something
| like SeaweedFS, and call it a day. SeaweedFS handles lots of
| small files gracefully because it doesn't just naively store one
| object per file on the filesystem like most locally-hosted S3
| solutions (including Versity) do. And we'd get
| replication/redundancy on top of it. Unfortunately, I didn't get
| buy-in from the other teams maintaining the dependent services
| ("sorry, we don't have time to refactor our code, guess that
| makes it a 'you' problem").
|
| What I did instead was descend into madness. Instead of writing
| each file to disk, all new files were written to a "cache"
| directory which matched the original filesystem layout of the
| server. And then every hour, that directory was tarred up and
| archived. When a read was required, the code would check the
| cache first. If the file wasn't there, it would figure out which
| tarball was needed and extract the file from there instead. This
| only worked because all files had a timestamp embedded in the
| path. Read performance sucked, but that didn't matter because
| reads were very rare. But the data absolutely had to be there
| when needed.
|
| Most importantly, backups took less than an hour for the first
| time in years.
| winterbloom wrote:
| what happens if you suffer catastrophic failure, you'd lose up
| to one hour of data?
| bityard wrote:
| The system had RAID, so disks could be replaced without
| taking anything down.
|
| We never had a catastrophic failure, but yeah, and hour of
| lost data wouldn't be the end of the world. For regular
| maintenance we would just coordinate with the QA team to
| pause testing temporarily. (Testing was only around-the-clock
| near a product release.)
| jeffrallen wrote:
| > Our S3 API is now served by Versity S3 Gateway and backed by a
| plain simple Btrfs filesystem.
|
| With apologies to the SRE Book ("hope is not a strategy")...
| Btrfs is not a strategy.
| arend321 wrote:
| It's interesting that cloud providers are unable to provide
| stable S3 as a service. Hetzner is unable to deliver stable
| object storage, but given the article neither are OVHCloud and
| UpCloud.
| jerf wrote:
| "Our current (April 2026) object usage is: 14 million objects,
| 119GB"
|
| I mean, I appreciate the openness about the scale, but for
| context, my home's personal backup managed via restic to S3 is
| 370GB. Fewer objects, but still, we're not talking a big install
| here.
|
| This is pretty much like that story of, if it fits on your
| laptop, it's not big data.
| eichin wrote:
| My _phone_ has 327.1G of storage in use right now :-) (years
| ago a friend of mine who was also old enough to complain about
| "640k" suggested that we should just denominate storage in
| dollars, so "reasonable" amounts scaled with Moore's law...)
| rapnie wrote:
| For S3 self-hosters check out Garage, who are developing with
| NLnet funding: https://garagehq.deuxfleurs.fr
| russelg wrote:
| Covered by the article:
|
| >I ran local experiments with Minio, SeaweedFS, and Garage. My
| primary objection to all of them was the operational complexity
| tonymet wrote:
| great writeup. Is s3 a customer or internal requirement? Why not
| write to your disks? Easily 1/3 - 1/5 the price and better
| performance.
| Unsponsoredio wrote:
| love this. one person, 119GB, two drives, rsync. no kubernetes,
| no distributed cluster, no nonsense. just works.
|
| this is the kind of setup that lets you actually go to bed
| without checking your phone every 20 minutes.
| sigmonsays wrote:
| how do you provide highly available fault tolerant storage?
|
| versity does not include any erasure coding or replication...
| CoolCold wrote:
| You can wire lsyncd for near real-time syncs and btrfs snapshots
| for offline storage
___________________________________________________________________
(page generated 2026-04-18 23:02 UTC)