Post 9r6CYXCgJIRfe0SbMu by elw@mastodon.sdf.org
(DIR) More posts by elw@mastodon.sdf.org
(DIR) Post #9r4pwNU5bbNiLZ2jom by sir@cmpwn.com
2020-01-17T01:32:36Z
15 likes, 15 repeats
So, SourceHut is not hosted in anyone's cloud. I own all of the hardware outright and colocate most of it in a local datacenter.I just built a new server for git.sr.ht, and boy is she a beat. It cost me about $5.5K as a one-time upfront cost, and now I just pay for power, bandwidth, and space, which runs about $650/mo for *all* of my servers (10+).Ran back of the napkin numbers with AWS's price estimator for a server of equivalent specs, and without even considering bandwidth usage it'd cost me almost TEN GRAND PER MONTH to host JUST that server alone on AWS.AWS is how techbro startups pile up and BURN their investor money.
(DIR) Post #9r4qHChAuhwEy05uwy by sir@cmpwn.com
2020-01-17T01:37:47Z
3 likes, 1 repeats
Running all of sr.ht on AWS at our current scale would probably cost me more than $250K/year
(DIR) Post #9r4qgmDE02M2MOHOSW by kick@blob.cat
2020-01-17T01:43:05.376428Z
0 likes, 0 repeats
@sir My employer's literally bought buildings to be as close to exchanges as possible and I'm pretty sure that was significantly cheaper (and way faster!) than the second-best that you can get with cloud services.
(DIR) Post #9r4rTK4RPzolGnJsf2 by alex@gleasonator.com
2020-01-17T01:51:51Z
3 likes, 2 repeats
@sir That's crazy. I've been contemplating hosting my Fediverse servers on a bunch of Raspberry Pi's to cut hosting costs. Unclear how feasible it would be yet, but clearly a different scale!
(DIR) Post #9r4rbRQ7yZ2SNiKnnU by sir@cmpwn.com
2020-01-17T01:52:17Z
1 likes, 0 repeats
@alex you should do this if you're prepared to lose all of your data when all of your SD cards fail
(DIR) Post #9r4rjS9f84B0yZSAdM by sir@cmpwn.com
2020-01-17T01:52:29Z
0 likes, 0 repeats
@alex friends don't let friends store important data on SD cards
(DIR) Post #9r4rrGDdOGwEH32B1M by alex@gleasonator.com
2020-01-17T01:56:10Z
0 likes, 0 repeats
@sir Yeah I was just thinking about I/O speed too. I'd dedicate one Pi to the database and connect a proper SSD to it. The rest of the nodes could be configured to fall back to other ones if they fail.
(DIR) Post #9r4s1Xx2JikYAKAvj6 by sir@cmpwn.com
2020-01-17T01:57:59Z
0 likes, 0 repeats
@alex then one rpi acting as a load balancer? Tbh I feel like this is definitely going to end in tears
(DIR) Post #9r4sJL5KtLlvTTXdIW by alex@gleasonator.com
2020-01-17T02:01:07Z
0 likes, 0 repeats
@sir Yeah that's what I was thinking. It's similar to the setup I'm already using in my cloud host: 1 load balancer, 1 db server, and a bunch of disposable app servers. If the load balancer or db node fail it would be very painful. Maybe I could use a more reliable machine for just those nodes.
(DIR) Post #9r4sUH8HvWzOOy8Z1s by sir@cmpwn.com
2020-01-17T02:02:03Z
0 likes, 0 repeats
@alex tbh I don't trust raspberry pis with anything close to production workloads and I don't let them within sight of important data
(DIR) Post #9r4smnkxN62enL3egS by Kalvo@noagendasocial.com
2020-01-17T02:06:35Z
0 likes, 0 repeats
@sir @alex I personally use a NAS running a ZFS mirror and NFS that I use for anything running on a Pi.
(DIR) Post #9r4syJg126KsymEn6u by 6a62@ms-olive.club
2020-01-17T02:08:39.567862Z
2 likes, 1 repeats
@sir @alex the raspberrypi.org site is hosted on a pi cluster ( https://www.techrepublic.com/article/low-cost-raspberry-pi-4-cluster-hosts-official-pi-website-handles-record-traffic-for-pi-4-launch/ ). Also the raspi4 was supposed to get support for network booting with a firmware update at some point, which would at least eliminate SD cards as a failure point
(DIR) Post #9r4t0bLMJqBcLpfIcy by alex@gleasonator.com
2020-01-17T02:09:04Z
0 likes, 0 repeats
@sir Fair, I've had Pis burn out several times. But they're cheap AF. It would be really awesome if these little single board computers could be legitimately used for production, it would level the playing field.
(DIR) Post #9r4tFsELoeGt4KfZgG by sir@cmpwn.com
2020-01-17T02:10:07Z
0 likes, 0 repeats
@6a62 @alex would also probably be okay to boot to a read only SD card and maintain no local state
(DIR) Post #9r4tJ90XtWqbyIluqW by passenger@anime.website
2020-01-17T02:12:26.377547Z
0 likes, 0 repeats
@sir @6a62 @alex how would you even split mastodon into "disposable app servers?"
(DIR) Post #9r4tiviPwDRcH7aW5Q by sir@cmpwn.com
2020-01-17T02:13:00Z
0 likes, 0 repeats
@passenger @alex @6a62 oh, I was mentally extrapolating this discussion to my plans for an ARM cluster for builds.sr.ht. I still wouldn't use rpis to run mastodon instances under any circumstances
(DIR) Post #9r4tzBwRNRb7EVtwtE by grainloom@cybre.space
2020-01-17T02:19:55Z
0 likes, 0 repeats
@alex @sir why not use something from Pine64? I heard they have better I/O.the Raspberry Pi is not the only SOC after all.
(DIR) Post #9r4u2aPmqHqm0LFLmq by Kalvo@noagendasocial.com
2020-01-17T02:20:38Z
1 likes, 0 repeats
@sir @6a62 @alex I have been running my Pi 3 4 node cluster using Docker swarm and NFS for all data storage for about a year and a half without having any SD card failure. All for personal use though not public. So far so good.But I most definitely have had data corruption issues in the past. Maybe it's just me but for over a decade now I have held the opinion that SD cards are some of the most unreliable storage I have ever used. And I used zip disks...
(DIR) Post #9r4uAnost5MftZu54a by sir@cmpwn.com
2020-01-17T02:20:14Z
0 likes, 0 repeats
@grainloom @alex I sent them an email asking for a quote on a pine64 cluster but I haven't heard back. @PINE64?
(DIR) Post #9r4uCuDW4cJOJcQaW0 by sir@cmpwn.com
2020-01-17T02:21:30Z
0 likes, 0 repeats
@Kalvo @6a62 @alex all of my microSD cards are less than one year old. I've seen dozens of microSD cards fail
(DIR) Post #9r4uSph4etPgv6qd7I by PINE64@fosstodon.org
2020-01-17T02:25:02Z
1 likes, 1 repeats
@sir @grainloom @alex We're working on our own cluster of RockPro64 boards to power our infrastructure. Something like this might become available in the future.In the meantime, we have the SOPine Clusterboard: https://store.pine64.org/?product=clusterboard-with-7-sopine-compute-module-slots. 28 cores and 14GB of RAM in a compact package for just barely over $300. :)
(DIR) Post #9r4uVpMcLFZGlWhv3Q by adasauce@radical.town
2020-01-17T02:23:27Z
0 likes, 0 repeats
@sir scary aws calculator says it would cost me $35k/yr just for the bulk of my home setup. 🤢
(DIR) Post #9r4uVyjzTibzqciDEe by klardotsh@toot.cafe
2020-01-17T02:23:42Z
0 likes, 0 repeats
@sir @grainloom @alex @PINE64 they're probably slammed with the Pinebook Pro and PinePhone launches. Want me to cross post this toot to Twitter for ya?
(DIR) Post #9r4uVz6g7MlAyzKKbA by klardotsh@toot.cafe
2020-01-17T02:24:43Z
0 likes, 0 repeats
@sir @grainloom @alex @PINE64 context: I tend to get hella fast responses on Twitter relative to all other platforms
(DIR) Post #9r4uVzQsuEvHzemT5s by PINE64@fosstodon.org
2020-01-17T02:25:29Z
0 likes, 0 repeats
@klardotsh @sir @grainloom @alex Hey, I respond fast too! 😉
(DIR) Post #9r4uiWz7z8zhWrmHqK by passenger@anime.website
2020-01-17T02:28:13.313720Z
0 likes, 0 repeats
@sir @Kalvo @6a62 @alex But have you seen dozens of microSD cards fail at the same time (blackmantappinghead.jpg)? Also if you don't write to them SD cards are great.
(DIR) Post #9r4uisqMiP3bK3qbEu by sir@cmpwn.com
2020-01-17T02:23:55Z
0 likes, 0 repeats
@klardotsh @grainloom @alex @PINE64 sure, thanks
(DIR) Post #9r4ul44tuttmkEgc8e by alex@gleasonator.com
2020-01-17T02:28:25Z
0 likes, 0 repeats
@passenger Here's a nice reference showing the different services that comprise Mastodon. Sidekiq, Streaming, and Rails can each have any number of processes and can be split across any number of nodes@sir @6a62
(DIR) Post #9r4unxWOkZFjHLvbLk by sir@cmpwn.com
2020-01-17T02:24:10Z
0 likes, 0 repeats
@klardotsh @grainloom @alex @PINE64 I'll probably ask about it when I see them at FOSDEM too
(DIR) Post #9r4v0J8oC9Wjo0mZvc by sir@cmpwn.com
2020-01-17T02:26:06Z
1 likes, 0 repeats
@PINE64 @grainloom @alex was looking into that but deemed it unsuitable for my use-case. Would buy a similar cluster to the one you've been blogging about in a heartbeat and use it to bring ARM CI to the masses :)
(DIR) Post #9r4v9WSk2FkXjBV3lw by passenger@anime.website
2020-01-17T02:33:06.600340Z
5 likes, 3 repeats
@alex @sir @6a62 all those servicesmastodon-complexity.jpgpleroma-complexity.jpg
(DIR) Post #9r4vFAZwJPTHarYkaG by sir@cmpwn.com
2020-01-17T02:28:46Z
1 likes, 1 repeats
@passenger @alex @6a62 @Kalvo
(DIR) Post #9r4vFOZkGvdP0JZnE0 by passenger@anime.website
2020-01-17T02:34:10.317031Z
0 likes, 0 repeats
@sir @alex @6a62 @Kalvo >that cardI don't even want to ask... Is that SD card hardware raid?
(DIR) Post #9r4vFQ62cL3vilh6Ce by Kalvo@noagendasocial.com
2020-01-17T02:34:10Z
0 likes, 0 repeats
@passenger @sir @alex @6a62 And if I never turn my computer on it's completely fail safe 😜
(DIR) Post #9r4vLyIqLqXbY2o3Wa by sir@cmpwn.com
2020-01-17T02:34:20Z
0 likes, 0 repeats
@passenger @alex @6a62 @Kalvo yes
(DIR) Post #9r4vPh1SXctL71Rw00 by haitch@mastodon.social
2020-01-17T02:35:59Z
0 likes, 0 repeats
@alex What's your opinion of the PICO cluster?This is a x10 Pi4 edition, but there are others with different boards and a different number of boards.https://www.picocluster.com/collections/raspberry-pi4/products/pico-10-raspberry-pi4@sir
(DIR) Post #9r4vR5rBh3MwQMJSZE by PINE64@fosstodon.org
2020-01-17T02:36:01Z
0 likes, 0 repeats
@sir @grainloom @alex We're planning to have the bigger of our two clusters available for use as a compilation box for developers. 😊To be fair, the hardware architecture of our cluster(s) isn't that complex. It's basically a whole lot of RP64 boards connected together by Ethernet, and data being accessed from a NAS setup. There's some custom fabrication to fit it in a 2U case, but that's not necessary if you're not putting it in a data center.
(DIR) Post #9r4vXuqR2D6eIWDKca by sir@cmpwn.com
2020-01-17T02:36:46Z
0 likes, 0 repeats
@PINE64 @grainloom @alex the fact that I am putting it in a datacenter is why I deemed the other approach non-suitable for my needs
(DIR) Post #9r4vek87GpaZ5qjvPM by sir@cmpwn.com
2020-01-17T02:36:56Z
0 likes, 0 repeats
@haitch @alex I need a rackmount form factor
(DIR) Post #9r4viDUuXSZV2o3fHs by sir@cmpwn.com
2020-01-17T02:37:42Z
0 likes, 0 repeats
@PINE64 @grainloom @alex there are really not a lot of great choices out there for rack mounted ARM setups
(DIR) Post #9r4vk56SifwLaZVmHg by sir@cmpwn.com
2020-01-17T02:37:12Z
0 likes, 0 repeats
@haitch @alex err, sorry, losing track of threads
(DIR) Post #9r4vo10BuRu81mSDNw by PINE64@fosstodon.org
2020-01-17T02:40:11Z
0 likes, 0 repeats
@sir @grainloom @alex The rackmount case is likely to stay a one-off for us. There's not really enough market of people wanting to put together $1000+ ARM cluster setups to justify the manufacturing/setup costs to sell them. The network backplane board could be something which may be sold if people want it.
(DIR) Post #9r4vsgTbeO9VohRHrE by sir@cmpwn.com
2020-01-17T02:40:30Z
0 likes, 0 repeats
@PINE64 @grainloom @alex 🙁
(DIR) Post #9r4w4YphtX9JaNRKUK by sir@cmpwn.com
2020-01-17T02:40:46Z
0 likes, 0 repeats
@PINE64 @grainloom @alex for the record my price rance was well above that figure
(DIR) Post #9r4wDar9k13CT2QOUi by alex@gleasonator.com
2020-01-17T02:45:02Z
0 likes, 0 repeats
@passenger @sir @6a62 Wtf is up with all those pgBouncers everywhere, what's the source of that image, is it serious? I think you only need one pgBouncer.
(DIR) Post #9r4wGlsNE84vVrVeF6 by sir@cmpwn.com
2020-01-17T02:45:35Z
0 likes, 0 repeats
@alex @passenger @6a62 actually it's generally advisable to have one pgbouncer at each server and one pgbouncer at each client
(DIR) Post #9r4wQswHoylEJZA7tI by sir@cmpwn.com
2020-01-17T02:46:01Z
0 likes, 0 repeats
@alex @passenger @6a62 at least for HA setups, or as close to HA as one can get with pg
(DIR) Post #9r4wd6AeS530pKhmsa by alex@gleasonator.com
2020-01-17T02:49:37Z
0 likes, 0 repeats
@sir @passenger @6a62 Any more you could tell me about that? I'm pretty new to sysadmin and trying to understand why more pgbouncers is better
(DIR) Post #9r4wl61EtsEWfm7H4S by sir@cmpwn.com
2020-01-17T02:50:32Z
0 likes, 0 repeats
@alex @passenger @6a62 on the server allows you to load balance between different pg servers on that host, for example if doing a major version upgrade or something. On the client, it allows you to failover to the secondary server without extra work
(DIR) Post #9r4wrr9uY38Gc0OcG8 by alex@gleasonator.com
2020-01-17T02:52:19Z
0 likes, 0 repeats
@sir @passenger @6a62 Thank you!
(DIR) Post #9r4xnDBHLL8hQ0Nv5U by 6a62@ms-olive.club
2020-01-17T03:02:40.825019Z
0 likes, 0 repeats
@sir @PINE64 @grainloom @alex could try and get your hands on some ThunderX2 stuff
(DIR) Post #9r4xsQMRlczfkB5tq4 by sir@cmpwn.com
2020-01-17T03:02:58Z
0 likes, 0 repeats
@6a62 @alex @grainloom @PINE64 "try" unfortunately being the operative word here
(DIR) Post #9r4yOrdJuCOLarSGps by 6a62@ms-olive.club
2020-01-17T03:09:29.120018Z
0 likes, 0 repeats
@sir @alex @grainloom @PINE64 its not cheap but https://store.avantek.co.uk/arm-servers.htmlhttps://www.phoenicselectronics.com/gigabyte/thunderx2.html
(DIR) Post #9r4yWqxIJFUPo9biC0 by sir@cmpwn.com
2020-01-17T03:10:16Z
1 likes, 0 repeats
@6a62 @alex @grainloom @PINE64 oh nice! I've had a hard time finding these before
(DIR) Post #9r5045FsGSPBfThupk by vertigo@mastodon.social
2020-01-17T03:25:38Z
0 likes, 0 repeats
@sir I tried to tell our employer this, but . . .
(DIR) Post #9r50T8Hg8hc8InUmky by sir@cmpwn.com
2020-01-17T03:26:48Z
0 likes, 0 repeats
@vertigo I could probably offer an AWS competitor at half the price which powered all of the servers by burning most of the money you spent on us
(DIR) Post #9r5Q4HtJsg41fOuogi by glaurungo@niu.moe
2020-01-17T08:15:24Z
1 likes, 0 repeats
@sir to their defense, (business owners who buy that) they might be including cost of a full sysadmin team in that calculations.That being said, it's pretty good (financially and performance-wise) when a company has a complete cloud ban. Miracles can be done with older second-hand hardware.And tbh, physically connecting cables and drives is just more fun.
(DIR) Post #9r5glMpL0JXah3cIhk by gchaincl@mastodon.social
2020-01-17T11:25:25Z
0 likes, 0 repeats
@sir what kindof instance costs 10k/month?
(DIR) Post #9r6BJBbgDyk6MhGqTw by njha@fosstodon.org
2020-01-17T17:07:48Z
0 likes, 0 repeats
@sir @alex How do you even store data reliably for clusters? Right now I have glusterfs running duplicated across two SD cards, which I understand to be the sketchiest setup ever. Daily backups to B2 + a big HDD works for my usecase, but means that one specific storage node always needs to be up.
(DIR) Post #9r6BPHwJalqjk9pf7o by sir@cmpwn.com
2020-01-17T17:08:55Z
1 likes, 0 repeats
@njha @alex ZFS with 3 or more hard drives (ideally at least 5), frequent ZFS snapshots, and frequent off-site backups, with periodic backup integrity tests and prod restoration dry-runs
(DIR) Post #9r6CHVTrSkjNNUNPXM by lain@lain.com
2020-01-17T17:19:44.353797Z
1 likes, 0 repeats
@passenger @alex @6a62 @sir great diagram
(DIR) Post #9r6COv1OttHycQbB2W by asonix@asonix.dog
2020-01-17T17:20:52Z
0 likes, 0 repeats
@alex @sir I do this
(DIR) Post #9r6CYXCgJIRfe0SbMu by elw@mastodon.sdf.org
2020-01-17T17:22:38Z
0 likes, 0 repeats
@alex @sir on old pi models Bs as pictured here, you'd be hard pressed to run much. Better off with half the number is Pi 4Bs
(DIR) Post #9r6CxT3wqK482cGSOW by fluffy@freespeechextremist.com
2020-01-17T17:27:20.299741Z
0 likes, 0 repeats
@sir but it doesn't scale
(DIR) Post #9r6Dm2nkF2SdMUnnWa by lain@lain.com
2020-01-17T17:36:27.717483Z
3 likes, 2 repeats
@sir @passenger @alex @6a62 @Kalvo i like the traditional version of russian roullette more
(DIR) Post #9r6EDtusrvKBukQNv6 by alex@gleasonator.com
2020-01-17T17:41:22Z
0 likes, 1 repeats
@lain @passenger @sir @Kalvo @6a62 I feel that it's more like inverse whack-a-mole but I laughed
(DIR) Post #9r6FqJgdbVJEnVdv6G by txt_file@chaos.social
2020-01-17T17:58:36Z
0 likes, 0 repeats
@sir where is your colocation (state, region, city)?
(DIR) Post #9r6FwtAqZmJ2b3uIC0 by sir@cmpwn.com
2020-01-17T17:58:48Z
0 likes, 0 repeats
@txt_file Philadelphia
(DIR) Post #9r6G22FqF4qUCvhHlY by sir@cmpwn.com
2020-01-17T17:59:08Z
0 likes, 0 repeats
@txt_file wait, no, call off the nukes, it's actually in Miami
(DIR) Post #9r6IR8Ze5FJGxG3d1U by txt_file@chaos.social
2020-01-17T18:27:53Z
0 likes, 0 repeats
@sir thanks.