[HN Gopher] A terrible way to jump into colocating your own stuff
___________________________________________________________________
A terrible way to jump into colocating your own stuff
Author : ingve
Score : 146 points
Date : 2024-09-23 05:39 UTC (17 hours ago)
(HTM) web link (rachelbythebay.com)
(TXT) w3m dump (rachelbythebay.com)
| yjftsjthsd-h wrote:
| > 1. Install Linux on the box. Turn everything off but sshd. Turn
| off password access to sshd.
|
| Also, _test_ that it 's properly disabled with something like
| `ssh -v yourserver : 2>&1 | grep continue`, because there are a
| surprising number of ways for that to go wrong (did you know that
| sshd lets you Include multiple config files together in a way
| that can override the main one? _I know that now._ )
| Maledictus wrote:
| and `sshd -T | grep -i password`
| a-french-anon wrote:
| This. OVH's VPS had two .confs reenabling passwords. Now I know
| too.
| omgtehlion wrote:
| default ubuntu 22.04 has at least one, too
| omgtehlion wrote:
| I always remove all 'includes' in sshd config, just to be sure
| I will not get any surprises later...
| zokier wrote:
| I think doing the opposite is better usually, never making
| changes to the main /etc/ssh/sshd_config and always add your
| own customizations to /etc/ssh/sshd_config.d/; that way you
| have clear separation of your own and distro configs, which
| makes life easier especially when upgrading sshd. Although
| I'll readily admit that I don't myself follow that advise all
| the time.
| xyse53 wrote:
| This. And give your drop-in a prefix like 00-*.conf.
|
| Fwiw I don't think SSH adds the include line upstream. Most
| distros add it now.
| hedora wrote:
| In situations where I have more than one box, one (usually a
| little embedded SoC) gets openbsd on it, and is used as a
| bastion host, for exactly the reasons outlined here.
|
| The problems I hit with using Linux for this were different ten
| years ago, but, based on this thread, things got worse on that
| side of the fence.
| cuu508 wrote:
| What's the importance of having a switch/hub? Is it because you
| may want to add more servers later, but the colo host only
| provides one port in their router?
| re-thc wrote:
| Also if you want to manage the IP ranges provided amongst >1
| server.
| pests wrote:
| Basically. Need to pay for a bigger pipe. Same applies to power
| as well, gotta pay to get more capacity.
| kelnos wrote:
| So you can plug your laptop into it at the colo when you set it
| up so you can verify that the server is working right and can
| get to the internet _while you are still there_. And if it isn
| 't working right, it'll be a lot easier to keep everything
| plugged into the switch, rather than swapping between your
| laptop and the colo's network every time you try something to
| fix it.
|
| But yes, if you think you might want to add other machines
| later, preemptively putting a switch in there will make it tons
| easier.
| moandcompany wrote:
| Documenting and testing the hard reset/reboot procedure, as well
| as any expectations, for your colocated gear sounds like a good
| thing to add to this list.
| bigfatkitten wrote:
| Test to make sure it'll actually come back after a hard reset.
|
| Don't do what I did and put a Cisco router on a remote
| mountaintop radio site with the config register still set to
| 0x2142, and then go home.
| pferde wrote:
| Hey, that's just an excuse to go for another hike! :)
| bigfatkitten wrote:
| 3 hour drive in warm weather, or an expensive helicopter
| ride in winter!
| fanf2 wrote:
| Also, check what happens when mains power disappears and
| returns.
| moandcompany wrote:
| Systems Administration the Hard Way... it's an uphill hike,
| both ways, and in the snow :)
| krab wrote:
| A bit less terrible way in my opinion:
|
| Find a dedicated server provider and rent the hardware. These
| companies rent some part of the datacenter (or sometimes build
| their own). Bonus points if they offer KVM - as in remote
| console, not the Linux hypervisor. Also ask if they do hardware
| monitoring and proactively replace the failed parts. All of this
| is still way cheaper than cloud. Usually with unmetered
| networking.
|
| Way less hassle. They'll even take your existing stuff and put it
| into the same rack with the rented hardware.
|
| The difference from cloud, apart from the price, is mainly that
| they have a sales rep instead of an API. And getting a server may
| take a from few hours to a few days. But in the end you get the
| same SSH login details you would get from a cloud provider.
|
| Or, if you really want to just collocate your boxes, the
| providers offer "remote hands" service, so you can have geo-
| redundancy or just choose a better deal instead of one that's
| physically close to your place.
| rsanheim wrote:
| One hurdle that many companies who have only known cloud
| hosting will face here is significant: how do you find a
| reliable, trustworthy datacenter? One who actually monitors the
| hardware and also has a real human if your networking access
| gets screwed or if you need a critical component swapped at 2
| am on a Saturday.
|
| I used to have a short list of trustworthy companies like this
| I'd recommend to clients ~20 years ago when doing consulting. I
| think 3/4 of them have been gobbled up by private equity chop
| shops or are just gone.
|
| Nowadays noone gets fired for going with AWS, or resold AWS
| with a 100% markup from a 'private enterprise cloud' provider.
| krab wrote:
| You're right you need to find a company you can trust.
|
| And for a lot of startups it really makes sense to use AWS.
| But if you do something resource or bandwidth intensive (and
| I'm not even talking about Llama now), the costs add up
| quickly. In our case, switching to AWS would increase our
| costs by an equivalent of 4 - 8 devs salaries. After AWS
| discounts. That's a hard sell in a 15-person team even though
| half of our infra costs already are with AWS (S3).
| vidarh wrote:
| I often recommend a "embrace, extend, extinguish" approach
| to AWS: Starting there for simplicity is fine, then "wrap"
| anything bandwidth intensive with caches elsewhere (every
| 1TB in egress from AWS will pay for a fleet of Hetzner
| instances with 5TB included, or one or more dedicated
| servers).
|
| Gradually shift workloads, leaving anything requiring
| super-high durability last (optionally keeping S3, or
| competitors, as a backup storage option) as getting
| durability right is one of the more difficult things to get
| confidence in and most dangerous ones to get wrong.
|
| Wrapping S3 with a write-through cache setup can often be
| _the_ biggest cost win if your egress costs are high.
| Sometimes caching the entire dataset is worth it, sometimes
| just a small portion.
| krab wrote:
| Well, S3 is hard to beat for our use case. We make a
| heavy use of their various tires, we store a somewhat
| large amount of data but only a minor part ever goes out.
|
| The compute and network heavy stuff we do is still out of
| AWS.
| vidarh wrote:
| That's pretty much the one situation where they're
| competitive, so sounds very reasonable. Some of their
| competitors (Cloudflare, Backblaze) might compete, but
| the biggest cost issue with S3 by far is the egress so if
| not much goes out it might still be best for you to stay
| there.
|
| Sounds like (unlike most people who use AWS) you've done
| your homework. It's great to see. I've used AWS a lot,
| and will again, because it's often convenient, but so
| often I see people doing it uncritically without modeling
| their costs even as it skyrockets with scale.
| throwaway384638 wrote:
| S3 is a decent product with zero competition. You should
| keep s3, it's a fair price.
| vidarh wrote:
| S3 has plenty of competition. It can be a fair price if
| you rarely read and need it's extreme durability, but
| that leaves plenty of uses it's totally wrong for.
| amluto wrote:
| > Wrapping S3 with a write-through cache setup can often
| be the biggest cost win if your egress costs are high.
| Sometimes caching the entire dataset is worth it,
| sometimes just a small portion.
|
| Is there an off the shelf implementation of this?
| mbreese wrote:
| Minio used to support this, but removed the feature a
| while back. It was called "gateway mode". Sadly, I know
| that doesn't help much now...
|
| https://blog.min.io/deprecation-of-the-minio-gateway/amp/
| amluto wrote:
| Every time I contemplate increasing my use of S3 or
| similar cloud services, I get annoyed at the extremely
| sorry state of anything on-prem that replicates to S3.
|
| Heck, even confirming that one has uploaded one's files
| correctly is difficult. You can extract MD5 (!)
| signatures from most object storage systems, but only if
| you don't use whatever that particular system calls a
| multipart upload. You can often get CRC32 (gee thanks).
| With AWS, but not most competing systems, you can do a
| single-part upload and opt into "object integrity" and
| get a real hash in the inventory. You cannot ask for a
| hash after the fact.
|
| I understand why computing a conventional
| cryptographically-secure has is challenging in a
| multipart upload (but that actually all that bad). But
| would it kill the providers to have first-class support
| for something like BLAKE3? BLAKE3 is a tree hash: one can
| separately hash multiple parts (with a priori known
| offsets, but that's fine for most APIs but maybe not
| Google's as is), assemble them into a whole in
| logarithmic time and memory, and end up with a hash that
| actually matches what b3sum would have output on the
| whole file. And one could even store some metadata and
| thereby allow downloading part of a large file and
| proving that one got the right data. (And AWS could even
| charge for that!)
|
| But no, verifying the contents of one's cloud object
| storage bucket actually sucks, and it's very hard to be
| resistant to errors that occur at upload time.
| vidarh wrote:
| I usually just use a suitable Nginx config unless there's
| a compelling reason. It means you "waste" the first read
| - you just let post/put etc. hit S3, and just case reads,
| but it's easier to get right. It's rare this matters (if
| your reads are rare enough relative to writes that
| avoiding the cost of the first read matters, odds are the
| savings from this are going to be questionable anyway -
| the big benefit here comes when reads dominate massively)
| chpatrick wrote:
| I think if you want to host in Europe then Hetzner is the
| clear choice. They won't monitor your hardware for you
| though, you need to let them know if something breaks and
| they'll replace it very quickly.
| jareklupinski wrote:
| > how do you find a reliable, trustworthy datacenter?
|
| drive to a few, and shake some hands. in my exp, the
| difference between colos is usually "actual SOC2/ISO
| compliance" on one side, and "there are no locked doors
| between the parking lot and my rack" on the other, with not
| much in-between that's not for some specialty (radio), and
| these things can only really be seen for yourself
| hedora wrote:
| That's unfortunate. I consider SoC 2 compliance as a
| negative indicator of security (I've been on the vendor
| side of it, and have seen it have significant negative
| impacts on security and reliability in multiple
| organizations).
|
| Ideally, there'd be locked doors, and the data center
| wouldn't be subsidizing performative checkboxing.
| wadadadad wrote:
| I'm curious, can you tell us what the negative impacts
| you've seen are? Are there any audits that you can say
| are a positive indicator for security?
| agartner wrote:
| Spending all their security time/budget on box checking
| rather than actual security.
|
| I'd rather see open ended red team pentest reports.
| jve wrote:
| As a technical person working at datacenter and at still
| handle technical support requests at some capacity,
| interesting to read this stuff from the customer perspective.
| Good to know what is considered and important by customers.
| Maybe sales staff knows all that too well, but for me just
| invokes a smile and some pride in the job I do :)
| krab wrote:
| The absolute trust killer is when your customer finds out
| about some issue on their own - even though you were
| supposed to monitor that. :-)
|
| More important than raw uptime.
| vidarh wrote:
| This. I used to colo lots of stuff, but now mostly use Hetzner.
| But there are many in this space, and some of them even offer
| an API. And some of them (like Hetzner) _also_ offer at least
| basic cloud services, so you can mix and match (which allows
| for even steeper cost cuts - instead of loading your dedicated
| hardware to 60% or whatever you 're comfortable with to have
| headroom, you can load it higher and scale into their cloud
| offering to handle spikes).
|
| The boundary where colo and dedicated server offerings
| intersect in price tend to be down to land and power costs -
| Hetzner finally became the cheaper option for me as London land
| values skyrocketed relative to their locations in Germany, and
| colo prices with them. (We could have looked at coko somewhere
| remote, but the savings would've been too low to be worth it)
| ggm wrote:
| Remote power management can be a godsend. If you can get an ipmi
| console in, you want it.
| michaelt wrote:
| It is indeed very helpful when it's needed - but most IPMI is
| horrifically insecure, so usually it's not connected directly
| to the public internet. Instead it's on a separate network with
| special VPN access.
|
| As this document focuses on keeping things simple, they don't
| have the isolated network / VPN needed to use IPMI.
| qhwudbebd wrote:
| Ipmi is a bit of a double-edged sword. Network-connected access
| to a serial console (including uefi/bios console redirection)
| and the reset button can be a total lifesaver, I agree. I
| wouldn't want to be without a serial console and remote reset
| button either.
|
| But ipmi cards are little network-attached linux boxes which
| run prehistoric kernels and userspace, exposing as many
| services as the half-wits who put together the firmware image
| can shovel in, and are rarely if ever patched by the vendor
| unless there's some really public scandal.
|
| The standard thing to do is to isolate them on some kind of
| private management network in an attempt to shield the wider
| internet from the full majesty of the firmware engineers'
| dazzling skills, but that might be harder to do in the simple
| 'beginner' scenario Rachel describes.
|
| One good simple version when you get up to two servers instead
| of just one is to cross-connect so each machine has ipmi access
| to the other, but neither ipmi service is exposed to the wider
| world.
| omgtehlion wrote:
| Even if your server does not have proper IPMI, most colo
| providers have an assortment of ip-kvms, you can ask for
| those and access your server from any modern browser.
| qhwudbebd wrote:
| I'm pretty suspicious of trusting the firmware on things
| like that either, to be honest. (Personally, I tend to use
| serial cables and a little custom board to turn serial
| breaks into resets, but I know my NIH instinct is probably
| a little on the high side. It would be unreasonable for me
| to suggest someone new to colo start making random extra
| boards to stick inside their machines!)
| omgtehlion wrote:
| You do not need to stick it inside, it plugs into monitor
| and keyboard ports. Power-on and reset are usually done
| manually on request. And you do not need it plugged in
| all the time, only to reconfigure network or boot values.
| qhwudbebd wrote:
| That's definitely a level of remote-hands intervention
| I've not had from the handful of places I have racks of
| colo kit. You're right, it might be something some of
| them do for other customers though; I've never really
| asked. They'll certainly push a reset button, swap a
| drive or rack a server that's shipped to them. (Though
| hardware's reliable enough nowadays and I have go to data
| centres so rarely that it's a bit of a fun novelty, and I
| like doing it myself.)
| schmookeeg wrote:
| I've been colo'ing forever. I'd consider a Lantronix
| spider ipkvm available on request to be like minimum
| viable colo service. It's usually up in 5 minutes when I
| submit a ticket to the NOC.
|
| I bet your provider has something like that. It's a
| godsend when you screw up, say, the hypervisor running
| state somehow and need bare metal access to unbork it :)
| qhwudbebd wrote:
| My version of this is to use a serial console server for
| direct access, rather than vga + keyboard. If the kernel
| is still running fine but network access is down, I have
| a getty running on the serial port. If the kernel has
| locked up too, I hit the reset button by sending a 500ms+
| serial break (special circuit, highly recommended) and
| the bios has serial redirect, so I can do stuff like hit
| del to drop into bios setup or uefi shell at that point.
|
| But I expect you're probably right some or all of the
| providers we use do have something like that, as I
| speculated in the previous post. I've just never
| understood the point of vga-type stuff when bios/uefi
| serial redirect exists and serial console is more
| convenient anyway once the kernel has started, so never
| asked the question.
| sroussey wrote:
| Remote controlled power strip. Turn off and on a server that
| looks bricked to see if you can recover without remote hands.
| mjevans wrote:
| They're absolutely correct:
|
| " 1. Install Linux on the box. Turn everything off but sshd. Turn
| off password access to sshd. If you just locked yourself out of
| sshd because you didn't install ssh keys first, STOP HERE. You
| are not ready for this. "
|
| If you blindly followed the directions and got locked out, you
| can do exactly the same thing with other directions. You were not
| ready.
| kijin wrote:
| The great thing about having unfettered physical access to
| hardware is that you can easily recover from mistakes like
| this. No need to rebuild that EC2 instance. No need to beg a
| hosting company for IPMI access. You can just pull the plug and
| try again as if were your own PC.
| jiggunjer wrote:
| Isn't it the same with something like idrac?
| nasretdinov wrote:
| At least it doesn't say to set PermitRootLogon and remove the
| root password :)
| Foobar8568 wrote:
| AKA, "We are compatible with passwordless authentication
| mechanism." Or SSO, depending of the use case....
|
| Le sight.
| at_a_remove wrote:
| And I find this to be mean-spirited ("snarky") and the opposite
| of instructive. Perhaps I am new, or rusty, and I do _not_ know
| about SSH keys or remember them. Bang! Trapped! And then how is
| a newbie to trust the rest of this, much less learn?
|
| Good instruction will tell you _why_ and the consequences of
| missing a step. Perhaps some options.
|
| But perhaps I want too much, the title does include the word
| "terrible," after all.
| hedora wrote:
| Say you get stuck at this point, and can't fix it yourself.
|
| That's a strong indicator you won't be able to support the
| setup moving forward. What if apt wedges some firewall rule,
| or the machine starts OOMing?
| SR2Z wrote:
| Then your software will be set back by a day! The horror!
| johnklos wrote:
| I'm writing a how-to for colocating for people who're closer
| to newbies. In this case, Rachel's article did suggest it's
| not for newbies.
| behringer wrote:
| Forget the switch and pi and get a ubiquity router. Much more
| powerful and simple to setup. Does require some knowhow.
|
| Also you could see if your local hackers space has a server rack
| seszett wrote:
| The most difficult step I find is just barely mentioned, finding
| colocation space at reasonable price is difficult these days.
| c0l0 wrote:
| I faced the same problem, until I found a non-profit community-
| operated housing provider (a registered computer club/society
| that has been operating since at least the early 2000 who is
| also a LIR (https://en.wikipedia.org/wiki/Regional_Internet_reg
| istry#Loc...), and they give shelter, bandwidth and IP address
| space to anyone who participates in covering their operational
| costs.
|
| I gladly do, and it's the best hosting experience I've had so
| far, and I used to have rented dedicated iron from Server4You,
| Hetzner, Webtropia and the like from 2005 on. Maybe there's a
| similar hidden gem in the area you live in, and you just do not
| know about it? :) Mine flew under my radar for nearly 20 years,
| and even though I knew they existed, I was not aware they'd
| colo other peoples' boxen at very fair rates.
| acatton wrote:
| > finding colocation space at reasonable price is difficult
| these days.
|
| The author of the post seems to be living in the bay area. It's
| easy when the number of nerds per km2 is high, disposable
| income is even higher, and driving a car for 2 hours is
| considered "next door".
|
| I think for losers like myself, living in the middle of nowhere
| with low disposable income, the best solution is just to rent
| out a dedicated Kimsufi box (from OVH), or a server off the
| Heztner auctions. (Or whatever is the equivalent in North
| America) It's much much more cost effective than collocating.
| emptiestplace wrote:
| What audience was this written for?
| schmookeeg wrote:
| I was gonna ask the same thing, but then she called me a usual
| broken people and I decided "not me" was a good enough answer
| :)
|
| This was a tortured way to setup a colo. I bring rackable gear,
| no switch, and use the colo's "setup station" to do final
| config, then I rack the sucker. They can always VLAN me and my
| laptop together somewhere for a final check before I leave. I'd
| be worried if they couldn't.
| kkfx wrote:
| I suggest a very different reasoning: we know WFH works, we also
| agree that in large part of the world we have enough sunlight for
| p.v., battery storage could be cheap [1] and FTTH connections
| start be be very good for modern time. Who have already
| understood where I want to go?
|
| Well, we could run decentralized systems at home for SMEs/Startup
| common needs from modern homes of remote workers, just renting
| them a room with proper local equipment the modern remote worker
| probably already have (VMC, A/C, "the big UPS" from domestic
| p.v.) with not much more expenses than using someone else
| computer also named "the cloud". Then if/when needs mount some
| sheds with p.v. + local storage and good connections can pops up.
| Who need a CDN in such setup? Then/if a real small datacenter
| could born.
|
| Are you horrified? Well, now try to see the recent history, when
| Amazon choose not to buy expensive and reliable SUN servers
| choosing cheap and crappy PCs or Google choose to save money with
| rackmounted velcro-assembled PCs with a built-in UPS, i.e. a
| naked velcro strapped one. See the history of buying cheap low
| quality ram because we have ECC, lo quality storage because we
| have checksums anyway and so on. Then think about some real
| recent datacenter fires and what they told about "how brilliant
| and safe" was their design.
|
| Long story short: as we ditch big iron in the past, we ditch
| mainframes, it's about time to think ditching dataceter model as
| well for a spread set of small unreliable but many machine rooms
| at homelabs scale. This, like FLOSS gives us reliability,
| cheapness and ownership as well. This ALSO decouple hw and sw
| giants, witch is a very good things in terms of free markets. It
| will also force he OEMs to be flexible in their design and
| ditching slowly from modern crappy fats-tech also some giants
| deprecate [2] as unsustainable, for more balanced tech because
| well, such homelabs last much longer than typical ready-made PCs
| and craptops or NAS from the mass distribution market.
|
| [1] buying from China directly instead of feeding local importers
| who skyrocket the price instead of lowering it when Chinese
| supplier lower them
|
| [2] https://unctad.org/system/files/official-
| document/der2024_en... of course suggesting the wrong answer
| omgtehlion wrote:
| It is also useful to configure mutiple IPs on the same interface,
| one for Colo, and one for your lab. You can do this on all OSes,
| even on windows.
|
| And if you failed to do this and all advices in the article: then
| kindly ask colo provider to attach ip-KVM (if your "server" is
| not actual server with ipmi/bmc), 99.9% of places have these. And
| most of them will provide it free of charge for limited amount of
| time.
| elric wrote:
| Back in the late 90s/early 00s when I was a precocious teenager,
| I ran a somewhat popular website. At some point it made sense to
| just buy a 1U rack moubtable server and having it colocated
| (commercial webhosting was expensive then). I couldn't find
| anyone to give me a ride to the datacenter, so I took a bus. By
| the time I got there my arms were numb from carrying the bloody
| thing.
|
| There was a single security guard, I signed in and he gave me
| directions and a big keychain. The keys opened most of the rooms
| and most of the cages around the racks. To this day I remain
| mystified at the level of trust (or nonchalance) that security
| guard had in a spotty teenager.
| qhwudbebd wrote:
| Having run hosting companies from the mid 90s as well, from
| memory this kind of thing was pretty normal, even in allegedly
| secure places like London Telehouse.
|
| Quite a few of us in that era were juggling it with being
| students, so it wouldn't surprise me if the security staff were
| used to it and expected you to look young enough to be their
| kid!
| anal_reactor wrote:
| Modern society is absolutely paranoid. Zero-trust policy,
| everyone is treated like a potential terrorist.
| thelastparadise wrote:
| You're treated lile a terrorist when you are a law abiding
| citizen.
|
| Actual criminal? Society will bend over to accommodate you.
| phil21 wrote:
| It has nothing really to do with that.
|
| To this day I can get into pretty much any rack or room I
| feel like at datacenters everyone here has heard of. It
| just takes experience these days and a bit of charm. Plus
| having a million keys and staff rack combination codes
| doesn't hurt. These were freely given and simply added to
| my collection over time, nothing stolen or social
| engineered.
|
| I've never done anything nefarious with these abilities,
| and no one I know has either. It's simply a matter of
| practicality when you staff a 150,000 square foot facility
| with 2 security guards who have no idea what they are
| doing.
|
| If I (and many others) had wanted to, we could have caused
| multi-week/month outages you'd be reading about on the news
| with 5 minutes of effort. This is basically the status quo
| for any sensitive industry.
|
| The world turns because 99.9999% of people want to give you
| a hug vs. hit you. Society falls when that ratio goes much
| lower.
| viraptor wrote:
| That won't work everywhere. I've been to a datacenter in
| London around 2010 where the entry was similar to the
| automated airport passport booths. The doors would not
| open if you were not registered for a visit and there
| were no visible guards (I hope someone was around in case
| you got trapped...) I wanna say it was Telehouse West,
| but my memory is not great.
| zorked wrote:
| Back in the early 2000s I had a job that required me to enter a
| certain phone companies' datacenter. But, I couldn't, because I
| was a consultant and it was an employees-only area. I had
| permission to make changes to machines in a rack but they
| didn't allow me to enter. There was a guard who would check
| badges but who didn't really check if he recognized people, and
| there were some people who had to enter and leave multipel
| times in a day. Myself, I couldn't get permission to enter via
| the correct channels.
|
| With my director's unofficial approval I was allowed to _try_
| to enter the datacenter. So I just walked very confidently
| towards the entrance, nodded to the security guard like all of
| the regulars who didn't bother showing their badges, and he let
| me in.
| cheschire wrote:
| You can get in anywhere if you just carry a ladder.
|
| https://www.youtube.com/watch?v=NiEMcjSQOzg
| neomantra wrote:
| Don't forget hearing protection! All the fans in there make it
| crazy loud. Facilities often supply earplugs, but consider using
| it with over-the-ear protection.
| andyjohnson0 wrote:
| Interesting article.
|
| > Plug everything in and turn it on.
|
| Most server racks will have C13 or C19 outlets for power
| distribution. For consumer-type devices designed to plug into a
| wall socket, suitable cables or an adapter strip would probably
| be required.
| methou wrote:
| IPMI/BMC is a must if plan to keep the server running longer than
| a release cycle of your Linux distribution.
|
| Also remember ask your provider to assign you a private vlan and
| vpn to it for your own access only.
| rpcope1 wrote:
| A working BMC is great, but I sure wouldn't want it facing the
| open web. The colo would either need some sort of private
| network, or you'd need to add a router and firewall of some
| sort to lock down access to the BMC.
| efortis wrote:
| My tip is adding hot spare disks so you don't have to replace
| broken ones.
|
| For example, in ZFS you can mirror disks 1 and 2, while having 3
| and 4 as hot spares with: zpool create pool
| mirror $d1 $d2 spare $d3 $d4
| thelastparadise wrote:
| How about make it so you can smash the whole rack with a
| sledgehammer or datacenter burn down and system hums along like
| nothing happened.
| iphoneisbetter wrote:
| No; because that costs at least an order of magnitude more
| money than a simple colo setup.
| ForHackernews wrote:
| Why would you do this instead of having the hardware at your
| house? For better/faster connection?
| gsck wrote:
| Reliability, and depending on your work load the ability to use
| it commercially. Your house isn't going to have any meaningful
| reliability for either the network or power.
|
| Your house will have one connection to the outside work, the DC
| should have multiple for redundancy (In case a digger goes
| through your network connection), and same for power any good
| DC will have their own redundant power supplies (Batteries,
| generators etc).
|
| And if your server is serving a lot of traffic, your consumer
| home ISP might not be too happy about it. Theres a reason they
| split plans between home and business.
| nucleardog wrote:
| > Your house isn't going to have any meaningful reliability
| for either the network or power.
|
| Not really a serious argument, but maybe worth looking at
| this these days. Mostly because I think as engineers we all
| have the desire to hit 100% uptime and reliability when in a
| lot of cases "close enough" might be a lot further than you
| think.
|
| On the power end, a lot of people are getting to some form of
| local generation and storage (e.g., solar + batteries). Small
| backup generators aren't entirely unheard of. And on the
| network end, I'm probably not the only person that went to
| multiple connections when my ability to earn income became
| tied to my ability to be online from home.
|
| As it sits right now, my power's not the most reliable in the
| world but because of that I already have a fair sized
| generator and a lot of fuel stored on-site (~5 days of
| gasoline, ~10 days of propane). I'm directly off of a fairly
| regionally important power line, so I'm usually fairly high
| up on the priority list during any major outage. (I mean, not
| "hospital" but if my power's out so are tens of thousands of
| others generally.) My grid power's rarely out for that long,
| and my local power even less since I have the generator. If I
| finally got around to adding some solar and storage to bridge
| between grid going down and the generator firing up I'd be
| able to pretty comfortable ignore power outages for over a
| week before I started needing to worry about making trips out
| to dump fuel in the generator or calling for some propane.
|
| As far as connectivity, I've got three connections across two
| providers. All my traffic hits a real data center first so I
| can bond the connections (because residential) then flows to
| and from my house. One of the providers is wireless so the
| only single point of failure for cut lines is a few feet in
| front of my router.
|
| I'm well aware this gets nowhere close to the reliability of
| a data center. I wouldn't run anything safety or life
| critical this way. I wouldn't run anything making a
| substantial amount of money this way. But there's a lot of
| stuff that fits in below "successful SaaS" and "life
| critical" that I think could tolerate an hour of downtime
| here or there that would get along just fine like this. And
| even if you think "well, you're clearly an insane person no
| one else is going to do that" I think there's still lots yet
| that could tolerate a day or two of downtime every year or
| two just fine. There are a lot of smaller services out there
| that businesses rely on that are hosted in AWS but aren't
| much more reliable than if they were running in someone's
| basement and they survive just fine.
| dboreham wrote:
| Typically having the necessary batteries and generators at your
| house is illegal.
| gr3ml1n wrote:
| I can't imagine it's actually illegal. Maybe cost-prohibitive
| and a bit noisy, but plenty of people have UPS systems for
| their local devices and backup generators.
| johnklos wrote:
| That's just nonsense and is not based on fact at all.
| theideaofcoffee wrote:
| Missing step -1: find someone to get on your staff that knows
| what they are doing, or at least find a contractor with a good
| history of doing this kind of thing. After a certain point, you
| really shouldn't wing it. But if you're at that point, you should
| know if you actually need your own hardware, or if public cloud
| or other options are good for you. Some need their own, most
| don't. There's a lot of nuance there that is difficult to express
| in a short HN comment.
| imperialdrive wrote:
| Agreed. It blows my mind a bit that there is a generation of
| folks that only know Cloud vs Colo. Colo is a walk in the park
| compared to cloud, assuming you've done basic PC building and
| networking. Heck, Cloud _can_ be a walk in the park too for
| most use cases. If you don 't have time to learn it yourself
| over a month or two, there should be folks available to
| consult. A lot can be brainstormed in an hour.
| jareklupinski wrote:
| > the bare minimum required to run your own hardware in a
| colocation environment
|
| i remember the look in the admin's eyes when they asked "alright,
| what kind of hardware are you looking to install?" and I said "oh
| i have it right here" and pulled two Intel NUCs out of my
| backpack
|
| > Consider bringing a screwdriver and a flashlight (any halfway
| decent place will provide those for you, but you never know).
|
| two multitools minimum, sometimes you need to hold a nut with one
| while loosening the bolt with the other
|
| the best space is the one that is right next to a
| Frys/Microcenter/whathaveyou
| theideaofcoffee wrote:
| > "oh i have it right here" and pulled two Intel NUCs out of my
| backpack
|
| NUCs would be like nirvana after some of the jerry-rigged crap
| I've seen dragged into facilities. Maybe I'd like them to have
| a secondary power supply, but then again you had two, but
| they'd make a fantastic little router for something not too
| traffic heavy. Lots of utility use cases for something like
| that in a proper facility.
| jareklupinski wrote:
| > I'd like them to have a secondary power supply, but then
| again you had two
|
| yup! it was cheaper to bring two NUCs than try to get N+1
| power redundancy set up for "atypical" (not u-rack) systems
| :)
|
| then they started making 1U NUC rack mounts...
| brodouevencode wrote:
| Is this cloud repatriation?
| zamadatix wrote:
| Get something like a PiKVM and drop all of the stuff about being
| very local since you can find a cheaper overall provider
| elsewhere and use smart hands once a year either for free or
| still cheaper than picking the local place you can drive to. Even
| if you do the things in this guide perfectly it'll break/hang/get
| misconfigured at some point and the PiKVM (or like) lets you
| remotely hard boot a box instantly without having to drive or
| open a ticket. It also enables you to easily reinstall the entire
| OS remotely if you need to.
|
| If your server/device has an IPMI... get a PiKVM (or like)
| anyways. Not only will you last more than 2 seconds without being
| hacked but it'll have more functionality and be much faster.
|
| If you're in the US there are lots of places in the Kansas City
| area that have ridiculously cheap pricing and it's decently
| centrally located in the country.
| cheschire wrote:
| https://pikvm.org/
|
| Thanks, I had never heard of PiKVM!
| renewiltord wrote:
| Imho, the easiest way is to get a Mikrotik switch, set up
| Wireguard on it, allow only Wireguard in, then use the Wireguard
| Mac App. Build your machines with AsRock Rack or Gigabyte
| motherboards. If you have existing, then add a PiKVM. You'll get
| an IPv4 with a half-cab in most DCs.
|
| HE will give you a full cabinet for $600/month
| https://he.net/colocation.html?a=707336661752&n=g&m=e&k=hurr...
| vfclists wrote:
| Is colo necessary of you have a 1000Mbit symmetrical provider at
| home?
| doubled112 wrote:
| I selfhost a bunch of personal stuff at home and have
| 1500M/940M at home, and I don't think bandwidth would ever be
| the issue.
|
| Frequent IP changes and the residential IP block could make
| mail hard.
|
| If I was looking to host something that mattered, and was a
| little more public, I would move that more for my own sanity.
| Not being able to find me via IP, network uptime guarantees,
| more robust backup power, etc.
|
| If you were some cloud admin and put your website on a resume
| or something, and it was down, I might use that to jump to
| conclusions.
| johnklos wrote:
| I'm writing up a how-to about colocating using one of those Pi
| colocation offers. Glad to see colocation get some more
| attention!
| mastazi wrote:
| She wrote a "part 2" just today
| https://rachelbythebay.com/w/2024/09/23/colo/
___________________________________________________________________
(page generated 2024-09-23 23:01 UTC)