[HN Gopher] Ask HN: Solo-preneurs, how do you DevOps to save time?
___________________________________________________________________
Ask HN: Solo-preneurs, how do you DevOps to save time?
When you're 1 man show (or tiny team) working on some web oriented
application beyond POC level and beyond heroku (for whatever
reason) -- managing your CI, deployments/rollbacks, DBs etc. looks
like a nightmare to me. Just properly setting up and maintaining a
small k8s cluster looks almost like a fulltime job. I wonder how
do your CI, deployment workflows look like? Any particular tools,
practices, guidelines, architecture patterns to reduce pain &
timewaste? How do you CI => deploy/rollback stateless instances?
How do you manage your DBs and backups? Etc.
Author : aristofun
Score : 296 points
Date : 2021-10-12 10:35 UTC (12 hours ago)
| SquishyPanda23 wrote:
| I personally like k8s a lot, but my general approach is to fake
| it until you make it and only add things once it's clear why I
| need them.
|
| I have a few old boxes that run VMs. Some of these VMs make up a
| virtual k8s cluster, but almost everything I need runs on bare
| VMs rather than on k8s.
|
| When I find something particularly annoying to administer, I move
| it off a bare VM and onto k8s.
|
| The main thing this gives me is a migration path off of "a bunch
| of custom shell scripts" toward something I believe will be
| useful in the long run.
|
| A bunch of shell scripts and rsync will work for a long time
| until they don't. And when they stop working you'll be in for
| some pain.
| maxk42 wrote:
| Scaled multiple self-hosted products up to tens of millions of
| hits / month. Scaled several for clients up as far as the
| hundreds of billions of hits per month. (Yes, hundreds of
| billions.) Never seen a kubernetes deployment that was (A)
| necessary or (B) simpler than some alternative.
|
| Unless you're doing something very slow / processor-intensive,
| you'll probably never even need an autoscaler before you can
| afford to hire an expert to do it for you. You'd be surprised how
| far you can get with a small VM running your whole stack.
| PaywallBuster wrote:
| probably deploy infra manually (ec2+rds or vps at random
| provider)
|
| Deployments can be automated
|
| I usually work with PHP, so I already have playbooks to
| configure/install php/nginx and others
|
| (just change few variables)
|
| deployment only needs to add a few steps like git checkout ( easy
| to reuse since you're probably using the same framework across
| projects)
|
| some examples here
|
| https://github.com/MalpraveCorp/ansible
|
| in specific
|
| https://github.com/MalpraveCorp/ansible/blob/master/mal-be-p...
| vishnugupta wrote:
| Not solo but I head a tiny tech team (4 engineers) so we don't
| have bandwidth for DevOps stuff. Here's our current state.
|
| We are all in on AWS.
|
| 0. Tech stack: NodeJS, React, RDS Aurora, Redis, S3, Route 53,
| SES, Lambda and a few more.
|
| 1. 3 EC2 machines behind an ELB. Deployment is scp + cd
| <deployment-directory> + ./deploy.sh prod
|
| 2. Aurora takes care of everything around DB for us. Backup,
| read/write replica etc., etc.,
|
| 3. No CI; our development cycle isn't rapid enough to warrant
| investing in CI.
|
| 4. We recently moved some of long running tasks to Lambda.
| There's definitely a big value here so we'll invest further on
| it.
|
| 5. Also, recently, we experimented with an internal service on
| "ALB + AWS Lambda"; it's a sweet combination so we'll invest
| further.
|
| 6. Frontend is hosted on AWS Amplify. It's an underrated nifty
| service; highly recommend it.
|
| 7. Datadog is for monitoring. It's quite good but a bit
| expensive.
|
| I can tell you from my experience that CI, auto-rollback etc.,
| are overrated for smallish teams given their development
| velocity. Even if you have to do once or twice a deployments per
| week you can do it manually, don't need to wade into the CI
| territory. And unless you have prior experience steer clear of
| k8s.
| adventured wrote:
| I've been a solo Web/Internet entrepreneur for most of 20+ years,
| non-stop.
|
| The DevOps side of things is trivial in the beginning and takes
| very little time compared to actual development.
|
| Ubuntu server, Nginx, Mysql or Postgres, PHP, Go, Redis.
| Configured reasonably, it's a ridiculously reliable stack where
| things very rarely go wrong.
|
| I prefer DigitalOcean these days. Takes <15 minutes to configure
| a base new setup - a template - from scratch, make a few
| adjustments and double check everything. From there I can pop up
| a lot of servers as needed. I usually tweak things based on the
| project, although that doesn't take a ton of time.
|
| And that's it. Back-ups on eg a database focused droplet are
| automated by DigitalOcean. Occasionally I have specialized back-
| up needs, and I'll do a bit of custom work for that. Most of this
| could be offloaded by using DigitalOcean's database service, I
| just prefer to limit cost by not.
|
| Under no circumstances would I use Kubernetes for a smaller to
| medium size service.
| popotamonga wrote:
| I do all like this:
|
| GiHub actions-> docker build->docker push to github package repo
|
| then on vm a 2 line script to docker pull && docker-compose up -d
|
| Database all by hand.
|
| Extremely easy i just push, wait build and restart
| FpUser wrote:
| Here is one particular but typical example - enterprise backend
| server written by yours truly, PostgreSQL and single page
| frontend application in Javascript. I do it for my own company
| and also develop in a same manner for my clients.
|
| 1) Server is high performance native C++ application running on
| rented native hardware on Linux. It processes thousands of
| requests/s in sustained manner hence no real need to use any of
| that k8s / docker / etc. There is also separate rented standby
| server.
|
| 2) I maintain a reusable single script that can rebuild complete
| system from the scratch with a single command. I periodically
| test it on local VM. It installs all needed compilers, tools,
| PostgreSQL, restores database from the latest backup, checks out
| from VCS and builds that C++ server, registers it as a daemon and
| refreshes Javascript Frontend.
|
| 3) I also wrote small server that subscribes to webhooks from VCS
| and can build artifacts. I only use it to pull and build
| Javascript part. It could do main server as well but being super
| cautious I trigger that step manually fingers crossed.
|
| 3) for DB I use automatic scheduled backups.
|
| All in all after I've debugged and tested new version locally the
| release takes seconds. Script is written years ago and does not
| require much maintenance other then updating artifacts to new
| versions (replace apt-get install XXX-31 with apt-get install
| XXX-32) and register and build ZZZ server instead of YYY server.
| Compared to insanity I saw in some orgs my setups are piece of
| cake.
| Longwelwind wrote:
| I'm not a solo-preneurs, but I have some open-source projects
| (mainly a live one, and one that I'm currently developping).
|
| Here are the list of DevOpss flow I've done:
|
| * One project is a website. I have a pipeline, triggered when
| something is merged to master, that builds Docker images, pushes
| them to a Dokku instance and deploy them.
|
| * One project is a JS library. I have a pipeline, triggered when
| a git tag is created, that packages the library and uploads it to
| npm. I use Netlify to host the documentation website of it.
|
| * For this same project, I'm planning to have a pipeline
| triggered for each PR to run the linter and tests.
|
| I work as a DevOps/MLOps, and I'm quite comfortable with CI tools
| in general. I hate doing things manually and I love automating,
| so it comes quite naturally for me to do it. Once it's set up, it
| rarely needs to be changed.
| woudsma wrote:
| Not a solo-preneur, but I've had great experiences so far with
| Dokku, an open-source Heroku clone. I'm running ~40 apps on my
| single VPS since a couple of years. I use it in combination with
| Github Actions, Bitbucket Pipelines and Gitlab CI. I still feel
| very flexible working with Dockerfiles and different cloud
| services. I've only had to update the dokku-letsencrypt plugin a
| couple of times in the past 5 years. Note that I haven't had to
| protect important databases with PII for example.
| lbayes wrote:
| Ex-Googler here and 20+ year veteran of SF startups.
|
| It sounds like you have a great set of intuitions, and you're
| right, all that infrastructure is a nightmare to set up and
| manage.
|
| Step 1: Question every requirement
|
| Step 2: Drop everything that is not critical
|
| Step 3: Profit!
|
| Git hooks and Makefiles are great!
|
| I have an HP workstation I bought from eBay under my desk that
| has 32GB of RAM, 2x 12 core CPUs and a static IP address. It's
| probably 8 years old, but fast enough to serve whatever we need
| for a long while.
|
| The machine (Old Blue), hosts our Git repos, web app, database
| and integration service (git hook that calls, 'make publish').
|
| We're not serving Google scale traffic, so we don't need Google
| scale infrastructure.
|
| Keep it simple whenever possible and don't let the modern stack
| complexity creep in until you absolutely need it.
|
| Even going to the cloud, when you've done it 10 times and know
| how, is way more work than you need when just starting out.
|
| Take on those costs and complexities only when your traffic
| requires it, and you may just find out that you never have to pay
| rent for compute.
| djakaitis wrote:
| I hope you're backing up customer data if you store any...
| lbayes wrote:
| Definitely.
|
| In fact, it's super easy to back up as it's in a SQLite
| database (for now).
| TekMol wrote:
| I scaled to millions of users as a solo founder and still run the
| whole show myself. AMA if you like.
|
| Here is what works for me:
|
| CI: From the terminal, I run my tests and commit to git.
|
| Deployments: rsync
|
| Rollbacks: Never did one. If something breaks, I fix it and rsync
| the fix to production.
|
| DB: MariaDB
|
| k8s: I don't use it. Computers are very fast these days. A cheap
| single VPS will get you a long way.
|
| Nightmare: Not really. I spend about 30 minutes per week on
| DevOps.
| nkzd wrote:
| How do you host your DB? Do you use managed DB service?
| TekMol wrote:
| I run it on my VPS along with the rest of my project.
|
| If I want to set it up on a new server, I start a fresh
| Debian machine, do "apt install mariadb-server" and I am good
| to go.
|
| I don't like managed services (like Heroku) because:
|
| 1: I cannot easily replicate the setup locally.
|
| 2: It creates vendor lock-in which will cause me trouble when
| the vendors service changes in a way I don't like.
|
| 3: I would not be able to put the whole setup of my project
| in one setup script. It is so nice to just have a single
| script that contains all the steps that need to be done (Like
| the "apt install mariadb-server") to run the project on a
| fresh machine.
| strzibny wrote:
| If someone wants to know how to host a database for the first
| time, I have a full chapter on PostgreSQL (and one on Redis)
| in my new book https://deploymentfromscratch.com/. There is
| also a demo that sets up an independent database server
| (which you can treat as your own managed db).
|
| It's easier than you think if you stick to your
| distributions' packages. Just need to know how to 1, do auth
| right 2, enable SSL 3, run with SELinux 4, set a different
| location (for attached storage) 5, managed backups.
| whitepoplar wrote:
| Any chance you have an HN discount code? :-)
| strzibny wrote:
| I want to do Show HN, but probably have to prepare a
| bigger preview for that. Grab the "ruby30" code from the
| Reddit announcement: https://www.reddit.com/r/ruby/commen
| ts/pg3c9c/i_posted_on_th...
| [deleted]
| CodinM wrote:
| What is the domain of your venture? (assuming it's not
| something you want to advertise specifically)
| TekMol wrote:
| I like to keep it off my HN profile. I feel I am more open to
| share details this way.
| lambdaba wrote:
| Do you mind putting in some contact details (even
| temporarily) on your profile? I'd like to have a chat with
| you some time, I was looking through this whole thread for
| a comment like yours and I resonated a lot with you. (my
| details are in my profile)
|
| Not immediately, but in the following weeks.
|
| Thanks again for sharing.
| tommiegannert wrote:
| Agreed on rollbacks. I think rollbacks are a thing when you
| don't know the code, or you're too busy to be able to debug.
| Like when being on-call for a large codebase.
|
| If you're solo, you probably know enough that fix-forward is
| the way to go. (This assumes the system isn't critical enough
| that it needs fixing even if you're asleep, but then you would
| probably not be solo anyway.)
| orky56 wrote:
| Not sure if it's a philosophical question but as a
| solopreneur I have always rolled back since it keeps my
| commit chain cleaner rather than have previous versions that
| were buggy. I want to minimize downtime and a rollback is
| always faster than a fix-forward even when you know, or think
| you know, the fix.
| juanse wrote:
| Which tech stack are you using?
|
| PD: I follow a similar approach. I am glad you said it.
| launchiterate wrote:
| What are the most important things you learned and what would
| be your core advice for someone who wants to run a solo
| operation? How do you schedule your time to work on your
| project and how has it changed over time? Thanks in advance!
| TekMol wrote:
| I would say the most important thing is a combination of two
| things:
|
| 1: Work on something that you yourself or your users want to
| exist. Don't think about what others _might_ want. Make
| something that you consider pretty cool shit. Or something
| your users are asking for. Something where you or your users
| say "OMG when this is ready I will fricking love using it! I
| can't wait!".
|
| 2: Most founders, me included, often get stifled. They stop
| putting time into their project because they think it is just
| a dream. Remember that all amazing things have been built by
| people just like you. And to keep putting time into building
| something that you yourself or your users want.
| launchiterate wrote:
| Thank you so much !!! And congratulations on all you have
| accomplished!
| aristofun wrote:
| Best comment so far. Thank you for being specific and to the
| point.
| CodinM wrote:
| I'm using Hetzner, Terraform, GitHub Actions, Docker to automate
| the build and release of an app I'm planning on launching
| (..eventually). I use Terraform to provision the Hetzner VMs with
| Docker running on them, in there I have a container with the
| GitHub Runner that pulls Flutter and creates a nice build
| environment as well as release env (using Fastlane), then it uses
| the latest code, compiles, releases to Play store. Works like a
| charm, initially it was just a Docker container on a Pi, but the
| main reason I went that route was being able to move it
| whenever/wherever.
| NomDePlum wrote:
| I'm starting to consider this myself and have been looking at
| https://serverless-stack.com/#guide as a way to prototype and
| build an MVP. The guide has quite a bit in it that I believe can
| be repurposed to that end although it doesn't cover backups etc,
| more integrations with AWS services.
| jayair wrote:
| Yup it's meant to be a starting point for projects of all
| sizes. We built our own stuff using it as well. Feel free to
| reach out if you have any questions. Or join us on Slack:
| https://launchpass.com/serverless-stack
|
| Good luck!
| mfrye0 wrote:
| I agree with most people here. Definitely keep it simple.
|
| With that said, I run a semi resource intensive operation, so
| I've invested a bit into dev ops to keep our costs down. My setup
| is currently on AWS, primarily using ECS, RDS, Elasticache. Infra
| is managed via Terraform.
|
| I felt ECS was a nice balance vs K8s, as it's much simpler to
| manage, while getting the benefit of maximizing resource
| utilization.
|
| For CI / deployment, I use Github Actions to build an image and
| push, then start a rolling refresh to update the containers to
| the new version. It was pretty easy to setup.
|
| On DBs, RDS handles all the backups and maintenance. For
| migrations, I use https://github.com/amacneil/dbmate.
|
| Happy to answer any other questions you have, as I've learned a
| lot through trial and error.
| RapperWhoMadeIt wrote:
| I have this friend who recently founded a start-up with two other
| guys. But he basically is the only one doing any DevOps, so we
| could say he is a one-man show regarding the technology side of
| things. Two weeks ago we talked about how he is developing his
| product and he told me he will go full FaaS as much as possible,
| because it lets him forget about the whole Ops side of things and
| focus in real product development (what will ultimately bring
| revenue to the company). I am a big fan of playing as much
| sysadmin as possible for my own developments, and I like the
| CI/CD mentality associated with microservices and containers. All
| I am saying is that that conversation with that guy really opened
| my eyes to the fact that maybe in some scenarios (specially at
| the birth of a company) going full serverless might improve the
| time to market of your product. I am in no way saying that
| serverless also has a ton of disadvantages, e. g. vendor lock in.
| aranchelk wrote:
| My strategy: get really clear on business
| needs/objectives/internal SLAs for infrastructure. In my
| experience a lot of time-consuming stuff revolves around uptime,
| data recovery (RTO, RPO), security, compliance, and scaling.
| Depending on your business and the stage you're in, a lot of that
| may not matter, so don't build anything you don't need now.
|
| I'm a solo-founder, with many years working in IT, and I focused
| on DevOps for part of it; I know what best practice looks like
| and it would be easy to fall down that rabbit hole doing an
| unnecessarily complex buildout. Currently I'm doing a small
| private beta, I'm avoiding 99% of standard practice. I've got a
| single EC2 node running Redis as the only datastore and NodeJS +
| Nginx + certbot, a cron to do backups to S3. No CI (there are no
| other devs to integrate code with) I run all tests locally and
| push/rollback with rsync. All code, assets, and server config
| (except creds) are in a monorepo.
|
| If the server goes offline, I will have a little downtime, that's
| fine. If I run out of memory for Redis (not likely to happen
| soon), I'll change to a different datastore or scale up the node.
| If I lose data, I can restore from S3, and additionally the
| architecture is such that clients will re-push their latest
| changes.
|
| Do the bare minimum to support the business, stick with what you
| know, outsource what you can, and properly value your time.
| BiteCode_dev wrote:
| Talking about myself, as a python dev:
|
| - cheap VPS for hosting, postgres for the db, nginx as a proxy,
| redis for everything else, including caching.
|
| - deploy python projects by packing them as zipapp with shiv,
| then use fabric to ssh and performs any migrations necessary. No,
| not even ansible.
|
| - build, lint, format and test, like all automatizable stuff, are
| made using pydoit. If you are solo, you don't need a CI service.
| Your laptop is the CI machine.
|
| Scaling is always the same story:
|
| 1 - start with the cheapest VPS possible. This will force you to
| code with some constraints. Not crazy, just enough to balance it
| out.
|
| 2 - once you start seing your load average rising, just move the
| db on a second cheap VPS. You just tripled the load you can
| sustain. Not doubled, tripled. Most solo projects will stop here
| actually.
|
| 3 - wait again, collect some perfs data when you see the site
| starting to be slow. Check for slow queries, code hot paths, etc.
| Optimize that, add some cache, setup tasks queues. Use cron to
| purge / regenerate cache if you need to smooth out the curve.
| Maybe slap some varnish for extra peps, or just cloudflare.
|
| 4 - now you got what looks like a real load, and a more realistic
| code base, so when you peak again, request more perfs for your
| server, or migrate to another bigger one. You can scale
| vertically to crazy highs nowadays. Really, really crazy. You can
| get terabytes of RAM, 64 cores servers, etc. At your current
| scale, your service should generate enough money to pay 10 times
| for it anyway. But you probably won't need to. Even cheap servers
| are beast, look at the current leaseweb offer:
| https://www.leaseweb.com/dedicated-servers#NL
|
| For EUR320.09 per month, you get: 2x 16 cores
| 2.30GHz 28GB DDR4 4x960GB SSD 30 TB
| traffic
|
| To put it in context, myspace used to serve all its users with 2
| servers only, until they reached 500,000 accounts. Only then it
| was too much. This was with hardware (and price!) in the years
| 200x.
|
| 5 - you will probably never reach this point. This is the point
| where kubs, load balancers, sharing, etc. start to be
| interesting.
|
| How to preserve your data:
|
| - Raid
|
| - Dump the db with a _randomized_ cron
|
| - rsync the dump and all assets. You can do it to another server,
| or just your laptop at the beginning.
|
| You can get fancy with database replication if you want, or use
| backups that stream in real time.
|
| But there is one trick: not all data are equal. Identify some
| data in your db that you can't afford to lose, and make sure this
| one is saved separately and very regularly while being given
| priority. There are plenty of data, if you have a hole in it,
| most people don't care. E.G: if 1 tweet out of million from 10
| years ago is missing, do you think it affects the service?
|
| Monitoring:
|
| - Sentry
|
| That's all. It's free (or cheap), and for a small service, you
| don't need real time. It's ok to be down a few hours once a month
| for most services at first. I have a service with 700k unique
| users a day, it still goes down sometimes. It's a blip.
|
| Sometimes, log into the servers, run htop and checks what's up.
| You can install open telemetry later if you really need to, but
| for now, even HN hug is not going to kill you.
|
| Summary:
|
| - Modern software and hardware are amazing. Max them out.
| Horizontal scaling is hard, and expensive.
|
| - You don't need a perfect service. Unless you are handling
| patient cancer data, that is. Don't worry about perfect uptime, 0
| data loss, etc. If you are a solo dev, the cost for that is huge.
| Just do 97% right[?]
|
| - what worked 20 years ago still work today, and will likely be
| there tomorrow. And you can move from that to the cloud later.
| The reverse is not that nice.
| trhr wrote:
| People are renting 32 cores for E$320/mo?
|
| ...Anybody wanna rent 56 cores and 128GB of RAM for that rate?
| BiteCode_dev wrote:
| If you can give the guarantees and convenience that lease web
| give, then yes.
|
| Is your bandwidth unmeterred?
|
| Do you have redundancy everywhere, including link and power?
|
| Do you have software to remotely manage the server, including
| install, reinstall and emergency access?
|
| Can I pay you with the company credit card? Do you provide a
| billing system with VAT?
|
| If the answers are yes, then congrats, you can open your own
| hosting business.
|
| But remember next year, the servers will be obsolete.
| a_square_peg wrote:
| I run https://oikolab.com - it's a weather data service with
| global, hourly historical weather data from 1950 to 16-day
| forecast. It launched about a year ago and I think it's probably
| the largest data offering of its kind.
|
| I don't have dev ops experience (I still don't know much) so a
| lot of what I did was to exhaust the limits of the setup before
| implementing any new infrastructure, sort of learning as I go
| along, with many 'oh, so that's why they do this' moments. I've
| read about K8 but still can't see when I would need that though.
|
| A lot of what saved effort for me was to use off-the-shelf
| component whenever I can. I was quoted $10~20k to build a Django
| website for user & API management to connect to my back-end but
| it was much easier to use Azure API Management service
| (~$400/month), which also came with simple, reasonable looking
| web front-end that I was able to launch within a week.
|
| Before moving things to cloud (Digital Ocean), I've also
| exhausted what I was able to do with simple NAS servers (overall
| I processed about 200TB of raw data with ~1TB/day on-going using
| what I call a 'poor man's HPC' setup) - luckily I'm based out of
| Hong Kong so had access to fiber-optic internet at home.
| terrortrain wrote:
| Nice, I bookmarked your site. Weather data is surprisingly hard
| to navigate. Did you source it from NOAA.gov or somewhere else?
|
| I tried to slap together a simple weekend app based on
| historical weather data by zip code, and never even got to the
| coding. Just spent most the weekend trying to decipher those
| API's.
| a_square_peg wrote:
| The primary forecast data is from NOAA while the historical
| data is from ECMWF (sort of the European equivalent of NOAA).
|
| I've been told that my API is easy to use - feel free to let
| me know your thoughts if you get a chance.
| todd3834 wrote:
| I think the key to getting a good answer here is the "beyond
| heroku" part. It is hard to make suggestions without knowing why
| a platform as a service no longer works.
|
| There are so many different options available but they very for
| different needs.
|
| Regardless, I would assume that you are going to have to pay more
| for a managed solution that managing everything yourself.
|
| I know the founders at https://releasehub.com/ and it sounds like
| it might hit that sweet spot for "outgrowing heroku" but "don't
| want to do full time devops".
| Inhibit wrote:
| First you worry about meeting your paying customers need. Then
| you spin up just enough infrastructure to handle effectively
| meeting the need to continue getting paid.
|
| You can do that with some shell scripts on an instance and an
| e-mail service and manually copy the whole mess for backup if it
| makes sense. The _how_ of implementation is very case specific.
|
| Thinking more on it I'd sum up as "provide a service that doesn't
| require that you be Google". If you start by specifying an
| impossible amount of infrastructure maybe there's an issue with
| management :).
| leros wrote:
| I could run things myself on k8s or other self-managed systems
| but I have better things to spend my time on and it's not my
| expertise.
|
| I host my apps and databases in Heroku. Yes it's more expensive
| than alternatives, but it's easy and I trust their database
| backups.
|
| For everything else, I find managed services.
|
| Basically, I'm a developer and I let other companies handle most
| of my DevOps for me.
| adamcharnock wrote:
| I think this is going to be a pretty contrary comment, let's see
| how this goes.
|
| I don't know what everyone is doing with kubernetes that takes up
| so much time and effort, especially in a SME environment.
|
| I'm currently operating three separate kubernetes clusters for
| three different clients. And this is just me, there are no other
| tech people involved day-to-day. These clusters require minimal
| maintenance, just the occasional upgrade.
|
| These clusters are small (1-10 nodes), and are on either GKE or
| bare metal. The biggest is around 100-200 running pods. I have a
| CI/CD pipeline setup in GitHub actions for each client.
| Development & CI uses docker-compose.
|
| I used to do the old-hat 'scp your code to the server' kind of
| deployment, and this was a _lot_ more pain. Kubernetes is big,
| but it solves _so many_ problems. Ensuring everything is actually
| running, networking, SSL certs, persistence, configuration, etc
| etc.
|
| I've been running this setup for 3+ years now. If I cast my mind
| back, I think I spent a weekend reading the k8s docs concepts
| section, then pretty quickly deployed a cluster on GKE. Given it
| was a team of one, I ditched RBAC which removed some complexity.
| Getting the CI/CD pipeline setup took some time (maybe a few more
| days), and I built more tooling as a went.
|
| I really think kubernetes can be really useful for tiny teams,
| but there is a lot in there that is aimed at big teams. Use the
| bare minimum possible at first, and grow from there.
|
| ---
|
| Now here is a bonus unsolicited contrary comment.
|
| Deploying k8s used to really require a cloud provider,
| particularly for persistent volumes and load balancing. But I
| think that has changed over the last few years. OpenEBS and
| MetalLB look like they are really addressing this.
|
| Also, has anyone noticed that dedicated servers are cheap as
| chips now? I think the equivalent dedicated server would cost
| around 10x as much as an AWS instance. (I'm looking at Hetzner in
| particular, but OVH isn't far behind)
|
| Given what is currently available from the k8s ecosystem, there
| seems to be a really strong case for deploying on bare metal
| these days.
|
| For example, at a ballpark you can get a 400GB+/40core cluster
| for around $300/month (which also includes another 80GB/10cores
| of ancillary gubbins). The cost for that easily comes in at
| around $3,000 on AWS (ex any storage).
|
| OpenEBS gives you replicated storage, MetalLB gives you your load
| balancing, and each instance has a couple of terabytes of NVMe
| too. And, if you're worried about reliability of physical
| servers, at that price you could replicate your setup across
| datacenters at minimal cost.
|
| ---
|
| I should add a caveat here. I've been working freelance in tech
| for 15 years and I am a massive generalist. I do everything from
| business consulting to design, devops to development, and I also
| run an ISP so know my way around networking/BGP etc. That being
| said, setting up a quick cluster on GKE doesn't require many of
| those skills.
|
| Having written this ambling comment I guess I'll also say that I
| am am available if anyone wants a cluster setting up. I'm also
| about to start providing managed k8s services.
| bckygldstn wrote:
| For my API SaaS I run everything non-critical (marketing website,
| analytics, customer dashboard) on a single server, either part of
| a single django monolith or using an official docker image with
| some command-line arguments. Only the API gets duplicated and
| load-balanced with Cloudflare.
|
| Dedicated servers, specced to provide lots of capacity for
| spikes. A 20-node k8s cluster could fit on 2 beefy dedicated
| servers for about the same cost. Decreased infrastructure
| redundancy but massively increased operational stability through
| simplicity.
|
| Everything runs in a docker-compose project: one for the API,
| another for the everything else monolith. I've worked for a
| couple of small companies that ran with docker-compose, so have a
| good sense of the weaknesses and footguns (breaking your
| firewall, log rotation, handling secrets, etc).
|
| CI is running `make test` on my dev machine. Deployment is `git
| pull && docker-compose up --build`. Everything sits behind
| haproxy or nginx which is set to hold and retry requests while
| the backend is down, so there aren't any failed requests in the
| few seconds a deployment takes, just increased latency. I only
| deploy to the API once or so per week, that stability reduces
| headaches.
|
| DB backups are done with cron: every hour a pgdump is encrypted
| then uploaded to backblaze. Customer subscription data is
| mirrored from stripe anyway so an out of date DB backup isn't the
| end of the world. Error if the backup fails after a few retries.
|
| Sentry on everything for error alerting. All logs go into New
| Relic.
| domk wrote:
| Don't.
|
| Use a PaaS like Heroku, hook it up to your GitHub and a CI
| platform. Get your database hosted on Heroku or somewhere similar
| - plenty exists for anything you might need (Postgres, MySQL,
| Redis, Mongo...).
|
| Don't do things that increase your operational overhead like
| using microservices or running unusual databases for which a good
| hosted solution doesn't exist. Stay away from anything k8s.
|
| It will be a bit more expensive but well worth the extra money,
| compared to the time you'd spend on operations.
| barelysapient wrote:
| Keep it stupid simple.
|
| Use tech that you understand and minimize the layers of
| complexity between deploying, starting and troubleshooting an
| instance.
|
| For me, I like digital ocean droplet(s). Stupid simple IAM. SSH
| scripts for CI deployments. Everything sits behind a Cloudflare
| cache.
|
| The backend sends key events like restarts and fatal errors into
| a dedicated Slack channel. Its my confirmation a deployment
| worked and first alert if something crashed.
|
| My business partner and I are currently running a $1m/arr
| business with this setup on a single digital ocean instance,
| Python3 and Nuxtjs.
| whitepoplar wrote:
| 100% this! Just curious--what's your business?
| barelysapient wrote:
| We're in the hospitality sector.
| piyiotisk wrote:
| I've used this in the past https://github.com/piyiotisk/fieldbot-
| server/blob/master/.gi...
|
| Basically, building a docker image and server it using Docker
| compose. Of course you'll need NGINx as a reverse proxy running
| on the server.
|
| If I do it again I'll use a service like render. Not worth
| managing this myself.
| hubarb wrote:
| Lots of easy to use CI tools about. Bitbucket, Github both have
| easy to use config based pipelines/runners. Very little to manage
| there. Build containers and get somebody else to run them for
| you. There are lots of options to do that too. There should be
| very little overhead in managing that.
| redis_mlc wrote:
| You can just use EC2 with yum update and RDS to do 90% of what
| you mentioned.
| vmt-man wrote:
| I prefer Gitlab CI for solo projects, it's super convenient. Just
| create .gitlab-ci.yml file, attach runners on my pc, laptops and
| go.
|
| Deploy with scp/sftp.
|
| Also it's very useful to have DEV.md file for each repo. It helps
| to remember how to build and deploy your project. Documentation
| for future me :)
| atom_arranger wrote:
| I had a K8s setup running on Google cloud, using their managed
| database service. This was kind of nice, but expensive, cost
| ~$100 a month, and I was using very low specs.
|
| I have multiple projects, none of them making money currently, so
| $100/month for each basic app is not good.
|
| I switched to using Dokku on Vultr, $12 a month. You can easily
| create Postgres databases and link them to Docker apps. I haven't
| bothered to setup CD yet but it looks like it should be simple,
| for now I just push to it when I want to deploy. Liking this
| setup so far.
|
| For frontend I use Netlify, I use their redirects to proxy the
| backend.
| b20000 wrote:
| this is why everytime i post here on HN that i want to make my
| toolset smaller, and i'm not interested in learning more tools. i
| catch a lot of flak, but i've been in this boat now for more than
| 15 years and I can say that if you need to manage everything with
| 1 or 2 people, you need to approach this differently than what a
| normal startup does especially if you are not able to raise
| funding.
| herbst wrote:
| I do it KISS. Deployment for me is always a git hook, usually
| with a private gitea server. So I have versioning and rollbacks
| available with a cute UI if necessary.
|
| DBs run side by side and are simply backed up regularly with
| whatever backup solution the VPS offers.
|
| That's it.
|
| Out of school I put way more effort in building my infrastructure
| but in reality a good VPS, something like Cloudflare and some app
| based caching can run millions of daily views over 20+ different
| apps without issues.
|
| Edit:// it's mostly rails apps I host btw. For my wordpress
| installs i use dedicated VPSes because those scale horrible.
| playpause wrote:
| I like this answer, but could you expand a bit on why gitea?
| Never used it myself, but it looks like self-hosted OSS clone
| of GitHub? If so, my instinct (in terms of KISS) would be to
| just use GitHub, so I don't have to set up and maintain my own
| thing. What made you decide to go with gitea, out of interest?
| herbst wrote:
| GitHub would be the opposite of kiss in my opinion. External
| dependencies and pushing to remote servers are just pain
| points.
|
| > git init --bare
|
| On your remote server is all you need to spawn a repo to push
| to, usually hook examples already included.
|
| The hook can then directly checkout to my live directory, run
| migrations and restart the server without weird remote
| automations after every (or specified) push.
|
| Gitea is just a little extra (setup once, touch never thingy)
| for quick edits and checks (it's a wonderful lightweight
| piece of software too)
| iKevinShah wrote:
| > Out of school I put way more effort in building my
| infrastructure but in reality a good VPS, something like
| Cloudflare and some app based caching can run millions of daily
| views over 20+ different apps without issues.
|
| I read such messages (and also agree to it mostly) and then I
| encounter my peers wanting active-active clusters with K8s on
| top or something and I feel I like a dinosaur.
|
| Refreshing to see KISS still being used. Also gitea. What an
| awesome piece of open source software.
| Sanguinaire wrote:
| Don't fall victim to resume-driven development; keep on
| KISSing.
| bullen wrote:
| I made my own HTTP app. server and JSON database on top of that.
|
| The server accepts .jars over HTTP with code (and files) so I can
| hotdeploy while developing on live on the entire cluster in real
| time. My turnaround is about 1 second.
|
| The JSON database allows for schema-less simplicity, and it has
| all the features you need like indexes, relations, multi crossref
| and security (and then some, like global realtime distributed
| while still being performant) in 2000 lines of code.
|
| I have zero pain developing the most scalable (and energy
| efficient) backend in the world, yet very few seem to care or use
| it: https://github.com/tinspin/rupy
|
| It has been proven on a real project with 5 years uptime and
| 350.000 users: https://store.steampowered.com/app/486310/Meadow/
| programd wrote:
| Does "Book of Travels" run on the same infrastructure?
|
| https://store.steampowered.com/app/1152340/Book_of_Travels/
|
| I know it had some server issues on launch - too popular it
| seems.
| bullen wrote:
| No, nor I or fuse had any involvement with BoT. I wouldn't
| post here if I was...
|
| BoT did not fail because it had too many players, you can
| clearly see how many are playing, maybe it has a complex
| backend... simplicity is your friend!
|
| Meadow for reference launched with ~150 (on T2 Micro at 30%)
| concurrent players per machine and peaked at 450 (on T2
| Medium at 25%). That's before you could buy more credits so
| we had to upgrade to M4 Large after a while.
|
| And Meadow is an action game, not a point and click!
|
| The rupy chain is solid and proven and has been running
| Meadow for 5 years without any write downtime except last
| week when one instance got nuked by AWS or somebody at M&D.
| That's a pretty good track record of 100% read uptime!
|
| The Meadow launch probably was the only MMO to have a
| flawless launch with high pressure per server so far, and it
| did so over HTTP with a JSON file database! :D
|
| It's also the only MMO with one database spanning 3
| continents in real-time.
| Aulig wrote:
| I'm a noob when it comes to devops, so you might not want to do
| it my way.
|
| I keep it super simple. I have a git repo where I directly commit
| into master. When I want to take the changes live I ssh onto my
| VPS, I git pull the changes and restart the website service.
|
| I use an SQLite DB that I automatically copy (backup) to my local
| machine daily via the Windows equivalent of a cron job. Once it's
| on my local machine it's also backed up to other places as part
| of my local backup process.
|
| I run my tests manually when I think I've made significant
| changes.
|
| Otherwise I just react to errors ASAP (I get an email
| notification from sentry when a new error occurs)
|
| A lot of this doesnt work with a bigger team of course.
|
| It's a balance between risk and time investment. The simpler your
| dev ops stuff, the higher your risk of introducing bugs into
| production. But it saves you a lot of time not dealing with that
| stuff.
| moltar wrote:
| AWS CDK, and use the library constructs and solutions they
| provide. It's like LEGO blocks.
| brightball wrote:
| You avoid it as much as possible. When your time is that
| valuable, lean on PaaS options like Heroku, Fly.io, Gigalixir,
| etc.
|
| If it's just you, you can afford to use a simple shell script to
| run tests locally before deploying or use a
| Travis/Circle/Cloudbees CI option too.
|
| You have enough to focus on with the application itself.
| throw03172019 wrote:
| I use Convox. It's great and has been stable the last 5 years for
| us at many different scale levels.
| malwrar wrote:
| Not a solo founder, but I do a lot of ops professionally. Unless
| you have a specific reason, I say it's best to avoid complicated
| tooling until later.
|
| k8s, ci, etc are all really useful and solve a lot of hard
| problems, but god are they a bitch to set up and then monitor,
| fix when something weird happens, secure from nasty people
| outside, patch when some update becomes necessary, resist the
| urge to "improve", etc. This doesn't include the time it takes to
| work these tools into your project, like k8s for example really
| requires you grok its model of how stuff it runs is supposed to
| look. There's a reason entire companies exist that simply offer
| managed versions of these services!
|
| If you're using fewer than like 10 boxes and don't work with many
| people, you can get away with spooky arcane oldhat sysadmin
| deploy techniques (scp release to prod hosts, run deploy script,
| service or cron to keep running) and honestly it's pretty fun to
| just run a command and poof you're deployed. Then you can just
| wait until it starts making sense to use fancier tooling (more
| load, more sites/DCs, more services, more people/teams, etc).
| Monitoring is always good to start with, for other stuff a good
| litmus test is that price points for hosted k8s & ops stuff will
| start to make sense, you'll be paying in time in any case. Who
| knows, maybe when that happens something new will pop up that
| makes even more sense. Until then, reducing the amount of stuff
| you need to worry about is always a good strategy.
| noir_lord wrote:
| Agree with all of this.
|
| If you want an intermediate step you can just make docker
| images of your application and deploy and run those manually
| (again with shell scripts).
|
| You can get a long way that way without having to deal with
| kubernetes.
| NotAnOtter wrote:
| Yeesh
|
| I cannot imagine not setting up at least a basic CI. Even if
| you remove all unit/integration test alarms, a CI is still very
| useful. The 20 minutes of build -> upload -> deploy script ->
| check prod vs a 1/2 day to set up a CI and forget it.
|
| I'm not sure you save much time but you save a lot of headache
| emidln wrote:
| I think you mean automation is very useful. I've worked in a
| lot of places where we had a single `make deploy` that could
| compile, test, upload the artifacts, deploy the artifacts to
| hosts, start the new services, and slowly transition the load
| balancer to point at the new instances while automatically
| backing out if error metrics started lighting up. This is
| just automation, it doesn't have to happen on every commit.
| Especially when working solo or on a small team, I might not
| want to maintain a lot of VCS branches and instead, just
| deploy when I feel it's ready. You have to do all of that
| stuff anyway, so if you later decide to hook it up to a VCS
| trigger, you're all set.
| NotAnOtter wrote:
| I guess that's fair, I'm probably coming into this having
| only seen it done a certain way and blind to other
| possibilities.
|
| That being said, it would take a lot to convince me a
| standard CI/CD pipeline is not worth the investment. My
| only experience with non CI/CD deployment process was a
| _very_ manual process at a startup, then when I saw my
| first CI /CD it felt like magic.
| oauea wrote:
| > If you're using fewer than like 10 boxes and don't work with
| many people, you can get away with spooky arcane oldhat
| sysadmin deploy techniques (scp release to prod hosts, run
| deploy script, service or cron to keep running)
|
| If you have multiple people on the team that have access to
| servers, this will shoot you in the foot a year or two later.
| Good luck replicating that server setup when no one has any
| clue whatsoever about all the small tweaks that were performed
| ad-hoc over the years.
| martijnvds wrote:
| Next step up is something like Ansible to manage it all.
| oauea wrote:
| Yes. Then you realize that people may still SSH into your
| servers and manually tweak things when they need the fix
| "now". So then you start restricting SSH access and forcing
| everything through Ansible. And now you're deep in
| deployment best practices again ;)
| malwrar wrote:
| I think you're describing the growing pains that justify
| paying the costs of more complicated tooling. The big
| difference is that instead of needing to figure it out as
| a solo founder with bigger problems, in this scenario you
| get to figure it out as a solo founder with a team and
| presumably some extra $$.
| davedx wrote:
| > spooky arcane oldhat sysadmin deploy techniques (scp release
| to prod hosts, run deploy script
|
| I never realised I was using spooky arcane oldhat stuff! I feel
| wizardly now.
|
| My projects (for small clients and myself) basically use this.
|
| - A "build.sh" script that does a local build of back end and
| front end
|
| - A "deploy.sh" script that scp's everything to the server
| (either a digital ocean VPS or an EC2 instance), runs npm
| install, runs database migrations and restarts with pm2
|
| So running my entire CI pipeline in terminal is: ./build.sh &&
| ./deploy.sh
| codegeek wrote:
| rsync and scp are two of my top favorite commands.
| noduerme wrote:
| Basically same here. In cases the deployment has to be zipped
| and manually deployed to elastic beanstalk. On some EC2
| instances for projects running a PHP stack, I don't even
| bother with this, just SFTP script changes to the server
| straight out of my IDE. Take care of any rollback needs with
| local Git versioning.
| dangerface wrote:
| This is a great setup if you work with a lot of clients.
| Every client we work with wants everything to run on their
| infrastructure that they provide, and then they provide the
| weirdest jankiest setup you could imagine. Because of this I
| keep everything basic if it cant run on a fresh linux install
| or less (looking at you docker) then its going to cause
| issues down the line.
| downrightmike wrote:
| You're a wizard davedx!
| codazoda wrote:
| My favorite deploy is using a simple `git pull` instead of
| scp. I also avoid complex build tools when I can. You either
| need your built files in the repo or you need to avoid them.
| Either are fine for my small personal projects. The only real
| exception I make is keeping any necessary secrets out of my
| repo. Those get dropped on the server manually.
|
| This also solves the roll-back problem mentioned elsewhere.
| Just checkout the previous version (as long as you don't make
| any side effects like DB updates that are incompatible).
|
| I admit, this requires relatively simple software.
| sbayeta wrote:
| I do the same thing (just a couple os customers, o en
| Linode per customer). Hopefully I'll need something more
| complex eventually.
| OJFord wrote:
| Why pull, as opposed to push(ing to the server as remote)?
|
| Then you could have any necessary restart or migration or
| whatever run in a post-receive hook.
|
| I realise that's starting to get back in to tooling and
| config etc., but it's not that complex, and if you're
| already using git anyway...
| bootwoot wrote:
| This is great. Until you need to roll back.
| bkanber wrote:
| git checkout abcd1234 ./build.sh && ./deploy.sh
|
| I don't see the issue.
| dsgrillo wrote:
| If you have any migration, you probably want to rollback
| them as well
| mrweasel wrote:
| That's sort of pet peeve of mine: Migration are done
| separate from code deploys. Version 1 of your code runs
| on schema version 1. Schema version 2 does not make
| chances that will break code version 1. Code version 2
| can utilize the chances made in schema version 2, but
| you're still able to rollback the code.
|
| Each schema migration should also come with its own
| rollback script.
|
| The downside is that you might need three migrations for
| some operation, but at least you won't break stuff.
|
| The assumption that you can do a schema migration while
| deploying new code is only valid when you have very
| limited database sizes. I've seen Flyway migrations break
| so many times, because developers assumed it was fine to
| just do complicated migrations on a 200GB database. Or a
| Django database migration just lock up everything for
| hours because no one cared to think about the difference
| between migrating 100MB and 100GB. And I've never seen
| anyone seriously considering rolling back a Flyway
| migration.
| spfzero wrote:
| Agree with this and have practiced and advocated for it.
| Make the schema changes to support the new feature first,
| then verify existing software still works. Deploy the
| schema change. Then develop the new feature, test, and
| deploy the program. That way you can deploy and rollback
| without needing to synchronously run a bunch of migration
| routines.
| qw wrote:
| I had a similar setup a couple of years ago, where I had to
| deploy without downtime. My solution was to simply have the
| old version running in parallel until I was certain the new
| version was ok
|
| ----------
|
| _Static website:_
|
| 1. Setup NginX using a symlink to the current version
|
| 2. Copy the new files to a separate folder
|
| 3. Change the symlink to point to the new version
|
| ----------
|
| _REST service (behind NgninX reverse proxy):_
|
| 1. Current version runs on port X.
|
| 2. Deploy new version and run it on port Y.
|
| 3. Update NginX config to use port Y and reload
|
| 4. If you need rollback, just reverse step 3.
|
| This can be done using scripts or Ansible too if necessary.
| marcosdumay wrote:
| Why? Checkout what version you want to roll back to, and
| deploy it.
| brightball wrote:
| Keep a simple tool like Ansible around for those spooky
| admin tricks and you can take advantage of the Ansistrano
| plugins for smooth deploy and rollback (Ruby's great
| Capistrano tool ported to Ansible).
|
| https://ansistrano.com/
|
| It's pretty fantastic.
| 1MachineElf wrote:
| This looks awesome. I'm currently in the middle of
| learning Ansible now for my FjeeBSD jails. Have any other
| plugins to recommend?
| brightball wrote:
| Anything by geerlingguy on Ansible Galaxy.
| lenova wrote:
| > Anything by geerlingguy on Ansible Galaxy.
|
| I've been writing some Ansible playbooks recently for the
| first time in years, and came upon geerlingguy's work.
| That guy is a powerhouse when it comes to writing Ansible
| roles/modules!
| thujlife wrote:
| rollback.sh
| gglitch wrote:
| Naive question: what's the correct way of doing this when you
| need to cp your content to a dir like /var/www/html that your
| user doesn't own (when logging in as root is prohibited)? My
| "spooky" (and probably very stupid) method is an Expect
| script that lets me supply a password to sudo.
| nightfly wrote:
| Ansible can do this (you tell it the user to initially log
| in, and that you need to "become" another user for a task)
| quells wrote:
| Usually these directories are owned by another group (www
| or www-data) so that your web server program (nginx,
| apache, etc) can access them without running as root. If
| you add your user to that group, you should be able to
| manage files in your web root without sudo. Be careful with
| permissions, though - you may need to chown to set the
| group after copying so that the files are readable by your
| web server.
| darkr wrote:
| Ideally you want some atomic deployment strategy, rather
| than having to deal with whatever crap might have been in
| the directory previously. Something like - extract your
| deployment artefact to a new, unique directory - and then
| read-only bind-mount that directory as /var/www/html.
|
| > and probably very stupid) method is an Expect script that
| lets me supply a password to sudo.
|
| The old school sysadmin way of doing this would be to have
| a dedicated deployment user (ssh pubkey auth only -
| possibly restricting only specific commands), with a
| sudoers[.d] configuration to allow that user to run an
| explicit group commands without a password (NOPASSWD)
| alfonsodev wrote:
| a pattern I have seen before is to clone the branch commit
| of each release to a folder /var/releases/{release_commit}
| and then html is not a folder but a soft link so
| /var/www/html -> /var/releases/latest_release_commit this
| is useful if you need to revert back quickly. But then
| releases folder needs to be cleaned up, or you can run out
| of space.
| Sohcahtoa82 wrote:
| That's dangerous unless you've properly configured your
| web server to block access to your .git directory.
| handrous wrote:
| Safer/easier to keep the .git somewhere else entirely,
| rather than in the working directory. Git's got arguments
| that let you do that. Then you can check out to a dir
| that doesn't have the .git dir in it at all.
|
| Alternatively, it's pretty easy to use git-archive
| (optionally, with an exclusions list to keep out files
| you don't want to serve publicly) to do something similar
| without ever cloning the remote repo. You can fetch a zip
| or tar.gz with a given commit's files, straight from a
| remote repo.
| dangerface wrote:
| Login as the user that owns that dir, don't try to hack
| around the user permissions.
|
| Also you probably don't want your web under /var if its a
| partition and you fill it with lets say user content then
| you will lock up your server.
| leetrout wrote:
| This is the way
| Grimm665 wrote:
| If you like this style of deployment, and you haven't looked
| into Ansible yet, give it a try, you may enjoy it. I had
| similar build processes before using Ansible and I have found
| it a really positive step up.
| wefarrell wrote:
| At this point CI is pretty easy to setup and monitor and it's
| integrated with practically every development/deployment tool,
| so I have to disagree about waiting to implement it.
| JTbane wrote:
| >you can get away with spooky arcane oldhat sysadmin deploy
| techniques (scp release to prod hosts, run deploy script,
| service or cron to keep running)
|
| You would be surprised at how many old enterprises still do
| this. Sure, it's ugly, but it can be simpler than k8s, and
| anyone with scripting chops can understand it.
| nanis wrote:
| Indeed, for a bunch of hobby things, I use
| rsync -rvzC --no-p --no-g --chmod=ugo=rwX --stats -e "ssh .
| ${remote}:${path}"
|
| where `${remote}` is defined in `.ssh/config` and `${path}` is
| in the project config to move files over to a new deployment
| directory. Then, a quick command over SSH changes the symlink
| to the site's configuration file and restarts the web server.
|
| The sites are either on Linode or SSDNodes. Has been working
| for me for decades and don't need to change it for these hobby
| things. After all, the log files prove that these things get
| more traffic from exploitation probes than real people ;-)
| NortySpock wrote:
| Also a hobbyist sysadmin at home, but after reading a few
| comments on the internet, I found that the rsync -z compress
| flag was bottlenecked on a maxed-out CPU thread of my
| Raspberry Pi 4 NAS transfers.
|
| Admittedly, these are mostly "local" drive-to-drive transfers
| over USB rather than server-to-server transfers over house-
| wide gigabit. But consider trying the transfer without
| compression.
|
| I'll definitely have to read up on your other flags though,
| if your rsync works from just one machine rather than both
| that might solve a minor problem case I have where the
| destination machine lacks rsync...
|
| https://unix.stackexchange.com/questions/188737/does-
| compres...
| nanis wrote:
| Really good observation. All the settings are from a long
| time ago when link quality was way worse than what we get
| these days.
|
| I should review those options, definitely.
|
| > -C, --cvs-exclude auto-ignore files in the same way CVS
| does
|
| Just goes to show you how long I've been using this stuff
| :-)
|
| The additional options may not be relevant now that I am
| just deploying creating a new target directory each time.
| dbbk wrote:
| Or just deploy it on Heroku and call it a day?
| clairity wrote:
| yah, this (or something similar) is the most straightforward
| solution. the whole point of heroku is to abstract away
| devops for solo/small dev groups and make deployments "one-
| click". it's unclear why that option was explicitly ruled out
| here.
| marstall wrote:
| I'm fully on team spooky admin - however on my latest project,
| I've been loving being on the free tier of tools like vercel,
| which deploys from git commit, and firebase, which is mostly
| maintenance free. That really reduces the amount of devops in
| my day.
| p0nce wrote:
| I have no DB, static website, deploying in production on a VPS
| with git (manually) and rsync + Cloudflare. I just don't know
| what kubernetes is, nor do I encounter any of the problems you
| mention.
|
| > Just properly setting up and maintaining a small k8s cluster
| looks almost like a fulltime job.
|
| ok.
| e2e4 wrote:
| Caprover on a decent VPS like Hetzner.
|
| p.s. previously did quite a bit of AWS, Google, firebase; but the
| above had been working the best for about a year
| eatonphil wrote:
| I'm building an open-core data IDE that runs as a desktop
| application or server. Since it's open source, all tests run on
| Github Actions for free. This includes basic e2e testing using
| Selenium on Windows, macOS, and Linux (e.g. [0]); and
| unit/integration tests.
|
| If it were a private repo I'd still shell out for Github Actions
| or Circle CI most likely. I'd also consider buying a chunky-
| enough minipc for ~$500 and an older mac mini and set up runners
| on them.
|
| For the moment private runners isn't a problem. But soon I'll
| need to start integration-testing proprietary code paths like
| querying Oracle or MS SQL Server. In that case I probably need to
| set up a dedicated box with all the right licenses so I can run
| CI jobs on it.
|
| [0]
| https://github.com/multiprocessio/datastation/blob/master/.g...
| ArtWomb wrote:
| Choose a platform like App Engine. It's auto-scaling. Supports
| bigtable. And opentelemetry. Manageable from anywhere via Cloud
| Shell. You can run multiple instances and partition load between
| them. Even includes a free tier ;)
|
| https://github.com/GoogleCloudPlatform/golang-samples
| BiteCode_dev wrote:
| You did that, as a solo dev with little money, and made a
| profitable business out of it?
| ArtWomb wrote:
| not quite yet ;)
|
| ps some of the google cloud next announcements today around
| appsheet, appscript, looker, ai vertex notebooks, and "no-
| code" solutions are starting to look really sophisticated!
| scottydelta wrote:
| Since all the people in this thread do some kind of personal
| hosting/DevOps, I am posting one of my recent comments on how to
| make the security of your infrastructure rock solid:
|
| > I self-host three VPNs to protect my infrastructure heavy
| startup https://quantale.io
|
| I am a big fan of Pritunl which is opensource and provides
| network security with a lot of ease. I am in no way affiliated
| with them, I am just a big fan of Pritunl. I use Pritunl to limit
| access to servers and web applications for my different teams.
| For each user, you can generate a profile and assign the servers
| and port they have access to on the server. For eg:
|
| - Only dev team can access ssh port(22) on stage server and not
| open to internet.
|
| - Any one in the team can access stage version(port 443) for
| testing purpose.(Not open to internet)
|
| - Only I can access all ports on all Prod servers(only 443 open
| to public)
|
| What hackers can't see, they can't attack. Especially the port 22
| on your servers should only be accessible to you and not the
| internet.
|
| I self-host one instance each of OpenVPN and Wireguard with Pi-
| Hole which is then used to access my Pritunl Server adding extra
| layer of security.
|
| Each of these 3 servers can be hosted on Hetzner $2/month
| instance. With a mere $6, you can add an extra layer of security
| to your infrastructure. Pritunl itself also provides subscription
| so that is also an option.
|
| https://news.ycombinator.com/item?id=28671753
|
| If you want to discuss more about this or security in general,
| feel free to reach out to me on my email on profile.
| davidkuennen wrote:
| For the backend of my app[1] I'm mainly using Google Cloud:
|
| * Database: CloudSQL/PostgresQL (with Query Plan for analysis)
|
| * Services: Kubernetes (Services in Go)
|
| * CI: Cloud Build
|
| It's pretty easy to maintain.
|
| [1] https://stockevents.app
| terrortrain wrote:
| Some if the comments here are way over the top IMO. Like a full
| time DevOps person went solo and spent a few weeks setting up
| their perfect deployment process.
|
| If we're talking a plain saas type deal, I'd keep it simple,
| elastic bean stalk, or use a heroku or render.com like setup
| until you grow to the point of hiring a team. If it's just a
| basic saas, I don't see how a 1 man team could really out grow
| this setup. I've seen 100 person teams using heroku.
|
| K8s is just way too much work. Even cloud formation is to much
| for my tiny show.
|
| Use the automated backups setup by your host for your db. If you
| need to roll back, just redeploy your previous git hash. I
| typically use GitHub actions to deploy, so rolling back is just a
| matter of force pushing the prod branch to the sha I want to
| deploy
|
| Skip micro services, they are too much work for a small time
| thing, and don't really provide much benefit.
| nickspicer1993 wrote:
| My entire stack is in a docker-compose.yml that looks like this:
|
| services: certbot haproxy
| db api client
|
| And gets deployed by a GitLab config file that looks like this:
|
| build: stage: build script:
| - docker-compose -f docker-compose.yml -f docker-
| compose.release.yml up -d --remove-orphans --build
| soneca wrote:
| I only have side-projects, very different from a full-time,
| revenue earning solo-preneur, but I only use serverless. My
| preferred stack now is NextJS, hosted on Vercel with FaunaDB. The
| main reason is to "outsource" devops to them.
|
| I am not sure how it would work on a larger scale, but for my use
| is perfect.
| [deleted]
| jeffbarg wrote:
| We use https://porter.run. Pretty great.
| nkotov wrote:
| I had a chance to setup several businesses in the past and to
| setup the foundation - I ended up writing a lot of templates for
| myself. Nowadays, if I want to setup a SaaS company, I have 90%
| of the foundational groundwork covered (AWS, Terraform, CI/CD,
| Github, etc.)
|
| We're building [1] something similar to solve this problem but
| just on the infrastructure end. I really believe in the power of
| templates. In our future release, we're planning on launching a
| "infrastructure" catalog of commonly used setups so that you can
| just get the foundation in place. Why re-invent the wheel every
| time when you can just spin up a boilerplate that gets you 90% of
| the way there?
|
| [1] https://atomizedhq.com
| wenbin wrote:
| Solo founder here. (Typed on my phone)
|
| I've been running my small company for 4 years so far. It's a
| podcast search engine & api.
|
| Product & infra & Devop & all kinds of manual processes all
| evolve over the past 4 years. Things were added / improved on
| demand.
|
| 4 years ago: 3 digital ocean instances + ssh to manually deploy
| code. No users. So no devops.
|
| Now:
|
| - ~20 EC2 instances, provisioned via ansible
|
| - to deploy code: run a bash script on my MacBook, which is
| actually to run ansible. The script can also rollback to any
| specific git sha
|
| - use rollbar + Datadog + PagerDuty for monitoring & alerting
|
| - 3 Postgres database EC2 instances (1 master, 2 slaves). Daily
| cron job to automatically upload db dump to aws s3 and google
| cloud storage
|
| - no CI. No docker / kubernatte
|
| - a few interesting devop events are sent to slack, so I'm aware
| of what's going on for the entire system on my phone. Eg, when a
| db dump was successfully uploaded to s3, when a Django request is
| slower than 5 seconds, when an important cron job fails...
|
| Not very sophisticated, right? But in practice, we rarely have
| outages and spend very little time to do devops.
|
| The most annoying thing was to upgrade Postgres to a new version.
| We had to prepare for an entire week and minimize downtime (in
| practice, disable db write for < 1 minute or so).
|
| I've got a couple old blog posts on engineering for solo founder:
|
| - https://www.listennotes.com/blog/the-boring-technology-behin...
|
| - https://www.listennotes.com/blog/good-enough-engineering-to-...
| marvinblum wrote:
| I found the HashiStack to be manageable by a single person or
| very small teams. It's easy to set up and hasn't let us down so
| far. We get away with using shell scripts to test and build our
| containers for now. The trickiest part is scaling as soon as it
| becomes necessary. Our current three-node cluster handles a few
| hundred requests a second without issues, but I migrated the
| database three times already, and I'm in the progress of
| migrating it a forth time to a much bigger machine.
|
| You can find more about our setup and ops here:
| https://pirsch.io/blog/techstack/
| jbrun wrote:
| We use an external firm. Works great.
| bdcravens wrote:
| ECS gives you pretty much everything K8S gives you, in a fully
| managed package. Obviously you can't (simply) run ECS in
| development, but sticking with Docker Compose keeps the semantics
| close enough, as well as giving you advantage of clean separation
| from any other code on your machine.
|
| CI/Deployment: hosted CI, merged PRs to main branch auto-deploy
| (I've used both Codeship and Github Actions with ECS)
|
| DB: managed RDS
| frankwiles wrote:
| Keep in mind you can run EKS in aws to get K8S in a fully
| managed package as well.
| bdcravens wrote:
| My understanding is that with EKS, you still are responsible
| for parts of the process that are managed under ECS (for
| example, firewall and security rules as opposed to leaning on
| ALB and Target Groups in ECS)
| strzibny wrote:
| I strip it down to a bare minimum.
|
| Some people probably don't realize that you can have a full-
| featured git-push deployment including a PostgreSQL database and
| Redis on few hundred lines of simple Bash code. The only
| exception to keeping it bare is security (proper SSL, SELinux,
| etc.).
|
| - 1 cheap VM per project (I prefer Digital Ocean at this time) +
| snapshots
|
| - No CI (run tests locally), no staging until later
|
| - Little bit of Bash to configure everything, no IaC
|
| - Simple systemd services (+ maybe systemd socket activation)
|
| - git-push deploy... bad release? repush previous version
|
| - Automatic system updates and log rotation
|
| - Few auxiliary scripts for Rails console (./railsc.sh), backups
| (./backup.sh), etc.
|
| - External error and performance monitoring, but keeping raw logs
| on server (rarely need them)
|
| - Stable CentOS/Rocky Linux with long time support, rootless
| access
|
| I teach all of that in my book https://deploymentfromscratch.com/
| and I basically run my oldest side-project on the book demo which
| is similar to this. Some people might not like Bash, but it's
| surprisingly refreshing, and I keep my script flexible and
| idempotent:
|
| Set everything up after providing IP address and domain name in
| settings.sh: $ ./setup.sh
|
| Change the database configuration after changing the config file:
| $ ./setup.sh -u postgresql
|
| Deploy a new version: $ git push production master:master
|
| Later on I would separate the database and have staging, but it's
| overkill for my projects right now.
|
| The funny thing is that apart from having CI and staging we could
| have the same setup at 2 early stage startups I worked on...and
| it would probably last 3-5 years with maybe an in-place server
| spec upgrade, no kidding. People really like to overdo operations
| and you know where overdoing it leads to? Mistakes. You maintain
| K8s and then forget to do something basic security-wise that
| almost kill the company (seen).
| cblconfederate wrote:
| So none of you edits the code directly in the production machine?
| namenotrequired wrote:
| Unpopular answer alert, but it worked well for my context for the
| few years before I hired a team.
|
| I didn't do CI/CD. I just reviewed my own code the next day,
| improved it and put it live. As the only dev I knew more or less
| how my code worked and what risk factors to prepare for in each
| deploy, and serious, business-impacting production issues were
| rare.
|
| A simple stack of Meteor on Galaxy + Mongo on Atlas meant all my
| tools integrated well together and a whole lot of devops things
| were easy enough that I rarely had to think about them after they
| were set up once.
|
| I can think of many cases where this would not work but it worked
| well for my context.
| welder wrote:
| I'm a solo founder with a product used by 300k+ people. Keep it
| simple, don't solve problems before they become problems, and
| keep all config in Git. I would use Heroku, but my usage
| requirements makes that cost-prohibitive. Heroku is good for low-
| traffic projects aka Enterprise B2B. I don't use k8s, I just use
| shell scripts that import each other for setting up new
| DigitalOcean servers. I separate my shell scripts into 3
| categories:
|
| 1. common (Google leap second smearing, increase max open
| connections, misc security and performance changes)
|
| 2. infra (install Nginx, install RabbitMQ, install Postgres,
| install SSDB)
|
| 3. product (run common stuff, then infra stuff, then any stuff
| specific to a single server)
|
| Sometimes it breaks after upgrading Linux, but it's easy to fix.
| For CI, I use GitHub Actions. Deployments are mostly built with
| GH Actions. If deployment fails it just keeps the old server
| code, there is no need for rollbacks I just re-deploy with
| working code. If deployment passes but there's still a prod bug,
| I don't rollback. I just commit a revert and deploy that or
| commit a fix. LetsEncrypt renewal, log aggregation, db backups is
| all custom with combination of RabbitMQ and CRON + Shell scripts.
|
| I miss out on autoscaling, so someday I'll invest the time to
| migrate web servers to hosted k8s. With limited time as a solo-
| founder I have to prioritize for impact meaning incremental
| improvements usually never get done.
|
| Not having k8s and autoscaling isn't causing any problems, won't
| increase revenue, and won't save any meaningful expenses so it's
| currently only an incremental improvement for me.
|
| Finally, I have a small Slack community for bootstrapped
| founders. Let me know if you want to chat in there about
| specifics.
| kingkool68 wrote:
| Bash scripts!
| JanSt wrote:
| Digital Ocean App platform is doing most of my work and getting
| better by the day
| joshghent wrote:
| As others have said, try to avoid any kind of DevOps if you can.
| But here's my strategy:
|
| Monitoring - Loads of cron jobs pinging slack (monitoring disk,
| CPU, Network I/O etc), Healthchecks.io, TurboAPI, Nginx Amplify.
| Mostly built up over time
|
| DB - Done entirely via migrations. Most of the time they are
| small migrations so I can do it by running the command directly
| on the instance
|
| Deployments - GitHub actions to build a docker image (there was a
| prebuilt template). Watchtower then runs on my server to pick up
| the new image and that's then deployed automatically. There is no
| rollback system as I've not needed that so far (and 99% of the
| time you won't).
|
| Backups - DB backup is done via a cron pushing to AWS S3 Glacier.
| The app code is stored on GitHub already so no need to back that
| up. Happy to share the script if you like :)
| oneplane wrote:
| "It depends"...
|
| For simple monoliths it's a non-CI method plus scp, for
| infrastructure-only it's SaltStack and Terraform. For something
| where the DevOps part is the deliverable, it can go wild pretty
| fast: GitLab CI with various CI steps (static analysis, building,
| unit tests, integration tests, browser tests, blob storage of the
| result), then CD steps depend on what it is that was built;
| docker images just get pushed to a registry and another async job
| listens for pushes and does the deploy in Fargate or Kubernetes.
| For some jobs with larger needs you get into ArgoCD and
| Kubernetes resource deployments that way.
|
| Essentially it depends on the zoom level:
|
| Zoomed in to the max: just a build.sh or makefile, optionally
| called by a CI or IDE of your choice so the building is exactly
| the same no matter what invoked it.
|
| Zoomed out a little: the results of the build need to end up
| somewhere, sometimes a step in between like a package registry,
| docker image registry etc.
|
| Zoomed out to level 3: once the deployment gets actually done,
| the runtime itself is responsible for checking for database
| migrations, locks, maintenance mode, so that has to be part of
| the startup. If it's a multi-instance workload it depends on
| database migrations if there has to be downtime.
|
| Zoomed out to level 4: if it also needs to deploy 'other things',
| like rules and records to Cloudflare, resources in AWS, resources
| in K8S and even update some stuff in buckets or DynamoDB, that
| requires some orchestration and DSL like Terraform, SaltStack,
| Ansible and application-specific CD like ArgoCD or Flux or even
| just the aws cli tools to trigger a refresh.
|
| Customer decides how much money they want to spend and that
| scopes what zoom level they get. Usually depends on how many
| developers or ops people they have themselves and how involved
| they want to be and what continuity or guarantees they are
| looking for. The only thing that is not optional is that
| everything has to be in Git. If a customer doesn't want that,
| they cannot be my customer. I'm done living in the 90's.
| swlkr wrote:
| My solo ops/data stack is:
|
| - docker
|
| - dokku
|
| - sqlite
|
| - litestream
| easytiger wrote:
| Don't use them. They require time to maintain and if you are old
| like me half of them make very little sense anyway.
| jon-wood wrote:
| I have no idea why there's nothing here saying "don't". Go and
| get a Heroku account, hook it up to your Github repository's
| master branch for deploys, use Github Actions for CI, and then
| get on with life.
|
| Yes, you'll pay more for a Heroku Postgres instance than you
| would for a VPS on Digital Ocean with Postgres running on it,
| likewise you'll pay more for Heroku Dynos than another VPS to run
| your application server. On the other side though, your backup,
| rollback, deploy, and scaling strategies can all be "Heroku does
| that", and you can focus on the much more valuable job of
| building a business.
| kirillzubovsky wrote:
| This is what I've been doing for years, but Heroku Postgres is
| so bloody expensive, it just doesn't make sense for apps that
| don't make much money. So I just moved my data to an RDS
| instance, shared between multiple apps. Cost savings are great,
| headache savings much less so. I would say if you are going to
| move off Heroku, do it with the apps/projects where you can
| f-up, or have downtime without much repercussion. AWS stuff is
| a deep dark forest, undocumented and confusing. I'd say for
| anything that is critical, just keep paying Heroku. It isn't
| worth it.
| Torwald wrote:
| Exactly. And when you are truly "beyond POC level and beyond
| heroku (for whatever reason)" then of course it is time to hire
| people.
|
| I see this all the time. Entrepreneurs trying to solve the
| people problem by avoidance. The problem with the super
| efficient player is twofold. First, it's not really scaleable.
| Second, what if the person gets sick or is in vacation?
| sebiw wrote:
| Heroku Postgres - after all those years - still is vulnerable
| to MITM attacks. You can't reliably verify the authenticity of
| the connection between Heroku's Dyno and the Postgres database.
|
| They use self-signed certificates on Heroku Postgres servers
| without a certificate authority present in the chain. No chance
| to establish reliable peer verification. A shame, I would love
| to use Heroku more often without compromising security.
| tootie wrote:
| I'd even go higher up the value chain with something like
| Vercel or Netlify if your stack fits in their offering. Heroku
| is a definitely a good option if you need more flexibility. The
| costs won't be a bottleneck until you actually have enough
| scale that you can afford DevOps staff.
| hyuuu wrote:
| Vercel has the best dev UX in my opinion, everything just
| "works" the way you expected it the first time, and if not
| their documentation is very straight forward.
| strzibny wrote:
| Until Heroku has a huge downtime (like recently) and you are
| f*ked. If you have your own VMs, you can quickly move them to a
| different provider, but it's hard to start figuring out servers
| when Heroku is down and your business stopped.
|
| That said, I would also recommend Heroku to many.
| jon-wood wrote:
| I can almost guarantee that over a period of years you'll see
| significantly less downtime that can be attributed to Heroku
| than you would on VMs that you manage. Even if that isn't the
| case, if Heroku are responsible for it then you can mostly
| just kick back and let them fix it, rather than spending 24
| hours without sleep migrating VMs between providers.
|
| The context here is also that of a single entrepreneur
| building a business. At that point in the lifecycle of a
| product downtime really isn't that big of a deal, customers
| tend to be pretty understanding so long as you recover.
| gwbas1c wrote:
| I really like Heroku, when I work on NodeJS projects.
|
| BUT: In the last year I've had some rather serious downtime
| issues _that were 100% Heroku 's fault_. (Poor encapsulation of
| Postgres configuration via the connection string, and they
| mistakenly claimed that I violated their TOS when I didn't.
| They apologized and fixed the situation.)
|
| I'd really like to see some good alternatives to Heroku. Not
| just because of their screwups, but because the languages that
| I want to work on have limited support in Heroku.
| mthwsjc_ wrote:
| which languages do you want to work on?
| orky56 wrote:
| Totally agree. There's some impostor syndrome in play here. If
| you don't need the scale, why waste your time managing a more
| complex setup? Heroku might be slightly more expensive in the
| short term but your time is better spent building/selling than
| future-proofing or just doing it because larger operations have
| to.
| bizzleDawg wrote:
| This! Plus it includes SSL provisioning, staging environments,
| background workers etc.
|
| Other competitors like render.com are starting to look like
| they'll be competitive on features and are way cheaper too.
| tmcw wrote:
| Yep! Used Heroku at a previous gig (now a mid-size startup),
| worked great, now on Render.com and it's also great. Also
| factor in fly.io and railway as quickly-growing platforms in
| the same PaaS sort of space.
| crcastle wrote:
| I head up Dev Advocacy at Render (previously at Heroku).
| Happy to answer any questions about Render.
| darkr wrote:
| +1. For pre-A series/pre-revenue/pre-1k paying customers and
| other early stage businesses you can do a lot worse than
| starting out on Heroku, especially if you don't have
| sysadmin/cloud/devops expertise (and even if you do, that time
| could/should be spent on much more important things at this
| stage).
| bingohbangoh wrote:
| Heroku is by far the best for this but just to throw in some
| alternatives: * Digital Ocean Apps -- somewhat
| finicky but works very similar to Heroku, it gets closer to
| bare metal so I prefer it * render.com * AWS
| Elastic Beanstalk -- though setting this up is non-trivial, it
| is very similar to Heroku with its "set it and forget it"
| * AWS Container Services -- if you're using Docker *
| Google App Engine * Supabase -- geared for SPAs but this
| may be the ticket depending on what you're building, is open
| source
|
| You may want to use your own VPS but spend for a managed
| database solution as this has the most potential for
| catastrophe.
| jon-wood wrote:
| I would advise against Elastic Beanstalk, last time I used it
| the product had the stink of being on life support - they've
| not really added any new features to it in several years, and
| even before that it was incredibly flakey.
| pirate787 wrote:
| I use Elastic Beanstalk daily it is maintained and supports
| everything on AWS, they've added ALB and WAP for example.
| It runs on Cloud Formation. Never experienced anything
| "flakey"
| dougbarrett wrote:
| What additional features are you looking for? I've been
| using beanstalk for the past few years, and just wrapped up
| doing a large deployment on it. I've never found beanstalk
| to be a bottleneck or not have a feature I was looking for.
| crummy wrote:
| Fly.io is a pretty nice alternative too.
| fulafel wrote:
| Also AWS Copilot if you like CLI based tooling, similar to
| Elastic Beanstalk but easier (at least if you're comfortable
| making a Dockerfile)
| bingohbangoh wrote:
| Was not aware of Copilot! Looks much better than Elastic
| Beanstalk
| rajacombinator wrote:
| The whole point of devops or any other devthing is efficiency. If
| doing some devops stuff will increase your efficiency then do it.
| But you probably just need to find someone who will pay for
| something.
| imedadel wrote:
| I use Dokku (https://dokku.com/). It works with GitHub Actions
| and it's enough for most use cases.
| iwd wrote:
| If the hours you spend implementing and maintaining your DevOps
| exceed the hours of downtime you prevent, you're probably not
| making good use of your time. The less you build, the less you
| have to maintain.
|
| I've been running a Python-based, highly custom web store solo
| since 2007, and it supports multiple people doing fulfillment. I
| host on Opalstack so as to outsourcing patching, email config,
| database maintainence, etc. I run directly in a Git repo
| (actually Hg, it's that old) and releases are "git pull &&
| ./restart.sh". Rollbacks are "git checkout ...".
|
| I've had to migrate/rebuild the VM about every 5 years. Tech
| changes enough in that time that no automation will still work
| unmodified. So I just keep good notes about what I did last time,
| and figure out what the new equivalents are when I finally have
| to do it again (updating the notes, of course). Database and
| Conda are easy to port. It's usually DNS and email integrations
| that are a pain.
|
| As others have said, KISS is key. Industry DevOps is for a work
| setting with a decent-sized team, where you can afford the
| overhead of maintaining it all in order to make the overall team
| more efficient.
| cgarethc wrote:
| I wrote a series of Medium stories about my experience with tech
| stacks for solo work - including deployment (start here:
| https://medium.com/geekculture/surely-building-software-cant...).
| My needs are slightly different to OP, because I'm not assuming
| everything will have to scale, so I'm optimising for "scale to
| zero" (only pay for what I use) and low maintenance... but I'm
| taking approaches that I know will scale if they have to (scale
| technically, not scale to have a team working on it).
|
| In short, I lean heavily on serverless and Github Actions.
|
| Each story covers a different pattern in my stack and I apply
| these principles:
|
| -I only want to pay for what I use -I don't have a lot of time
| available for learning or building -I don't have time for
| maintenance activities -I'm not a good UI designer or front end
| engineer
| [deleted]
| vikR0001 wrote:
| I'm a javascript developer. My build tool is Meteor and I host on
| Galaxy. Deployment is one line of code in the terminal. Scaling
| is handled automatically. :)
| kaino128 wrote:
| Pieter Levels (the guy who made Nomad List) reached $1 000 000
| ARR with a single VPS, all his code in an index.php file and
| using FTP to transfer the latest code to his server...
|
| I think if you're a solo founder you have to accept on day one
| you're at a time disadvantage when it comes to the actual hours
| you'll spend on moving the business forward vs admin, ops, legal
| stuff etc etc and so you have to really fight smart!
|
| I.e K8S? Definitely don't need it. Complex CI/Deployment
| workflow? Nope, just the minimal shell scripts to do clean
| deploys and rollbacks. Pay for a HA RDS so you never need to
| worry about DB replication etc.
|
| If you just work with the minimal set of tools you're comfortable
| with and relentlessly keep things simple it's possible. I've
| built high availability stuff as a solo founder and still found
| (at least some...) time to work on sales and the business.
| epolanski wrote:
| What is a HA RDS?
| neilk wrote:
| High Availability Relational Data Store
|
| An example would be AWS' RDS
| jamie_ca wrote:
| High-availability relational data store.
| qw wrote:
| I think you are right. There's a big difference between the
| technology I would use in a larger organization compared to
| solo projects.
|
| Unless you build a side project to learn a new technology, you
| should use something "boring" that you know well.
|
| Here's my personal choices in preferred order:
|
| 1. Static web sites - Just upload to S3 behind a CDN and forget
| about it
|
| 2. Server rendered website - PHP. It may not be something you
| brag about, but in my experience it just "works" and has
| excellent uptime.
|
| Note: If I had to recommend something to a larger organization,
| I would probably suggest Next.js to attract developers. PHP is
| often associated with "old school" tech
|
| 3. REST service - Spring Boot. I have so much experience with
| it and there are so many Spring projects that supports a huge
| range of technologies from monitoring to data
|
| 4. Scripts - Python
| alberth wrote:
| This!
|
| Exactly my point that I was making here in this thread:
| https://news.ycombinator.com/item?id=28840902
| jermaustin1 wrote:
| So I am a solo-preneur and have launched lots of web-apps in my
| career. And the DevOps hasn't really changed for me is incredibly
| simple: commit source control and copy the files to the server.
|
| I've done this since writing my first CGI app in 2003. And it
| hasn't changed other than the commands I call. Back then it was
| manual through GUIs (visual source safe and FTP) and now it is
| git and scp, but 80% of the process hasn't changed in almost 2
| decades.
|
| If it breaks (which it shouldn't, but inevitably does), I just
| git checkout the previous commit and scp again, and try and fix
| the issue.
|
| If I have a "high availability" product (which I hate doing), I
| will release onto a clone of the production server, then test it,
| swap to that clone, deploy to the others, swap back and kill the
| clone. It is a much more involved process that I really don't
| like doing, so I've stopped making high availability
| applications.
|
| But that is web, so it's super simple. Also to avoid issues
| regarding data schemas, I always try to make sure my changes are
| additive. And on startup, my web server checks the database
| version then runs any upgrade scripts if needed. If the deploy
| fails, the db changes stay, and the code just gets updated. This
| is sloppy, but it's simple.
|
| When a real schema change needs to happen, I abandon the project
| (just kidding). I usually do it in multiple updates. New table
| with the schema changes, upgrade script transfers the data, and
| code to point to new table. Manually verify that all of the data
| copied. Backup the old table and drop it. Then another update to
| rename the new table to the old name (if needed) and it's done
| with minimal disruption in service.
| austinpena wrote:
| For my side project I try to keep it as simple as possible.
|
| After I run tests locally, I've got a makefile that deploys to
| either Google Cloud Run or fly.io.
|
| Database is Cloud SQL which works well enough for me.
| tomcooks wrote:
| Whenever I do anything on a VPS I religiously write down each
| step in a README file, so that then I can create bash scripts and
| automate processes.
|
| The more I do this, the more I learn, the more I automate, the
| less I worry.
| sentinel wrote:
| My rule of thumb when working on side projects is to keep it
| simple:
|
| - use tech-stacks that I know
|
| - even simpler - use node.js / express (i.e. use only one
| programming language)
|
| - Standard AWS EC2, S3, Postgres instances
|
| - scale vertically before scaling horizontally
|
| - use GitHub Actions as much as possible
|
| - deploys and rollbacks are done manually
|
| - I keep a list of useful CLI commands in Notion
|
| If you're a small team you shouldn't need to think about
| kubernetes tbh.
| Amasuriel wrote:
| I recently started a small project on Azure for something new,
| but most of my experience is small companies on AWS which is
| similar.
|
| If you aren't already comfortable with K8 I wouldn't use them,
| there is a learning curve there for sure and not necessary.
|
| I wouldn't even worry about containers at all unless you are
| planning on using a managed container service like
| elasticbeanstalk.
|
| My CI pipeline is almost nothing, just run tests on main and dev
| commits. I have no deploy pipeline, I choose to actively deploy.
|
| My deployment workflow is a powershell script since I'm on Azure,
| if it was AWS and I was using plain EC2 instead of a container
| service I would probably use ansible to avoid having to repeat
| commands 2-3 times when I deploy something behind a load
| balancer, but it would be super minimal.
|
| DB backups are super easy to schedule with RDS or similar
| offerings. For rollback with database changes I would schedule
| downtime, let your customers know, replace your page with a
| maintenance banner, do upgrade, test upgrade then repoint site to
| your running instance.
|
| Overall I echo what many have said, use familiar tools as much as
| you can, don't worry about whether what you are doing will scale
| to even 2-3 people if it costs significant time. Use managed
| services where possible to save you time, usually the cost isn't
| high vs doing it yourself.
| wjd2030 wrote:
| Devops is a culture, not a verb. Success generally being measured
| by 4 key metrics:
|
| Lead time to change Deployment Frequency Mean time to restore
| (MTTR) Change Failure Rate.
|
| Though it often is, I do not believe it should be used
| interchangeably to refer to a suite of tools designed to enable
| CI/CD and SDLC Lifecycles.
| DethNinja wrote:
| I just use Ansible, Docker, Jenkins, Gitea, NewRelic, rclone, and
| some custom build services with GoLang for DB backups and
| maintenance, that's it.
|
| I'm fairly sure I can maintain up to 50 servers as solo developer
| with the above mentioned tools.
| deepsun wrote:
| I use complex setups of K8S (on Google Cloud, it's simpler
| there), proper security, terraform etc, because I already did it
| before, and it as easy for me to just reuse my configs/scripts.
| bithavoc wrote:
| I'm afraid after Heroku is not enough for you, if you don't use
| Kubernetes, you may end up inventing one.
|
| If you need a bit more than Heroku:
|
| 1. Try Render[0].
|
| 2. If it is not enough, go with any Kubernetes you can afford,
| any docker registry and Pulumi or Pulumi+Helm[1]. For monitoring
| you can use either NewRelic[2] or Weave Cloud[3].
|
| 3. If you find big cloud kubernetes too expensive, need more
| bandwidth, or are just willing to try more affordable alternative
| new players, maybe Civo[4] fits the need.
|
| In my case[5], since I've been bootstrapping for three years, a
| low-budget setup is key to keeping the lights on. Here are a few
| constraints I need to satisfy permanently-ish:
|
| - My workload is bandwidth-intensive, so EC2 egress is too
| expensive for my case and my situation.
|
| - It is not that cheap to run 12 micro-services in any of the
| existing Heroku-ish platforms, also private networking and
| routing of my app are a bit above "12 factor" at this point.
|
| - I can't afford the same availability as everyone else: I'm in
| the business of supporting other apps but I don't want to pass
| down the cost of AWS, so I can't afford to be "down because
| everyone else in AWS is down".
|
| - I don't know many things, and until I find PMF, keeping a low-
| cost setup gives me the resilience that I need as a founder.
|
| Therefore to solve my problem I use an unholy combination of
| Ansible and Pulumi to manage 3 clusters of Kubernetes for both
| stateful and stateless workloads.
|
| Why am I doing this to myself? I have mad respect for people who
| can run things in a VPS, use the hosting daily backup, and don't
| see the need to apply patches because they're behind Cloudflare;
| I just can't.
|
| More details:
|
| - Dedicated servers, 32GB RAM, 8 Cores, 500GB SSD(RAID 1), /29
| ipv4, 30TB of Bandwidth for $70/mo each. XCP-NG[6] as hypervisor.
|
| - To assign public IP to virtual machines, I run a tiny DHCP
| ubuntu VM managed by Ansible.
|
| - On top of XCP virtual machines, k3s cluster managed with
| Ansible.
|
| - To manage XCP and virtual machines, I use Xen-Orchestra[7]
| managed with Ansible.
|
| - Pulumi+AWS ECR builds and pushes docker images, secret and
| configuration management using paraphrase secret provider.
|
| - Hashicorp Vault backed by Postgres+AWS KMS managed by Ansible.
|
| - Ansible+Pulumi+Helm to self-host Apache Pulsar with ZFS
| snapshots, a cron-job and rsync.net alerts. WAL-G+AWS S3 to
| backup and archive Postgres/TimescaleDB managed with Ansible.
|
| - All logs aggregated by Gelf and Fluentbit into a self-hosted
| instance of Seq[8] managed with Pulumi+Helm.
|
| - Tailscale between all virtual machines
|
| - DNSimple for domains
|
| - Cloudflare for all things except the self-hosted docker
| registry.
|
| - Github Actions for CI but not a hard a requirement, I can run
| ops in my local computer when Github Actions is down.
|
| Cost: $210 /mo.
|
| Estimated cost in AWS: $2,800 /mo
|
| [0] https://render.com
|
| [1] https://newrelic.com/platform
|
| [2] https://www.weave.works/product/cloud/
|
| [3]
| https://www.pulumi.com/docs/reference/pkg/kubernetes/helm/v3...
|
| [4] https://www.civo.com/pricing
|
| [5] https://widemesh.com
|
| [6] https://xcp-ng.org
|
| [7] https://xen-orchestra.com
|
| [8] https://datalust.co/seq
| dangerard wrote:
| I replicated Anthony Simon's architecture [0] in a few days and
| it has worked beautifully so far with a toy project [1]. Probably
| overkill for OP, but the combination of GH Actions & Flux [2]
| means that starting (and killing) namespaced projects - with all
| the cluster benefits of automatic TLS certification, DNS record
| management, rate limiting, pod restarts, logging, autoscaling,
| CD, etc. - is a single push to a git repository. Really nice if
| you're taking a shotgun approach and trying different ideas
| quickly.
|
| [0] https://anthonynsimon.com/blog/one-man-saas-architecture/
|
| [1] https://www.wyzetickets.com/
|
| [2] https://fluxcd.io/
| blablablub wrote:
| For a couple of persons team it is important to keep the changes
| small, so errors can easily identified. That means daily or bi-
| daily upgrades. It also helps if you focus on a certain small
| project and finish that, before moving on to the next project.
|
| SVN checkout development, meld into testing. Copy production db
| into testing. Run a db upgrade script in testing. Test the new
| function for a day. If all ok stop the production HAproxy at 3am.
| run production DB backup, Checkout testing into the production
| server. mount the new dir into the NGINX path. Upgrade+start the
| DB, start production HAproxy.
|
| Except for the meld and testing part, this is all automated.
|
| A rollback on the app servers is pretty simple, just remount to
| the previous location. Even though there is a backup of the DB, a
| rollback there is not really feasible.
|
| Everything is running in LXC containers, which I treat as cattle.
| Creating new ones is either automatic based on
| time/demand/failover or a few clicks in a custom web interface.
| HAproxy automatically pics them all up via DNS.
|
| SVN and production DB are always backed up nightly and copied to
| a S3 compatible storage. Everything older than 6 months gets
| deleted. All the free space on the servers is globbed up with
| glusterfs to give a decent amount of fast,free storage.
|
| Using Bash as scripting language, there is literary no problem in
| devops which hasn't been solved with bash and the solution is
| somewhere in the internet.
| [deleted]
| throwaway2016a wrote:
| Don't:
|
| - Manage your own DBs for production. I used to manage my own but
| now I use IaaS for it. It's worth the extra cost even if your
| budget is low since it lowers your risk and lets you move faster.
| AWS Aurora is awesome but expensive. There's a SaaS/IaaS out
| there for most DBs. And they handle the backups!
|
| - Use Kubernetes. k8s is awesome, I love it, but it is a beast
| and probably overkill for your solo ops team that is also
| multitasking.
|
| - Manually SSH / rsync / scp directly onto prod.
|
| Do:
|
| - Version control (Git) everything
|
| - Use containers where it makes sense (I use Docker when I'm not
| using Serverless). It keeps your dev and prod in sync and helps
| with immutable deployments (see later points)
|
| - Use Infrastructure as Code
|
| - Use Github actions to do your deploys
|
| - Use immutable deployments. More time to setup but makes
| rollbacks easier and if you never hot patch production you can't
| create a unique state that can't be restored.
|
| Finally, (and possibly controversially): use serverless where you
| can (I personally use AWS Lambda, S3, and Cloudfront).
|
| My personal flow:
|
| - All my infrastructure is in a parameterized CloudFormation
| template.
|
| - The template is checked into Git with the service code.
|
| - After CI runs the tests, CD builds my serverless functions and
| uploads the zip.
|
| - Then it runs the cloudformation template.
|
| The same flow also works for Docker except "uploads the zip"
| becomes "pushes the container"
|
| Using this setup I can tear down and build up my setup in
| minutes, making it easy (and cheaper) to separate dev and
| production.
| sebiw wrote:
| > Using this setup I can tear down and build up my setup in
| minutes [...]
|
| Until you decide to move away from AWS. ;-)
|
| Nevermind, ignoring the potential lock-in there are many pros
| to this setup.
| throwaway2016a wrote:
| I've been using 10+ years and have definitely considered that
| a lot over the years. It's a valid point. I used to be
| concerned with that, but now I think that is not a big issue
| for two reasons:
|
| 1. Vendor lockin when you are a startup should be a low
| priority in my opinion. Your priority should be moving fast
| and getting customers.
|
| 2. To expand on 1. It's not really vendor lockin, the only
| things vendor specific are your infrastructure templates and
| interface code. Rewriting the interface for another cloud
| provider takes a fraction of the time of rewriting your
| codebase (source: I've done it many times).
|
| To come full circle to the question, one thing you should
| probably not do as a solo-prenuer is concern yourself pre-
| maturely with vendor lockin at the expense of getting a
| working product.
|
| Edit/Example: The code for interfacing with AWS Aurora is the
| same as any other MySQL or PostgreSQL code except for
| authentication. Switching to a non-IaaS database WILL take
| time but no more time than rolling your own. So your choices
| are not "lock in" or "freedom".., it's, spend lots of time
| setting up databases now or later when you have proven your
| business model (and potentially never)
|
| Edit 2: This is an anecdote certainly but... of the dozens of
| startups I've worked on and with, some of which have grown
| pretty large. The only vendor lock-ins I've ever found to be
| issues have been with software frameworks (especially ERPs).
| Never infrastructure. I used to be worried about
| infrastructure lockin, now I prioritize product market fit.
| dsgrillo wrote:
| > Don't: manage your own DBs for production. [...]
|
| Why, though?
|
| If you're really on a low budget, the extra $$ for a managed DB
| doesn't really pay for itself. I'd rather prefer spinning up a
| container running whatever DB I need and run daily backups to
| an external vendor.
|
| In my case, the website is free, living on ads revenue, so
| every penny counts.
| throwaway2016a wrote:
| Good question and I have an answer for you. It depends on
| your risk tolerance and resiliency.
|
| If you compare it to a single machine with multiple tenants
| and no replication, it doesn't make sense. But that comes
| with risks and given the OPs question in the context of a k8s
| cluster (which for HA is a minimum of 5 servers), I figured
| they are prioritizing time savings over money.
|
| Losing customer data, downtime, and data breaches can kill
| some companies before they even take off.
|
| If your particular business is not sensitive to that than you
| can run your own database. Or better yet, if you're a content
| business you may not even need a database at all.
|
| But even so, if you run your own DB you need to worry about
| backups and such so it may even still be worth the money. And
| if you are going for High Availability (HA), running a
| properly replicated cluster can be time consuming.
|
| The cost of a t3.medium RDS instance on AWS for instance is
| $0.068 / hour compared to a t3.medium instance on EC2 is
| $0.0416 per hour. That is an extra $19 per month. For my use
| case, paying an extra $19 a month is well worth the savings
| in time and the piece of mind I have knowing my backups are
| running and if the server crashes I can recover quickly. But
| you must consider use cases and your's might be different.
| dsgrillo wrote:
| You bring up good points, fair enough.
|
| I might being naive here, still: - losing
| customer data: the backup must be done frequently enough.
| So I'm not risking losing data. - downtime: I'd
| expect that RDS could still go south as well, maybe less
| frequently, but still. In my case I start another VM, run
| again the container and apply the backup. - data
| breaches: Of course one can misconfigure something here,
| but not sure how it'd be different using a managed
| database.
|
| Regarding the cost, I know that there are companies
| spending hundreds of $$ on their infra, but it just baffles
| me for most use cases. I still feel that one can get really
| far with just a budget of 50$/month.
| throwaway2016a wrote:
| True. You have downtime risk with RDS and managed
| databases as well. I have taken down RDS instances with a
| bad query and indexes once or twice.
|
| $50 total would be tough with fully managed AND high
| availability. I agree. But to digress from the tech side
| a bit and talk my own experiences, the question I ask
| myself is: as a solo-prenuer can I use that time to be
| doing something that will make me more than $X per month?
| If the answer is "yes, immediately!" I go with the
| managed server. If the answer is, "it would take me years
| to break even"... I may roll my own.
|
| To address your points, thought: I think managed
| databases do help here. Data breaches and outages are
| usually configuration issues. Since they start with a
| good known configuration, managed services do help. Also,
| I see not being able to SSH in as a positive in this
| scenario.
|
| For example: Say you misconfigured a database an break
| backups (happens all the time even to good DBs). That is
| much harder to do on a managed database.
|
| If you're managing just one machine and being down for
| 10-30 minutes while your backup is being restored is
| fine, you probably don't gain much by using a managed
| service. In that case I'd take frequent backups, upload
| them immediately off the server (probably S3), and
| containerize the database. In my architectures I always
| run at least 3 servers.
|
| With most database engines, if you have three servers you
| need to lose 2 to cause an outage. But a 3 server cluster
| is much harder to manage manually, tilting the scales
| more towards a managed database.
|
| Also, to your point, would blow $50 out of the water.
|
| If I can, I use a managed serverless database like
| DynamoDB (there is also others) that are pay as you go.
| You can easily run a service on just DynamoDB free tier
| if your use case is well suited to that kind of key/value
| stored based DB.
|
| You can also, always start with your own and switch to
| managed later as you add machines to the cluster.
|
| Just, whichever you choose... but "especially" when
| rolling your own... make sure you run fire drills. I.e
| test your backup and restore process regularly.
| Fiahil wrote:
| Because everyone here is hell bent on "spooky arcane oldhat
| sysadmin deploy techniques", I'll share my setup using k8s.
|
| - CI on github actions
|
| - Project management on post-its (short) and READMEs (long term)
|
| - deployment is done by a github action trigger on the `main`
| branch
|
| - Hosting is using AKS, GKE, or on-prem k3s on a raspberry pi.
| Want to restart a service ? just kill the pods.
|
| - Devops took about 2 days of work initially and now shared by
| every project for less than 30 minutes by project.
|
| - Deloying a test cluster (or a test k3s node) is reproducible on
| whatever hardware / cloud provider you get, sometimes I often
| create a full-blown dummy cluster for running test scenario
|
| - Certificate renewal is automatic (cert-manager)
|
| - DNS record creation to services/ingress is automatic
| (ExternalDNS)
|
| - Authentication using OAuth/OIDC is also set up automatically on
| whatever identity provider you have
|
| - Database backup is automatic
|
| - Load Balancing is built-in
|
| - Job scheduling is built-in (and -fuck- logs are much more
| accessible on a failed job container than in a custom cron + sh
| solution)
|
| - Service discovery is not needed
|
| - Monitoring and alerting is not, but cloud providers often have
| something for you
|
| Note: this is highly effective because I already had significant
| K8s experience, so, if you're still learning about what Ingress-
| Controller to choose for your OIDC proxy, then don't go that
| route.
| davidgl wrote:
| > Note: this is highly effective because I already had
| significant K8s experience, so, if you're still learning about
| what Ingress-Controller to choose for your OIDC proxy, then
| don't go that route.
|
| What would you recommend instead ?
| techn00 wrote:
| How did you set up the DNS part?
| Fiahil wrote:
| Via annotations on my ingresses, and, of course, with this :
| https://github.com/kubernetes-sigs/external-dns
| SergeAx wrote:
| First of all: no microsrvices, no Kubernetes. The whole business
| logic is a two binary (server + cli) monolith in a single
| container image.
|
| For CI GitLab is a bliss. Lots of examples and articles around
| the web. I made a test-build-dockerize-tag pipeline once and now
| just taking it from one project to another with minimal changes.
|
| For deploy - Docker Compose is enough. Maybe Docker Swarm some
| time in the future. Gitlab hosts docker images, rollback is a
| simple change of container version.
|
| For Postgres backups I am using WAL archive + daily snapshots
| from replica.
| piettes wrote:
| I might be biased here, as I started as a side project and wanted
| to learn k8s in the process. It turned out quite funny to set up
| and I am happy with the result, except for the struggle I had to
| add SSL. I have 1 frontend, 4 micro-services and 2 postgres. One
| cluster for test, one for production.
|
| I use Gitlab for CI, commits to master create a new docker image
| in the Gitlab registry. I manually trigger deployment on k8s with
| `kubectl rollout restart XXX`
| q-base wrote:
| Sounds like you are overthinking and/or over engineering it.
| Unless CI or deploy/rollback functionality IS your product then
| it does not matter. Until you have got traction for your actual
| product then deployments can have 100 manual steps that you can
| write down and follow - it does not matter.
|
| I have no idea what you are building or planning it to be but for
| my projects I have a single VPS with Hetzner per project running
| linux and hosting the applications. Deployment is done by a
| powershell script that copies files from my laptop to the server
| - that's it.
|
| If I were to think about clusters, cloud instances, load
| balancers and what not then I would be scared away before having
| delivered anything of value. Cross that bridge when you get
| there.
| cpursley wrote:
| That's easy, Render.com (it's like if Heroku were designed in
| 2021, and a lot cheaper). And can handle docker and distributed
| Elixir out of the box as well as free static sites). Took me just
| a few hours at most to set up and has been very reliable (I think
| it's k8s on Google Cloud under the hood but it's all abstracted
| away).
| bhouston wrote:
| Vercel, Porter, Heruko is sort of the simplest way to avoid
| DevOps. Use their integrated services for data and storage until
| you get large enough to afford a DevOps team.
| marcrosoft wrote:
| Servicing over 1MM API requests / week using API gateway and
| lambda. It's been set and forget for 6 years or so. AWS lambda is
| incredibly maintenance free if your use case fits the lambda
| constraints.
| bredren wrote:
| Solo founder, sold my last SaaS I had set up using "spooky arcane
| oldhat sysadmin deploy techniques" h/t to @Fiahil for that shade.
|
| On that it was a Django product with deploy scripts for staging
| and production, i.e. `./deploy_production` ran:
| #!/bin/bash ssh web@[production ip] << 'ENDSSH' cd
| /home/web/ ./pg.sh ENDSSH
|
| Where `pg.sh` pulled from `master` ran static and migration
| management commands and restarted the web server.
|
| Last summer I moved away from the spookies and now do all
| deployment using Github Actions.
|
| Commits to either a feature branch or `develop` run tests and all
| necessary deployment steps to a staging instance. (my stuff so
| far does not require k8s) and all containers are put up using
| docker compose. Commits to `main` (or `master` depending on the
| project)
|
| I blended a few guides written by Michael Herman to build my GA-
| based CI, and it took several weeks to work on the scripts and
| learn Github Actions. But it was time absolutely worth investing
| because my DevOps is modern enough now.
|
| No more bare metal stuff. No more scripts. Nice visual display of
| deployment success, "free" VMs for the deployment and CI UI
| integrated alongside my issues and code all on Github.
|
| Importantly, I still use git flow. I tried moving to trunk based,
| but the macros inherint in git flow (including the pycharm
| plugin) make this production / staging // master / devops or
| feature/ work very well (I'm not making PRs to myself)
| more_corn wrote:
| In what specific ways are you beyond Heroku? I call bullshit on
| that.
|
| I've got a big client who went from 6 infrastructure engineers to
| zero. They survived for half a year without any real problem. You
| either pay a company to manage your infrastructure for you or you
| pay a person. Devops people are expensive.
|
| Things I've done to automate DevOps Make sure you pay attention
| to automatic testing and deploys with Ci/CD. I like automated
| deploys on Gitlab. Trigger staging release on merge into main,
| and deploy prod on merge from main into a deploy branch.
| Infrastructure as code with Terraform. K8S is more trouble than
| it is worth (though it is robust after you invest 3 months into
| getting it tuned right, although a lot of people make critical
| mistakes in their config, I've seen a half dozen insane
| mistakes.)
| dyeje wrote:
| I would never switch from a PaaS unless I was expanding the team
| beyond just myself. Working on DevOps brings 0 value to my users.
| tbronchain wrote:
| I have worked mainly on ops/infra professionally the past few
| years and will strongly with the idea to avoid any complicated
| tool - i.e Kubernetes nowadays.
|
| I create my small/bootstrap projects the following way:
|
| 1- create a free tier AWS account - can work with any VPS/server
| really, but with AWS you can get it 100% free :)
|
| 2- create some Ansible provision/setup/deploy scripts
|
| 3- create some bash scripts to wrap this all
|
| Create the Ansible scripts:
|
| 1- Provision script, consisting of 1 EC2 instance, 1 RDS and the
| security groups. Store the IPs/adresses of the newly created
| instances.
|
| 2- Setup script, basic instance config, packages, app/project
| repo, you name it
|
| 3- Deploy script, to run every time I want to update the app,
| mainly just doing a git pull on the instance and some database
| backup/migrations (although on RDS backups aren't always
| necessary)
|
| I get these Ansible scripts wrapped into basic bash scripts,
| provide my AWS credentials via the AWS cli to keep it safe, few
| extra creds with Ansible vaults, SSH keys, publish in a Github
| [private] repo and I'm all set.
|
| It took me a couple of days to get fully operational and I was
| learning Ansible at the same time so it can really be done with
| basic features. Now it's done, I can reuse the same skeleton for
| every new project in a couple of hours!
|
| I find this solution extremely resilient. That's basically free
| hosting for personal projects. Every year, just need to run the
| script again when free-tier expire, and can change hosting
| provider anytime or upgrade the instance type if I decide to get
| a project publicly released. A small extra Ansible task to write
| for migrating the data and that's it!
| kureikain wrote:
| I divided my infrastructure into two part.
|
| 1. Tricky and unfrequent changes: - mail server, load balancer,
| database etc: for this I just write bootstrap.sh, rsync to server
| and run. - It's literally just a for server in `foor bar`; scp ;
| ssh ` done
|
| 2. Deployment/Frequent changes: - docker-compose to spin up
| everything. It's super nice. Again, the deployment is done with a
| `rsync` then `docker-compose up -f docker-compose-prod.yml`
|
| Eventually when deployment changes very frequent and need
| scale/ha I added in Kubernetes. K8S is way easiser to setup than
| you think and it handle all other suff(load balancer, environment
| variable etc).
|
| And my deploy now become: `kubectl apply -f`
|
| One trick I used is to use `sed` or `envsubst` to replace the
| image hash.
|
| For backedup, I again, literally setup cronjob from an external
| server, `ssh` into database and run `pgdump`.
|
| I also have a nice NFS server to centralize config and sync back
| to our git repo.
|
| I used this whole setup to operate https://hanami.run an email
| forwarding service for the first 3 months before I added
| Kubernetes.
| kukabynd wrote:
| I use Dokku and Caprover, Git hooks and two branches (staging,
| master) which I think is more than enough for a solopreneur.
| Dockerized apps are easy to reason about and fix in case of
| fires.
| scottydelta wrote:
| I have been working on https://quantale.io since a year and half.
| It's a web based alternative to Bloomberg Terminal.
|
| I personally manage the CI/CD infra as well as the data
| infrastructure for my startup.
|
| Here is my setup:
|
| 1. Self-hosted Drone.io instance connected to github(the instance
| is behind a firewall and can only be accessed via a self-hosted
| VPN. Github IPs are whitelisted so that auto build/deployment can
| be initiated via github hooks)
|
| I have drone.io config that helps me manange the versioning,
| deployment to stage/production, run tests, do rollbacks etc.
|
| for eg: merging to master on github auto builds the master branch
| on drone and deploys to staging along with pulling the latest db
| backup from S3 and adding to stage for testing
|
| 2. Self-hosted production and staging server that hosts my
| web/django app. It has a local postgres db and is encrypted and
| backed up on S3 every hour.
|
| 3. Self-hosted Elasticsearch cluster using Ansible playbooks with
| daily snapshot backups to S3.
|
| beyond this, I self-host 3 VPNs to protect my entire
| infrastructure. read my old HN comment about it:
| https://news.ycombinator.com/item?id=28671753
|
| All this is one time infrastructure setup and it all has been
| running smoothly for more than a year without any hiccups.
|
| I would be happy to help you setup a similar setup if you want.
| hit me up at vikash@quantale.io
| iKevinShah wrote:
| > 2. Self-hosted production and staging server that hosts my
| web/django app. It has a local postgres db and is encrypted and
| backed up on S3 every hour.
|
| Did you have any instance where you needed to restore the DB /
| Have you done restoration tests to ensure this is a feasible
| solution? Also, since you are using local postgres DB, do you
| not run into issues like transaction id wraparound ?
|
| Also, how many days of backup you keep on S3? Considering you
| create 24 backups a day.
| scottydelta wrote:
| > Did you have any instance where you needed to restore the
| DB / Have you done restoration tests to ensure this is a
| feasible solution?
|
| Whenever a build is deployed to stage server, it
| automatically pulls the latest version backup in last 1 hour
| and restores it on the stage for testing with the latest
| data. I have config to do the same restore to the prod if
| needed.
|
| I have been using the same setup for over 2 years to support
| an internal web app of a company.
|
| > Also, since you are using local postgres DB, do you not run
| into issues like transaction id wraparound ?
|
| This problem only arises when there is heavy load on the
| database, specifically high WRITE/UPDATE/DELETE operations.
| Quantale uses postgres to store only user info and all the
| other data is stored in Elasticsearch(twitter, reddit, news,
| SEC, Financial data on over 15000+ stocks). This is
| intentional because with the scalable server resources, my
| postgres setup can handle a large number of users before I
| see that problem. I can always scale up later when needed
| rather than over-engineering it right now.
|
| >Also, how many days of backup you keep on S3? Considering
| you create 24 backups a day.
|
| I have 7 days backups on S3 with versioning switched on for
| 60 days. So at anytime I can access the version backed up in
| last 60 days.
| iKevinShah wrote:
| scottydelta, Thank you for taking the time to give me more
| context / insight.
|
| Safe to say you have a verified way of backup and restore
| (with worst case data loss of an 1 hour).
|
| I am not exactly an expert on what stage does Transaction
| ID Wraparound comes in but still know that as one of the
| key criticisms of PostgreSQL so wanted to check.
|
| Also, 7 days backup with versioning switched is a smart
| tactic! Thanks once again for this.
| waspight wrote:
| I dont think that a hosted k8s with a hosted db on, say DO, would
| require a full time job? I can imagine it seems that way if one
| is coming from a large 1k+ employee company though. I always
| wonder how those companies can utilize so many people.
| vishnumohandas wrote:
| > CI
|
| GitHub Actions
|
| > deployments/rollbacks
|
| Docker. Scaleway offers a container registry that's ridiculously
| cheap[1]. Deployments are infrequent and executed manually.
|
| > DBs
|
| Again, Scaleway's managed RDS[2].
|
| Outside these, we have setup Grafana + Loki cloud[3] for
| monitoring and alerting. They have a generous free plan.
|
| For product analytics that can be derived from our database we've
| a self hosted instance of Metabase[4].
|
| [1]: https://www.scaleway.com/en/container-registry/
|
| [2]: https://www.scaleway.com/en/database/
|
| [3]: https://grafana.com/
|
| [4]: https://www.metabase.com/
|
| P.S. We were a 1 person company when most of this was setup.
| We're 3 now, works just as well.
| aristofun wrote:
| Thanks for being specific.
| no_circuit wrote:
| I decided to take a few years off work to just build on what I'd
| like. Perhaps in a startup studio model, so I have a bias for
| having something that is easily reusable, and that uses tech
| someone else can pick up and run with easily. I'll probably be in
| the business of dev/infra tooling.
|
| Currently going with a container image as the minimal deployable
| unit that gets put on top of a clean up to date OS. For me that's
| created with a Dockerfile using Alpine image variants. In a way I
| could see someone's rsync as an ok equivalent, but I'd do
| versioned symlinked directories so I can easily roll back if
| necessary if I went with this method. Something like update-
| alternatives or UIUC Encap/Epk:
| https://www.ks.uiuc.edu/Development/Computers/docs/sysadmin/....
| Anyone remember that? I guess the modern version of Epkg with
| dependencies these days is https://docs.brew.sh/Homebrew-on-
| Linux. :-) Or maybe Nixpkgs: https://github.com/NixOS/nixpkgs?
|
| Deployment-wise I've already done the Bash script writing thing
| to help a friend automate his deployment to EC2 instance. For
| myself I was going to start using boto3, but just went ahead and
| learned Terraform instead. So now my scripts are just simple
| wrappers for Docker/Terraform that build, push, or deploy that
| work with AWS ECS Fargate or DigitalOcean Kubernetes.
|
| No CI/CD yet. DBs/backups I'll tackle next as I want to make sure
| I can install or failover to a new datacenter without much
| difficulty.
| caffeine wrote:
| Use Linux, pretend you're in the 90s/2000s. If you can't afford a
| team, you're too small to need more.
|
| - Use NFS (EFS), put builds in a /releases folder
|
| - Bash scripts for build, deploy, release
|
| - Bash scripts to wrap apps (write pidfile, start, stop, etc)
|
| - Cron those scripts
|
| - Check crontabs into a repo along with all other config
|
| - Cron a bash script to pull from that repo and update crontabs
| on every box
|
| - Have a staging environment, deploy stuff there (esp DB
| migrations) and do smoke tests.
|
| - Have some kind of monitoring (I have a slackbot that sends me
| messages when things break)
|
| I general I follow the principle of proportional response, and
| n=2 (or 3 depending on how trivial the task is to automate).
|
| Proportional response means you invest in automation / tooling
| proportional to the pain you have suffered. Burning a day to
| build a bulletproof script only makes sense to solve a big
| problem. For a smaller issue, maybe adding a debug line or
| writing a line somewhere that you can copy-paste later is good
| enough.
|
| N=2 means you don't solve stuff until the second time you need
| it. This stops you from burning hours building clueless complex
| garbage to solve unnecessary problems - at least by the time you
| are automating, you have solved it once or twice by hand and you
| know where you are going.
|
| Elon's 5 principles have been very useful to me as a solo dev:
|
| 1. Make requirements less stupid
|
| 2. Delete the part/requirement
|
| 3. Simplify/Improve the design
|
| 4. Accelerate cycle time
|
| 5. Automate
|
| So in particular you might try 1-4 to see if you can magic
| problems away before you invest in building devops automation you
| will then need to maintain.
| alberth wrote:
| I'll get downvoted for this but use Dreamhost (if you're just
| starting out).
|
| Both their Shared and Dedicated servers are fully managed. You
| can run PHP, NodeJS, etc on it. They also fully manage MySQL too.
|
| All you have to do is drop your code onto their servers and it
| just runs. It's also hard to beat their pricing too.
| xrd wrote:
| GitLab and Dokku.
|
| You can run almost ANYTHING inside dokku. I use it to run a
| private/public docker registry, minio (amazon S3 clone),
| analytics via matomo, and a ton of mini apps that share a bunch
| of databases. It's incredible.
|
| With dokku, deploying your app is just "git push dokku main"
|
| GitLab plus CI is amazing. And, other than hosting, I don't pay a
| penny for it. If you self-host, you can put it behind a wireguard
| VPN. It's incredible. You can run as many runners as you want on
| your personal laptops, or on the cloud, and bring them up and
| down when needed.
| tnolet wrote:
| Simple. Don't use K8S. It's absolutely 100% the wrong way to go
| for solo devs with no product market fit.
|
| Use Heroku (or any other PaaS) and throw in some serverless stuff
| here and there.
|
| Again DO NOT geek out on the tech. Geek out on getting customers.
| ignas- wrote:
| This seems to (strangely) be an unpopular answer here, but I've
| spent a weekend to terraform everything. I run a couple of
| different projects which are yet to pick-up traction, so
| prototyping quickly is essential.
|
| What I have now is terraform on Github which handles Networking
| (VPC, subnets), ECS/ECR, ALB for my backend and S3/CloudFront for
| frontend.
|
| Everything is on AWS because I'm familiar with it, so it's
| faster. And I compensate for costs with AWS Activate + Free Tier.
| This combination is usually enough to understand whether the
| project will get traction or not.
|
| Actually spending little time on IaC was one of the best recent
| investments. But I guess it holds more value when you have a
| couple of projects, still exploring and these tasks get
| repetitive
| roland35 wrote:
| I find this helpful for my side project as well. It's amazing
| how quickly a) things break down and b) I forget how to set
| things up and get running again if I have to shelve and restart
| work on a project.
| aristofun wrote:
| I wonder why no comments about using docker containers directly
| or with Swarm
| rolisz wrote:
| For a recent small project, I tried out deta.sh. It takes out the
| pain of devops entirely, but it does constrain you on the
| architecture and database side of things.
| trhr wrote:
| I made the decision early on that a best practices CI/CD setup
| wasn't a time waster, but an integral part of building robust
| software.
| mibollma wrote:
| Having 90% of the system in a single Cloudformation template
| worked pretty well for me.
|
| On every stage (dev, test, prod) I can deploy it automatically up
| to two times (blue and green) by running a simple shell script.
| So when I update the application I just deploy it another time on
| prod, test it one last time by adding the IP of the load balancer
| in my local hosts file and if I'm satisfied with the result
| switching the DNS entry to the new version using a weighted DNS
| record with the ability to switch back until I shut down the old
| version.
|
| Doing anything continuous doesn't feel worth it in such a small
| setup with one update every 1-3 months.
|
| What I like most about the approach is that I'm free to change
| any aspect in the main template without any risk to break
| something in the live application. Only changes to the elements
| shared between versions need to be handled carefully.
|
| The main template includes VPC, network, fargate service,
| loadbalancer, firewall, KMS, DNS records, access rights, database
| tables, queues, monitoring metrics, email alerts
|
| It excludes everything that is shared from update to update which
| are defined separately and just referenced from the main template
| such as some database tables, persistent storage container
| registry, user groups (Only the groups, not the rights assigned
| to them)
| nullbytesmatter wrote:
| I run a solo SaaS product with 15,000+ users. I use the
| "arcane/wizardry" techniques still.
|
| I simply run "make deploy" and a new production build is made and
| rsync'd to production. Then the script executes a "deploy.sh" on
| the production server which reloads the system process to pick up
| the new binary.
|
| Poof.
| [deleted]
| [deleted]
| sealthedeal wrote:
| Just use Heroku until its time to scale
| [deleted]
| trhr wrote:
| I made the decision before starting that a best practices CI/CD
| setup wasn't a time waster, but an integral part of building
| robust software.
|
| Deployed into two off-lease dell servers; one for staging and one
| for production; via docker-compose.
| balabaster wrote:
| It depends very much what your tech stack looks like, but I use
| Git, CircleCI and Octopus Deploy to Azure for my stack. It's
| cheap in terms of maintenance and because I know them all like
| the back of my hand it's quick and easy for me to set up.
|
| As for rollback, I don't really use rollback. I have a roll-
| forward strategy. My DB deployments are migrations and so long as
| there's no data-loss caused by a migration (my migrations are
| always backwards compatible with the previous version) there's no
| need to roll back. Azure brings point-in-time restore which can
| be triggered if necessary.
|
| I think the key is like with anything. There's a learning curve
| on your toolset. Once you've overcome that hurdle and you know to
| set up automation for everything from the outset. You build your
| pipeline shell. You never do anything manually. If you need
| infrastructure of any form, it's always code-first, included in
| source control and done the right way. Never do anything manually
| and rely on your automation to carry you. It seems labour
| intensive up front, but once you're there, it's sustainable.
|
| It takes discipline not to cut corners. The minute you start to
| cut corners and get lazy is the path leading to your doom.
| decidertm wrote:
| I'm a technical co-founder at Northflank https://northflank.com.
| A platform to deploy microservices, cron jobs and stateful
| databases from development to production.
|
| I started working on Northflank to make it more simple for
| developers either solo or in a team to manage and automate DevOps
| complexity away.
|
| We support CI for GitHub, Bitbucket & Gitlab (SaaS and self-
| hosted) with either Dockerfiles or Buildpacks. We have a robust
| and feature rich platform via UI, API and CLI. Out of the box you
| get end-to-end DevOps: CI/CD/CD, Horizontal and vertical scaling,
| persistent workloads with managed (MongoDB, Redis, Postgres,
| MySQL, Minio), backups, restores, real-time observability and
| metrics, DNS, TLS, mTLS, Teams & RBAC and more...
|
| Let me know what you think of our offering + site!
|
| Platform: https://northflank.com
|
| Application Documentation:
| https://northflank.com/docs/v1/application/overview
|
| API Documentation:
| https://northflank.com/docs/v1/api/introduction
| aristofun wrote:
| From the first glance it's not clear how is it different from
| heroku and what exact pain points are you trying to solve
| sirodoht wrote:
| My web app is hosted on a server on Hetzner Cloud. I don't use
| Docker.
|
| For:
|
| * Database: PostgreSQL installed through apt in the same server:
| https://github.com/sirodoht/mataroa/blob/master/docs/server-...
|
| * Backups: MinIO-upload to an S3-compatible object storage:
| https://github.com/sirodoht/mataroa/blob/master/backup-datab...
|
| * CI: Github Actions + sr.ht builds:
| https://github.com/sirodoht/mataroa/blob/master/.github/work... +
| https://github.com/sirodoht/mataroa/blob/master/.build.yml
|
| * CD: (not exactly CD but...) ssh + git pull + uWSGI reload:
| https://github.com/sirodoht/mataroa/blob/master/deploy.sh
|
| * Rollbacks: git revert HEAD + ./deploy.sh
|
| * Architecture pattern: stick to the monolith; avoid to deploy
| another service at all costs; eg. we need to send multiple
| emails? not celery, that would mean hosting a redis/rabbitmq. We
| already have a database so let's use that. We can also use Django
| management commands and cron:
| https://github.com/sirodoht/mataroa/blob/5bb46e05524d99c346c... +
| https://github.com/sirodoht/mataroa/blob/master/main/managem...
| goodpoint wrote:
| For 1-person projects you just need VPSes, git, and dpkg-
| buildpackage.
|
| A single makefile command is enough to build .deb files, rsync
| and install them. Rollbacks are trivial.
|
| Put configuration files as well in git and in the .deb
|
| git tags give you a history of releases and also deployments.
| sharps_xp wrote:
| As a thought exercise, I thought about how I would architect a
| web application as a 1 man show with a relatively small amount of
| users.
|
| I tried to optimize for cost and very little devops at all. I
| still mull about it but so far, here is what my architecture
| would look like.
|
| - no servers to manage - static website hosted on CDN, and even
| dynamic user-specific pages would be built statically every 8
| minutes - all writes go to an SQS queue - every 8 minutes, a
| lambda is spun up to batch read the SQS queue and all writes go
| through a single writer process that writes to a SQLite file on
| EFS - every 8 minutes, a new static version of the site is built
| using the above hopefully updated SQLite file
|
| The one thing I hate about this is that I can't in good conscious
| say to the user that their write request succeeded b/c no state
| has actually changed on the backend yet. but I might be okay with
| this tradeoff and just simply showing only them their most recent
| write.
|
| The other tradeoff is when things go wrong, it'll probably go
| terribly wrong.
| theptip wrote:
| A lot of good answers here to the actual question in the OP, of
| "how do _you_ do devops ". And while it's good to look at other
| stacks for inspiration, particularly as a solo/small-team
| founder, the "correct choice" such as there is one, is in my
| opinion to do what you can do quickly/efficiently with your
| current skillset.
|
| I'd say the meta-level answer here should be to timebox your
| investment based on the tools you know, and an honest assessment
| of your uptime requirements. Also, know that what feels like a
| janky solution will usually get you much further than you think
| it will.
|
| For an initial PoC you should probably not be spending more than
| an hour or two setting up your infra, so Heroku or a single VM is
| probably the way to go. But you still _might_ want to start with
| k8s if it's the tool you know best; you can set up a k8s cluster
| in GKE with about as many clicks as it takes to set up a single
| node, and if you have a full "django k8s yaml" or equivalent in
| your toolkit then you could be up and running in an hour or two
| from scratch. I wouldn't recommend this to most folks though.
|
| Once you have some traction, beyond PoC stage, then you should be
| thinking about failure modes, their impact to your business, and
| how much time it's worth to spend to avoid them. If your users
| won't notice a few hours a week of downtime, then you don't need
| a multi-node setup like k8s or nodes behind a LB; a single node
| should suffice. If you have simple user data (like say comments)
| where in the worst-case losing everything since last night's
| backup would not sink your business, then running your DB with no
| standby replica and nightly backups might suffice. If you're
| storing financial transactions that's obviously not an option and
| your first DB with customer data needs replication (or however
| you want to get to RP0).
|
| Given all that, here are my suggestions for your specific
| questions:
|
| For CI/CD I'd say don't bother with CD, just set up Github/Gitlab
| to run your UTs on your PRs and on merges to master. Honestly you
| can just run UTs manually while building at PoC stage, but when
| you have paying customers I'd say you should take the 15 mins to
| wire up simple UTs. Getting a full Continuous Delivery pipeline
| set up won't save you much time vs. just manually deploying your
| artifacts, and at the beginning your tests are unlikely to be
| good enough to continuously deploy safely. At the point where you
| start having quality issues that your customers are complaining
| about, you might want to push for CD as a discipline thing; it
| forces you to write good-enough tests that you're happy to deploy
| if the tests are green. But it's not mandatory to deploy
| everything if your tests are good enough to do so.
|
| For rollback on your stateless instances, I like the pattern of
| having your single-node running your apps in a Docker container
| (I had one for Django and one for Nginx), that way the
| upgrade/downgrade is simply `docker pull <newimage>; docker stop
| <old image>; docker run <new image>`, and rollback is the
| inverse. But as long as you're just upgrading code and not OS
| dependencies, then you can do an equivalent operation by swapping
| out your code artifacts, e.g. checking out the new version of
| your python app & restarting your WSGI workers.
|
| For running your DB, I see a few folks suggesting running your DB
| locally on your node -- if you are an experienced DB operator
| that might be a good option, but if you're not, I would
| personally recommend using whatever managed SQL offering your
| cloud provider has; RDS or Cloud SQL will give you a durable
| instance with backups & easy restore with a few clicks in the UI,
| which is hard to beat in terms of ROI.
|
| Possibly-controversial recommendation: in general unless you've
| spent a lot of time working with Terraform/Ansible/etc. I would
| recommend against using infrastructure-as-code in the early days.
| It's hard to beat the ROI of a couple clicks in your cloud
| provider's UI to get your infrastructure set up, you might spend
| 10-100x more time getting your repeatable build implemented, even
| if it's just a couple mins vs. a couple hours. Clicking in the UI
| is not reproducible, but it will get you further than you think.
| As above though, if you're fluent in these tools and the
| difference in time-to-provision is very small, then absolutely do
| this now instead of later. But don't feel like you MUST do it
| "the right way" from the beginning.
| m0llusk wrote:
| Very carefully: Start with a very specific set of steps for
| security nodes and storing credentials. Then move on to backup
| and restore. All development and deployment details should have
| some kind of test to determine if the code or configuration are
| basically functional.
| c0nrad wrote:
| (Profitable Solo founder)
|
| Thankfully lots of tools do most of the heavy lifting. I use k8s,
| GKE does most of the work for me. It's very nice to have
| autoscaling for traffic spikes. Same with database (MongoDB
| Altas), dead simple autoscaling. I would never run my own k8s nor
| database.
|
| I wrote more details about some of the Ops stuff I do here in a
| previous similar question:
| https://news.ycombinator.com/item?id=26204402
|
| Coincidentally I also wrote some architecture notes about a new
| product last night: https://blog.c0nrad.io/posts/slack-latex/
|
| I think everyone's milage will very, but as general principles,
| staging is nice, reading docs saves time overall, tests help you
| sleep at night and make it easier to make changes 6 months in the
| future, simple health checks (or anything on a critical path)
| help you catch the real issues that need immediate attention.
|
| Good luck!
| 0xbadcafebee wrote:
| Hi OP and everyone else,
|
| DevOps is not a synonym for "operations work" or "server work" or
| "devs doing operations work".
|
| DevOps literally means "dev teams and ops teams working
| together". That's all. You don't "do DevOps" unless you are
| "doing the work of collaborating between two different teams of
| people".
| divan wrote:
| I try to use simpler/friendlier tools, so it can be managed with
| a simple SSH scripts for as long as possible. Use static binaries
| whenever possible. My typical stack is Go+gRPC+Flutter - so it's
| really easy to ship it as a single binary (for the web builds).
| For the DB I also started to use EdgeDB on some projects and it's
| such a relief in term of migrations.
| schappim wrote:
| I use https://www.hatchbox.io/ , like Heroku (git based
| workflow), but you can choose what infrastructure you run on.
|
| Hatchbox manages the deployment, backups, rollback etc.
| ensemblehq wrote:
| As a solo founder, your time is the most precious resource so all
| your DevOps effort should be minimized to the extent that you can
| prevent and debug issues. In practice, for me, this meant using
| tools that I'm already comfortable with (preferably in the same
| language as my codebase to minimize context switching) and
| resisting making things overly complicated with too many tools in
| the chain. For automated deployment, I'm using Fabric (for
| Python) that allows me to remotely run commands. DB backups are
| handled by cloud snapshots and rollbacks are ad-hoc based on Git.
|
| This was also how things began at a previous startup I worked at
| until we grew and we could hire more resources to fix things.
| chrisgoman wrote:
| using fabric for deployments (fabric2 kind of a pain but almost
| got most things ported over) even if our main app is in PHP
| [deleted]
| twobrainy wrote:
| I pay money so I can sleep easier at night!
|
| Envoyer and Forge for PHP stuff.
|
| Render/Heroku for node.js stuff. Netlify for static stuff.
|
| For the more complex bits I have, I use Pulumi to manage
| everything through IaC. It's still complex but at least it's
| robust and I can sense-chech changes to infra through a PR to
| myself.
| diiq wrote:
| Lots and lots of best practice exists specifically to help
| _teams_ , and especially teams with some normal amount of
| turnover.
|
| The problems of a solo dev are very different than a dev on a
| team. Knowledge silos don't exist. Distributed expertise doesn't
| exist. There's no one to mentor, no shared vision to maintain, no
| intertia to combat.
|
| I consult on big complicated team projects. I also manage
| multiple solo projects.
|
| On solo projects, deployment is a script I run from my dev
| machine. I'm the only person who deploys; anything else would be
| solving a problem I don't have.
|
| The only "CI" is running the tests before I run the deployment
| script. I'm the only one who needs to see the outcome of the
| tests. Anything more would be solving a problem I don't have.
|
| Architecture is whatever makes the most sense to me _personally_
| -- which is often _miles_ away from what I would recommend to a
| client, who needs something any new hire will recognize and be
| able to work with.
|
| I pay a service to manage backups so I can walk away from a solo
| project for months and know it's ticking away.
|
| The point is: solve problems you actually have. Don't try to run
| a "professional" operation by doing what big teams do all by
| yourself. Big teams have different problems.
| sjclemmy wrote:
| Totally agree with this. I do big corp work and the problem
| space is totally different from what is needed for a single
| developer.
| count wrote:
| 'knowledge silos' in a one person team are real: Me today and
| me from 2 years ago when I set a thing up are different people,
| and I don't remember that any more than a team mate in a multi-
| person team would know it. This is where 'professional'
| approaches can help considerably: using 'standardized' stuff,
| instead of bespoking everything (even if bespoking is much
| easier), or making sure you write a TON of documentation for
| anything you don't do daily...
| tarunm wrote:
| > I pay a service to manage backups so I can walk away from a
| solo project for months and know it's ticking away.
|
| Any recommendation of such service?
| js4ever wrote:
| Borgbase. I'm not affiliated with them, I'm just a happy
| customer
| albertgoeswoof wrote:
| Snapshooter
| marsdepinski wrote:
| Rsync.net
| bingohbangoh wrote:
| Tarsnap
| diiq wrote:
| Pretty much anyone you can pay to host a database, you can
| pay to back it up. Amazon rds, heroku postgres, google cloud
| sql... pick the service that works for you, and they have
| some affordance for backups. Backups are something I _expect_
| , rather than something I shop around for.
| dsgrillo wrote:
| My experience says to backup to a different vendor.
| sumtechguy wrote:
| Probably the best tool you have is somewhere to write your
| notes down. What you did, why you did it, how to reproduce it.
| That saves you time in a year or two when you go 'hmm I did
| something like this before have to do it again because some CPU
| ate itself'. My old notes have saved me a lot of re-work over
| the years. Because I had written down what I did so I do not
| have to rebuild it. I can just skip along the notes and be most
| of the way there.
|
| This bit seems to hold true across any size team for me. If I
| have notes I can spiffy them up and make cheat sheets out of
| them and bootstrap other developers faster. Or if it is just
| me, my feeble brain will forget odd details that I needed for
| something.
| Layvier wrote:
| I've been using a simple circle CI setup and AWS beanstalk on my
| side, and for the databases I only look for fully managed. I've
| had a few issues with beanstalk but it still saved me a lot of
| time, and it's pretty easy to find AWS credits. It's pretty good
| also to avoid wasting time on SSL setup as everything is
| integrated (if you use a load balancer) with Route 53 and ACM.
| Beanstalk uses application version so it's very easy to rollback
| to a previous version. I never liked DevOps much so this setup
| works pretty well for me, and seems quite scalable as well.
|
| If you're not very comfortable with k8s yet I really wouldn't
| recommend you to setup a cluster for a small project.
| prohobo wrote:
| I've been using Beanstalk for my new project, which is like an
| in-between for full on AWS and Heroku, but if you need you can
| decompose it into individual AWS components. It's more difficult
| to get started and there are way less sensible defaults, but I
| still got everything deployed for my POC in less than 4 hours.
|
| As for DBs / engines and such, I try to go for hosted services as
| long as they're not stupidly priced. I'm currently using Elastic
| cloud. If I need a relational DB I'll probably deploy Postgres on
| AWS and install some scripts to back up the data to an S3 bucket
| every day or so.
|
| I chose NOT to use k8s or Docker because, like you said, it's
| basically a fulltime job to maintain. I don't think it's needed
| at all unless you're a large organization.
|
| So basically: a mixture of roll-your-own, streamlining platforms,
| and hosted services. Do whatever works and is easiest, and don't
| worry so much about having ALL the goodies.
| dsgrillo wrote:
| I have currently 2 projects with some traffic (5k req/day)
| running in production.
|
| Both have a similar setup: - 1 droplet with
| docker pre-installed from DigitalOcean - clone directly
| the repo from github - together with the code, I have a
| folder with a bunch of docker images (caddy, mysql, php, redis,
| etc.) that I can easily spin up. - for any release, I
| manually ssh, git pull and run the migrations (if any) and
| manually rebuild any docker image if needed - I have daily
| jobs that dumps and zip the entire DB to S3 - if I have
| some deployment that I know will break any functionality during
| deployment, I warn the users before and accept that downtime.
| - never had to handle a "hard" rollback till now.
|
| I've planned to change this setup for while, but until now,
| didn't find any reason that justifies the effort.
|
| I spend 20$ (10$ per droplet) + few cents on S3 per month with
| them.
| lbriner wrote:
| Like most others said, if its not a painpoint now, just deploy
| manually and keep it simple. You will save time and money and
| more importantly, you can concentrate on what is important for
| early-stage companies: Nailing product-market fit so you can get
| the sales moving in.
|
| It isn't that hard, for someone with decent experience, to setup
| TeamCity/Jenkins/Octopus/Whatever but why spend the time? If you
| change something, you then have to go and spend time changing the
| automatic deployment.
|
| As someone else said, Devops is about velocity with quality, if
| velocity is OK and quality won't improve with tooling, don't
| bother.
| sergiotapia wrote:
| I recommend not doing any devops at all if you're a solo founder
| and trying to get it off the ground. You want to spend time on
| your product market fit, not fuck around with linux.
|
| New services like render.com make it really easy to get robust,
| cheap hosting up. Much cheaper than Heroku.
| vhodges wrote:
| https://fly.io
|
| - Easy Postgres clusters (but not managed) - Heroku like deploys
| (but less magic) - Easily geodistributed (deploy near your users)
| - reasonable pricing to get started (and not too bad beyond
| that).
| christophilus wrote:
| I lead a team of 3. We use render.com and have a handful of bare
| metal servers for CPU-intensive background jobs. The bare metal
| servers are auto-patched Ubuntu servers. It's pretty much 0
| maintenance. The server setup is automated via a single
| provisioning bash script. Render.com is quite nice, if a bit
| limited in its options vs Heroku. Highly recommended.
| fxtentacle wrote:
| I run my own Heroku clone which is then licensed by my other
| companies.
|
| Also, I review how much time each product / feature consumed each
| year and then decide what to deprecate. Each April I then have my
| own little party where I ceremoniously turn off last year's time
| wasters.
|
| I'd say it all boils down to "know your customer lifetime value"
| jamesmishra wrote:
| I wrote https://provose.com/ so I could quickly set up databases,
| containers, and filesystems on AWS. I wrote Provose because I
| needed to deploy a ton of infrastructure to AWS as a solo
| developer, but I could not afford to use a dramatically more
| expensive PaaS like Heroku.
|
| Provose is an open source project written in pure Terraform, so
| all of the high level abstractions are computed locally, and in
| general do not raise your AWS bill.
___________________________________________________________________
(page generated 2021-10-12 23:02 UTC)