[HN Gopher] How Fly.io and Tailscale Saved Notado
___________________________________________________________________
How Fly.io and Tailscale Saved Notado
Author : bsnnkv
Score : 135 points
Date : 2022-10-17 15:37 UTC (7 hours ago)
(HTM) web link (notado.substack.com)
(TXT) w3m dump (notado.substack.com)
| ricardobeat wrote:
| So what's the final cost of fly.io + proxy VM + postgres? That
| was the reason this whole thing started.
| anderspitman wrote:
| Fly.io and Tailscale are two of my favorite companies. I think
| the future of selfhosting is Linux services running in microVMs
| on Windows and Android devices that people already have lying
| around. Fly.io is proving that the VM side is feasible. You also
| need something like Tailscale to tunnel through CGNAT et al.
|
| I work on open source projects that solve many of the same
| problems, but it's a chicken/egg problem, and Fly.io and
| Tailscale are building the infrastructure that will show people
| what's possible.
| renewiltord wrote:
| I will migrate my tiny RDS and Cloud SQL side things to Fly.io a
| couple of weekends from now. It is definitely the sensible way
| for me to do this.
|
| I managed my own SQL DB once upon a time and I can tell you I
| would rather have it done for me. 10 years on from when I set it
| up, my provider was going to get rid of the hardware and couldn't
| VM migrate me, so I took it down. No urge to manage this stuff
| myself.
| RamblingCTO wrote:
| I'm curious as to why you chose to build your services in rust
| and then one microservice is in golang?
| bsnnkv wrote:
| Thanks for this question!
|
| Back in early 2020, Meilisearch did not have a Rust client
| library, but it did have a Go client library[1], and there was
| nothing comparable to lib/pq[1] in the Rust ecosystem which
| would allow me to create a listener on a table.[3]
|
| Go is generally my "fallback language" when something would not
| be practical to do in Rust; it has a very nice, mature
| ecosystem, and as long as you aren't condemning yourself to
| interface{} hell, it remains in my eyes a perfectly capable and
| reasonably ergonomic alternative for well-defined use cases.
|
| [1] https://github.com/meilisearch/meilisearch-go
|
| [2] https://github.com/lib/pq
|
| [3] https://pkg.go.dev/github.com/lib/pq#NewListener
| RamblingCTO wrote:
| Thank you for taking the time to answer, I'm always curious
| as to the why ;) Sounds like a good decision in that case!
| Good luck with your project!
| [deleted]
| queezey wrote:
| FYI: Your link for Overmind is to the wrong project. The process
| manager is https://github.com/DarthSim/overmind
|
| The article currently links to a deprecated Angular.js project
| with the same name (https://github.com/geddski/overmind)
| bsnnkv wrote:
| Thanks for taking the time to point this out, I really
| appreciate small acts of kindness like this.
| whimsicalism wrote:
| 3000 users sounds like too few to justify k8s
| bsnnkv wrote:
| Definitely. And just think, Notado started off on k8s without
| any users at all! As the article hints at, the choice mainly
| came down to laziness and complacency, at the cost of
| significant added complexity that would be paid by "future me"
| ("past me" is not always the most considerate...)
|
| I hope that one thing that potential solo devs take away from
| this post is that you can get very far without even needing to
| think about touching k8s (especially if you are already very
| comfortable with k8s).
|
| After all, HN and Pinboard comfortably serve many more users
| than our side projects likely ever will without bringing in any
| kind of heavyweight container orchestration platforms.
| jalino23 wrote:
| I fell in love with tailscale as soon as I figured out what it
| enabled me to do.
|
| it was very confusing at first cause I relate "vpn" to something
| like nordvpn or express vpn when I hear it. but no tailscale is a
| lot more than that.
| crawshaw wrote:
| thanks! we have a lot of work to do explaining tailscale. we're
| working on it.
| CharlesW wrote:
| "Make any device anywhere part of your own private, secure
| network."
| tptacek wrote:
| The challenge is that there's two kinds of VPN users
| (corporate and retail), and for corporate users, the thing
| you just described is simply a VPN.
| atonse wrote:
| We're a corporate user but to me it was "wireguard + SSO"
| and I was sold.
|
| The Okta groups sync is a beautiful cherry on top.
| tptacek wrote:
| I agree: for me at first the mind-blowing thing was
| seamless IdP integration. (Now I'd have ACLs neck-and-
| neck with the SSO stuff). But like, the extensible
| private network thing, that's what corporations use VPNs
| for. :)
| mbesto wrote:
| Just to further clarify as I've recently gone down the
| rabbit hole of Tailscale...but isn't the idea that I can
| make any _part_ of the device running on a private, secure
| network? In other words, even if I 'm connected to a node,
| I'm still running "on my network", whereas when I connect
| to a traditional VPN service it just kinda takes all of my
| traffic and puts it in that same network "space", the
| biggest difference meaning I no longer see a traffic hit
| when I'm reaching www.google.com because it doesn't mean
| all of that traffic has to get funneled through the VPN
| traffic.
|
| As you can tell, not a network guy here! I could be totally
| wrong.
| e1g wrote:
| By default, and in the most common configuration, the
| Tailscale "idea" is to intercept all network requests
| made to a reserved IP range (100.x.x.x) that cannot have
| public services on it (similar to 192.168.x.x). Each of
| your devices gets a static IP and can reach any other of
| your devices using their static IP. All traffic will be
| encrypted, and it will work really really hard to get
| through all NAT gateways and firewalls in-between.
|
| As those static IPs are in a reserved range, they will
| never clash with a public service like a website and
| Tailscale does not mess with your regular traffic.
|
| You could make things more complicated by setting up exit
| nodes and exposing subnets, but that's the general idea.
| So simple, it's brilliant for their standard use-case.
| atonse wrote:
| I am doing a company talk (we do them weekly) about network
| security and at least the way I was planning to explain
| tailscale to our staff was via a picture I still need to
| somehow build, but it goes something like this:
|
| You have a crowd of people (like one of those massive crowds
| you see in Tokyo etc). And you are both at either sides of
| the crowd, and you each have a can with a string connecting
| you to each other - and I explain that the crowd is the
| public internet, and tailscale allows you to create that
| string across the whole crowd in a way that nobody in the
| crowd can ever listen to you guys talk, even though the
| string literally passes through/over tens of people in that
| crowd.
|
| If someone has a photo like that, I'd gladly use it :)
| Gwarzo wrote:
| That is just a vpn
| atonse wrote:
| Yep. And Tailscale is the only VPN (like a mesh network,
| not "let me watch BBC Channel 4 from the US") product I'm
| aware of that someone who mostly understands cans,
| strings, and crowds, can setup in half an hour (with
| maybe just a couple questions to a tech person).
|
| I know enough about all the nuts and bolts under the hood
| with Tailscale and that makes it feel MORE magical to me,
| that with all those things to deal with, it still just
| works. And it's no compromise (uses wireguard, etc)
|
| The last time I saw that level of magic was in a product
| called Hamachi. It Just Worked(tm) more than a decade
| ago. LogMeIn bought them and ruined it.
|
| But with Hamachi I didn't know how it worked under the
| hood.
| bedast wrote:
| Man, marketing of those tunneling services as VPNs is actually
| causing damage. Those services are NOT VPNs.
|
| Tailscale is very much more like a real VPN than those other
| things that are pretending to be VPNs.
| NoraCodes wrote:
| They definitely are VPNs, just a specific subset of use cases
| of a VPN.
| dijit wrote:
| Semantically: those services are acting more like encrypted
| proxies, with some kind of exit far away from you
| physically.
|
| VPN's are _usually_ used to create Virtual _private
| networks_. Extending your LAN across several sites using
| software, or granting you access to your LAN when you 're
| not physically located close to it.
|
| It's hard to imagine why the meaning has changed as much as
| it has, because even though a consequence of this
| connection can be that you exit to the internet via your
| destination networks internet connection; I feel like this
| isn't the original use case at all, similar to how you
| _can_ use a chair as a ladder.
| combyn8tor wrote:
| I see your point.
|
| > encrypted proxies
|
| I suggest "encrypted tunnels"
| shp0ngle wrote:
| In 2022, VPN is mostly about "getting around netflix
| geoblocking" or "getting around Chinese firewall"
| brk wrote:
| Or, "making public/hotel/random wifi less of a security
| risk", which happens to be my current usage of Tailscale.
| NoraCodes wrote:
| > It's hard to imagine why the meaning has changed as
| much as it has
|
| Because the supporting infrastructure was historically
| called that, and it stuck. Just like how pocket computers
| are called "phones" because they evolved from telephones.
| awestroke wrote:
| How is the latency and performance of Fly.io micro vms compared
| to, say, a comparably priced DigitalOcean VPS?
| bsnnkv wrote:
| In the specific case of Notado, the Tailscale tunnel between
| Fly and Digital Ocean for database access naturally introduces
| some minimal additional latency.
|
| Besides this, I'm not sure I'm in the best position to compare
| the latency and performance of a service running 100% on a
| Digital Ocean VPS vs. a Fly.io micro VM. Hopefully someone else
| can chime in with their experiences on this thread.
| Gcplp wrote:
| I think that fly.io is still using firecracker for their
| microvm offering, although qemu has a similar machine type
| implemented.
|
| Having worked on firecracker and with firecracker, it does a
| good job in the networking area.
|
| Note that there's no multiqueue implementation available, so
| depending on how many connections and data you're passing
| through, it may start struggling if the workload is high.
|
| I don't know the specifics of how DO sets up their VMs, but I
| assume that they're not doing any fancy networking
| optimizations, so you may end up with similar issues on both
| setups.
| throwthere wrote:
| I love Fly.io and use it in production, but unless I'm missing
| something the author would have been better off just running
| their kubernetes cluster locally on a $6 VPS on digital ocean.
| That would also obviate the need for the procfile hack and
| tailscale.
| radicaldreamer wrote:
| A lot of super sophisticated engineering goes into solving
| problems which were created by the engineers themselves and
| obviously didn't need to be.
|
| I remember hearing that Oscar (a health insurance startup whose
| main differentiation was having a decent app) ran their own
| mesos cluster for some reason creating tons of cost and
| complexity with little to no reason (also operated tons of
| microservices with startup headcount).
| bsnnkv wrote:
| The larger story that I will be telling through this series
| of posts is the re-integration of the cast that powers Notado
| back into a monolith; hopefully it's something you and other
| readers will find interesting!
| jonas21 wrote:
| Alternatively, they could have just left things alone and paid
| the $100/mo.
|
| It's odd because this blog post also announces the end of the
| free beta for the service. It goes in depth about how they
| jumped through a bunch of hoops and introduced more complexity
| to the service because they were annoyed that DigitalOcean
| raised their prices. And at the end they're asking you to pay
| $5.99/mo for it?
| bsnnkv wrote:
| So interesting to see how different people interpret
| complexity.
| skrtskrt wrote:
| Not even the kubernetes cluster, the author eventually is just
| running multiple processes in a single VM it seems, and scaling
| vertically using the VM, which of course could have been done
| on DigitalOcean.
|
| However, people do seem to love the fly.io developer
| experience. It's a testament to fly.io that the auther seemed
| to _enjoy_ spinning up on fly.io.
|
| It's also a testament to DigitalOcean that it wasn't painful or
| expensive to partially migrate away from their services.
| bsnnkv wrote:
| > Not even the kubernetes cluster, the author eventually is
| just running multiple processes in a single VM it seems, and
| scaling vertically using the VM
|
| This is an excellent summary!
|
| > ... which of course could have been done on DigitalOcean.
|
| > However, people do seem to love the fly.io developer
| experience. It's a testament to fly.io that the auther seemed
| to enjoy spinning up on fly.io.
|
| It could have been done, but it would have required me to
| introduce more moving parts and complexity to handle
| deployments of updates (do I keep the same VM and SSH in to
| update? Do I build a fresh VM with the latest version and
| then cut over the traffic? etc.) It's pretty hard to beat
| "fly deploy" in my opinion.
|
| Similarly, the vertical scaling story is much more
| streamlined on Fly.io - it's also pretty hard to beat "fly
| scale vm".
|
| I've mentioned in another comment on this post about how
| great the monitoring goodies that Fly gives you for free are.
|
| Another point that I didn't highlight in the article itself
| is that Fly.io is currently not collecting bills less than
| $5/month (this might change in the future, I don't know), and
| if you have a resource-efficient service (or services), with
| the numbers that most solo devs working on side projects are
| talking about, it's not difficult to stay under $5/month and
| have the excellent developer experience to boot.
|
| > It's also a testament to DigitalOcean that it wasn't
| painful or expensive to partially migrate away from their
| services.
|
| No complaints on this point; I'm also still very happy with
| Digital Ocean's managed database services.
| indigodaddy wrote:
| I'm thinking about a simple blog. Someone convince me that
| I should use fly.io and caddy and how/why deployment and
| subsequent blog/post updates would be easy and
| uncomplicated.
| bsnnkv wrote:
| I don't think I'd enjoy managing my own single-node Kubernetes
| deployment, monitoring, upgrades etc., but some might!
|
| The other point that I didn't really cover in this article was
| all the monitoring goodies you get for free with Fly;
| definitely not something to be overlooked as a solo dev.
|
| The medium-term plan is to move the database to Fly as well,
| but I wanted to share a way for people to do a partial
| migration to Fly if they are not quite ready for a full
| database migration.
|
| As for using a Procfile manager being a "hack", once we are
| freed from the constraints of Docker containers and back in the
| realm of VMs, I think running multiple processes again becomes
| the norm.
|
| I don't quite agree with the "one machine (or machine-like
| abstraction) => one process" thinking in this regard.
| bmorton wrote:
| I can't think of another for-profit company that gets as much
| positive attention on the front page of HN as Fly.io. Either it's
| a really great product, or HN is turning into Reddit where
| "sponsored content" is carefully folded into the organic.
| bsnnkv wrote:
| It's a really great product!
| theonething wrote:
| We tested it and experienced intermittent deploy failures so
| we didn't go with it.
| ilrwbwrkhv wrote:
| 3000 users? You can save the data on disk and not worry about it.
| Get a vultr 5 dollar micro instance. I don't know why all these
| engineer driven businesses focus on the wrong things. You don't
| need fly.io. Of course they will tell you you need them, I don't
| blame them. But fly.io or kubernetes should always be done by a
| migration team which you can hire once you have insane profits.
| Yes profits, the other tiny detail most businesses are missing
| these days.
| mrkurt wrote:
| I won't tell you you need Fly.io. I am offended you'd put us in
| the same bucket as kubernetes though, rude.
| jabo wrote:
| It's interesting to see how being in the same league as
| kubernetes could either be considered a compliment or an
| insult depending on who you ask!
| GiorgioG wrote:
| K8s is a complexity sinkhole for 99% of companies. It's an
| insult.
| mountainriver wrote:
| Uh huh here we go. What other tool do you propose for
| running many containers with dependencies on one another?
|
| Or why should I target a bunch of different cloud APIs to
| make my tool consumable when I can target one?
| ipaddr wrote:
| You've inspired me to give it a chance.
| alfalfasprout wrote:
| It's funny how we spend SO much time dealing with K8s
| related nuances and limitations. In practice, I'm not so
| sure anymore even for a large enterprise that K8s
| actually simplifies things in any way or adds more
| robustness.
| ilrwbwrkhv wrote:
| Fair enough. You guys are definitely a breath of fresh air
| compared to kubernetes. And we already use you guys although
| Cloudflare is pitching us their Workers pretty heavily.
| mrkurt wrote:
| I hope their pitch is "rewrite your app in JavaScript, even
| if it's already JavaScript". :D
| bsnnkv wrote:
| "Yo dawg, I heard you like Javascript..."
| dimgl wrote:
| 110% this. For a small business with a small amount of users, I
| always suggest to *use the simplest thing that'll save you the
| most time and money to deploy your service*.
| jahsome wrote:
| Who needs profits when you can just throw an ICO out overnight?
| bsnnkv wrote:
| > You can save the data on disk and not worry about it.
|
| I one day aspire to have the cajones of ilrwbwrkhv.
|
| For the reference of other potential solo devs, if done right,
| Fly.io will still be cheaper than the Vultr 5 dollar micro
| instance (or equivalent) with an incomparably better DX.
| ipaddr wrote:
| With the threat of a much higher bill if done wrong. That
| $2.50 vultr instance provides a lot more like 20 gigs of
| storage nevermind security of a fixed bill.
|
| You might be able to save a little, you might pay a lot more.
| Hedge your bet and use vultr.
| [deleted]
| wooque wrote:
| Because founder is more interested in playing with shiny toys
| of Rust/Kubernetes/Fly.io/Tailscale instead of building
| business.
|
| Not a bad idea, he has something to pad his CV with when
| startup inevitably fails.
| Dowwie wrote:
| In re: "Tokio, Messaging Queues and Async Rust", the Rust
| community may one day have a robust, fault-tolerant data
| ingestion and data processing pipeline like Elixir's Broadway,
| but today it remains a customized, heavy lift (depending on
| developer expertise). It would be great to have, in Rust, though.
___________________________________________________________________
(page generated 2022-10-17 23:00 UTC)