[HN Gopher] Launch HN: LayerCI (YC S20) - Staging servers that a...
___________________________________________________________________
Launch HN: LayerCI (YC S20) - Staging servers that act like (and
replace) CI
Hi HN, Lyn & Colin here. We're co-founders of LayerCI
(https://layerci.com), which gives you a modern DevOps experience
(CI/CD & staging environments) with as little work as writing a
Dockerfile. Most teams need CI/CD (run the build and deploy every
time a developer pushes) or staging (host a server with my app in
it to share), but current approaches always have at least one of
these problems: - Simplistic (only run unit tests) - Slow (wait
10 minutes to run the same repetitive setup steps like "npm
install") - Complex (cache keys, base images, a slack channel to
reserve staging servers, ...) We've spent over a year iterating
with our customers to build a product that solves all of these
problems. Our configuration files (Layerfiles) look like
Dockerfiles, so regular developers can write and maintain them.
Here's one that creates a staging server for create-react-app:
FROM vm/ubuntu:18.04 RUN curl -sS
https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - &&
curl -fSsL https://deb.nodesource.com/setup_12.x | bash && apt-get
install nodejs python3 make gcc build-essential COPY . . RUN npm
install RUN npm test RUN BACKGROUND npm start EXPOSE WEBSITE
http://localhost:3000 We charge a flat $42/mo/developer on our
paid plan. Because it's a flat fee and not usage based, we're
incentivized to make things as fast as possible: Our current
margins come from a custom-built hibernating hypervisor that lets
us avoid running "npm install" thousands of times per day. We've
upgraded the free tier to 5GB of memory for new installations this
week. It's perfect for personal projects or small MVPs where you'd
like a powerful demo server that will build on every push and
automatically hibernate when it's not being used. The easiest way
to try out LayerCI is to follow our interactive tutorial:
https://layerci.com/ or look at the docs: https://layerci.com/docs/
We would love to hear your thoughts about CI/CD, staging, and what
we've built!
Author : colinchartier
Score : 65 points
Date : 2021-01-31 16:30 UTC (6 hours ago)
| 0xbadcafebee wrote:
| _" LayerCI is an all-in-one devops platform, but we provide three
| main value propositions to our users:"_ - Run e2e
| tests on every commit. - Collaborate easier with per-PR
| staging environments. - Set up CI/CD to build and deploy to
| production
|
| Anyone can do this without your company. If that's the value you
| bring, then I don't need your company. What value do you bring
| that I can't get somewhere else?
|
| Also, red flag: mis-use of the term DevOps for marketing
| purposes.
|
| _" There are a few ways of thinking about these files named
| 'Layerfile':"_ - Auto-discovered Dockerfiles that
| build entire virtual machines instead of containers, just as
| quickly. - Define a tree of virtual machines. Each
| subsequent layer can inherit all the running processes in its
| parents. - Trigger specific actions like build, push, test,
| and deploy in parallel every time you push new code.
|
| It's not explained well what benefit there is to using VMs rather
| than containers. Later on it briefly mentions possibly better
| caching than Docker containers. If it's actually better than
| Docker, you are _severely_ burying the lede. "Better than
| Docker" should be on the front page. It's also not explaining
| whether I need to throw away all the time and money I might have
| already invested in containers, or what this is and isn't
| compatible with.
|
| A lot of this solution is really built around optimizing one
| specific problem, which is caching during re-running whole CI
| pipelines. This limits the viability of your business. As soon as
| your customers figure out a new way to solve this problem, they
| don't need your custom platform anymore. I would consider LayerCI
| more of a specific feature of a much larger offering.
|
| (In particular, the 'correc't way to solve this problem is to
| stand up an environment for your PR and structure your CI so you
| only re-run the parts that affect the particular change you want
| to make, rather than re-running the entire pipeline. It involves
| putting more thought into running your pipeline but often means
| the only tasks that get executed are copying a file and
| restarting your app)
|
| P.S. You're installing the yarn GPG key and then not doing
| anything with it. Your node setup script isn't installing the
| yarn repo or yarn itself, so this is a throwaway command.
| colinchartier wrote:
| > In particular, the 'correct way to solve this problem is to
| stand up an environment for your PR and structure your CI so
| you only re-run the parts that affect the particular change you
| want to make, rather than re-running the entire pipeline. It
| involves putting more thought into running your pipeline but
| often means the only tasks that get executed are copying a file
| and restarting your app
|
| This is actually exactly what our platform helps with - instead
| of micromanaging this (and updating the configurations) you can
| rely on a friendly SaaS to do it for you.
|
| I wouldn't call us "better" than Docker, we're more like "Take
| Docker and make it work better for CI/CD and staging servers"
| malthejorgensen wrote:
| As an early user -- the speedups are very real! Feels great to
| just skip build step after build step, due to Docker-like layers.
| colinchartier wrote:
| Thanks, Malthe - it's been great working with your team.
| benkoller wrote:
| Great approach. I've been running ops teams for a long time, and
| eventually all teams have ended up spending some time optimising
| our CI tooling to involve some degree of caching - so you
| definitely are on to something.
|
| Before I clicked your pricing I'd really wished for an affordable
| pricing plan to run this on BYO resources, but only your
| enterprise plan seems to cover this. I always get an iffy feeling
| when I have to build my software on external resources I have 0
| control over. Your downtime will prevent teams from shipping
| their code - but I guess that thought is part of your upsell to
| the enterprise plan.
|
| Anyway, it's one of these ideas I'd wish I had years ago, so
| congratz to you.
| colinchartier wrote:
| BYO resources for CI sounds good in theory, but considering
| costs it's almost always a worse proposition:
|
| - 10 hours / month of maintenance + 200 hours of setup @ $60/hr
| = $13k in the first year, not including any infrastructure
| costs.
|
| If you just redirected that money to a hosted offering you'd
| get significantly more powerful servers without needing to
| allocate engineering / management resources - it also makes it
| significantly easier to push updates & monitor uptime on the
| host's end.
|
| Even traditionally hosted tools like Atlassian's suite are
| moving to cloud for the same reason:
| https://www.atlassian.com/migration/journey-to-cloud
| 1337shadow wrote:
| > - 10 hours / month of maintenance + 200 hours of setup @
| $60/hr = $13k in the first year, not including any
| infrastructure costs.
|
| Impressive numbers! would love to do these for you for a
| tenth of the price you are quoting, I'd still get like a 400%
| margin or something. GitLab runner is just installing the
| package with apt and running the register command once, then
| updating just magically happens when you upgrade the system.
|
| As for infrastructure, you will find dedicated servers that
| will perform extremely well for an equivalent
| 49$/mo/developer.
| colinchartier wrote:
| The type of CI you're describing would be in the "simple"
| category in the OP - if your goal is just "run a single
| docker build and check if it fails" there are lots of free
| hosted tools you can set up (github actions, gitlab CI,
| etc)
|
| There is in fact a burgeoning industry of devops
| consultants that set up CI pipelines for companies, I'd
| encourage you to consider it if you like the work!
| 1337shadow wrote:
| Actually we do, looking for more projects!
| colinchartier wrote:
| Will let you know if we know anyone that's looking for a
| consultant - do you have a website I could refer folks
| to?
| zbruhnke wrote:
| While this is likely true in many cases I'm in a very
| sensitive industry (banking) and we tend to self host things
| not for cost reasons but for security reasons. We spend a lot
| of time going through pen tests, getting SOC2 compliance,
| etc.
|
| Handing off something this critical can cause an even more
| painful audit in many cases so just a thought to consider
| cost is sometimes not the only factor.
|
| Looks like a really cool product though!
| colinchartier wrote:
| We're actually in the process of getting SOC2 and pen tests
| ourselves - another benefit of a hosted offering is it can
| (eventually) integrate into your compliance system (e.g.,
| vanta)
|
| A lot of our customers are in fintech (payroll, banking,
| etc) so we've spent a lot of effort on our security model:
| https://layerci.com/security
| Lyn_layerci wrote:
| Colin's cofounder here :) Excited to hear from you all
| old-gregg wrote:
| I see developer tools on HN. I upvote!
|
| Congrats on launching! I know this is a good idea because we
| actually invested time in building something similar internally.
| colinchartier wrote:
| Thanks, we're big fans of teleport - funny how small the
| developer tools world actually is.
| aloknnikhil wrote:
| Nice work! I played around with the demo and launched an SSH
| session for the last layer (hosting the website). And I was able
| to trigger a VM reboot from the shell. And now the website is not
| up anymore. Curious to know if there's any monitoring or alerting
| that I can see when something like this is happening? And how do
| I go about debugging it/restoring functionality?
| colinchartier wrote:
| Hey Alok,
|
| RUN BACKGROUND basically does "./thecommand&" so it will stop
| after a restart, the onboarding example is ephemeral for
| simplicity.
|
| If you want things to persist across restarts you'd have to add
| a systemd script or docker container the same way you'd run it
| in production.
| aloknnikhil wrote:
| Makes sense. Thanks!
| 1337shadow wrote:
| > - Slow (wait 10 minutes to run the same repetitive setup steps
| like "npm install")
|
| That should happen any time you change dependencies if your
| Dockerfile is efficiently setup, not every time you change a line
| in runtime code ...
|
| Maybe that:
|
| COPY . .
|
| RUN npm install
|
| Is not what you should do to have an efficient docker build
| phase, did you try this instead ?
|
| COPY package.json
|
| RUN npm install
|
| COPY . .
|
| Multistage builds will help if you build in two languages in the
| same Dockerfile.
|
| Anyway, still slow when you add/remove a package because it
| redownloads everything, my best trick is to share ~/.npm cache
| directory, but I manage this with buildah instead of docker
| build.
| colinchartier wrote:
| One of the benefits of Layerfiles vs Dockerfiles is that those
| two examples will do the same thing - we monitor which files
| are read so you don't have to micromanage copying specific
| steps.
|
| But I suppose "npm install" is a simplistic example, "npm run
| setup_our_ci_environment" would be a more likely step for a
| larger CI pipeline
| [deleted]
| colinchartier wrote:
| If you'd like to chat with me (CEO @ LayerCI) personally, I'm at
| colin@layerci.com
| technics256 wrote:
| What's the difference between this and something like hashicorp
| waypoint?
| thruflo wrote:
| Congrats on the launch :)
|
| The pricing incentives sound smart.
|
| Do you see the ability to replicate more complex architecture as
| a differentiator? For example, one of your homepage quotes
| mentions "running a Kubernetes stack inside of a Layerfile". Can
| you elaborate more on how Layerfiles enable this?
| colinchartier wrote:
| With Kubernetes, a typical workflow might look like this:
|
| 1. Start kubernetes cluster
|
| 2. Build docker images
|
| 3. Deploy docker images (helm, kubectl, argocd, ...)
|
| 4(a) Run unit tests (kubectl exec -l k8s-app=web rake test)
|
| 4(b) Run e2e tests (kubectl run cypress)
|
| 4(c) Create ephemeral environment (EXPOSE WEBSITE
| localhost:8000)
|
| Because we take memory snapshots after each step, you'd
| effectively get a fresh, fully-provisioned kubernetes cluster
| immediately after pushing instead of re-running all the steps
| every time (you'd skip to step 3) and then run 4{a,b,c} in
| parallel (they'd all "fork" the VM and get a separate copy of
| all of the resources)
|
| Here are a few links that go into more detail:
|
| - https://layerci.com/docs/tuning-performance/run-repeatable
|
| - https://layerci.com/blog/ci-at-layerci/
| thruflo22 wrote:
| Ok, so the more complex the infra, the more advantage there
| is in caching.
| colinchartier wrote:
| The idea is it starts as simple as a docker build (e.g., as
| easy or easier than GitHub actions, orbs, etc) but scales
| as your code/infra does.
| rumblecat wrote:
| I'm looking at your docs, and the RUN REPEATABLE command seems
| really powerful. But if the state is broken after a run, like if
| you have some pods stuck at Terminating, how would you recover
| things?
|
| Another question I have is how would you handle state that
| sometimes needs to update and sometimes doesn't? For example, it
| would be ideal to have a staging database that can keep having
| migrations and data added to it when new features are added, but
| we only want to checkpoint the changes to it from testing when
| the PR is actually merged.
| colinchartier wrote:
| If the state is broken, you can "rerun without snapshots" via
| the dropdown in the top right - and since future runs load the
| latest snapshot, they'd use a "clean" one.
|
| For databases - usually users have a named S3 bucket and use a
| secret to authenticate, since we take memory snapshots, the top
| of your Layerfile can be "start the database and populate it
| from this specific anonymized dump" and then you can edit the
| file in S3 and re-run without snapshots if you'd like to reload
| it.
|
| Here's the doc page for that:
| https://layerci.com/docs/advanced-workflows/layerfiles-can-s...
| data_ders wrote:
| How does dependency caching configuration work? I get wanting to
| make things as fast as possible, but I struggle a bunch a Python
| dep caching in Circle CI... is there a way to ensure a 100%
| rebuild? Also interested to see how you plan to support open
| source
| colinchartier wrote:
| There's a "CACHE" directive if you want traditional dep
| caching, but if you don't edit requirements.txt, a Layerfile
| will just entirely skip the pip install
|
| You can "SNAPSHOT disabled" if you'd like certain steps to
| _always_ run
| mrkurt wrote:
| This is super cool. I would like to know more about VM
| hibernation!
| colinchartier wrote:
| Hey Kurt, we're big fans of fly.io!
|
| Happy to chat about tech sometime, I think you folks are using
| firecracker for a similar use-case? Drop me a line at
| colin@layerci.com
| ignoramous wrote:
| Hi Colin, congrats on the (re-)launch! Fellow SUS 2019
| graduate here.
|
| I know layerci's tech precedes snapshot support in
| Firecracker [0], but are snapshots similar to layerci's magic
| or there's still a significant way for snapshots to go before
| it can be on par with layerci's tech? I ask because I
| remember you mention layerci uses techniques (from how Linux
| hibernates PCs) which are good enough for tests and staging
| but not production; [1] whereas Firecracker obviously can't
| rely on such tech so there must be key differences? Thanks.
|
| [0] https://github.com/firecracker-
| microvm/firecracker/blob/mast...
|
| [1] https://news.ycombinator.com/item?id=23036776
| colinchartier wrote:
| Firecracker is definitely heading in the same direction we
| did with their "diff snapshots", though they don't let you
| create a COW chain yet, so there's still quite a bit of
| development to be done on that front.
| tekno45 wrote:
| Can we get SSO for not a bajillion dollars?
|
| Seems like this functionality is really inflated across the
| entire SaaS space.
|
| Is SSO really that hard? that expensive?
| colinchartier wrote:
| Most developers log in to LayerCI via GitHub or GitLab - so if
| you have a GitHub plan with SSO, you'll get SSO with Layer as
| well.
|
| We even support self-hosted GitLab, so if you use a custom SSO
| provider there, developers could use that oauth flow for
| authentication + authorization.
| mamcx wrote:
| This sound very tempting!
|
| I run Rust CI for Windows/Linux (on azure pipelines) and it need
| art least 1h to complete, with several tricks already done.
|
| When something broke the iteration is very slow to fix it (you
| wait 1h, then find the problem, then publish, another hour, you
| forget something...)
|
| So I have a local deployment so i can fix things faster, but is
| like negating the whole point.
|
| If yours can help in deploy rust faster chime in
| https://www.reddit.com/r/rust/ because is a major pain point!
|
| P.D: I assume this not include osx/windows?
| colinchartier wrote:
| We definitely support rust - you could even use "RUN
| REPEATABLE" to reuse the state from prior builds, the same way
| you'd build locally.
|
| Happy to chat more and send you some swag if you try it out and
| write a blog post about your experience :) colin@layerci.com
|
| You can't 'FROM vm/osx' or 'FROM vm/windows' yet, it's
| relatively difficult to navigate licensing as a startup
| unfortunately.
| mamcx wrote:
| Yeah, lets try. If I can cut times for deployment the web
| sites is still a win. I could wait for the windows builds,
| them are for utilities...
|
| > You can't 'FROM vm/osx' or 'FROM vm/windows' yet, it's
| relatively difficult to navigate licensing as a startup
| unfortunately.
|
| Maybe allowing to connect to vultr/digitalocean? Them have
| cloud instances with windows.
___________________________________________________________________
(page generated 2021-01-31 23:00 UTC)