[HN Gopher] Remote Workstations for the Discerning Artists
       ___________________________________________________________________
        
       Remote Workstations for the Discerning Artists
        
       Author : el_duderino
       Score  : 83 points
       Date   : 2021-03-08 17:39 UTC (5 hours ago)
        
 (HTM) web link (netflixtechblog.com)
 (TXT) w3m dump (netflixtechblog.com)
        
       | dstaley wrote:
       | I was a little disappointed to see that there wasn't a discussion
       | of how Netflix provides (or doesn't!) a high-fidelity visual
       | experience. I imagine certain artists require fluid-motion and
       | previews without visual artifacting, so I was hoping there was
       | some discussion about how they provide that. I've used a handful
       | of remote desktop systems, and I haven't found anything that's
       | able to provide even a passable video watching experience. The
       | issue I've run into is that the technology designed for low-
       | latency video heavily utilizes compression, which falls apart
       | when looking at text.
       | 
       | I really wish Microsoft would just throw a ton of money at making
       | RDP an absolutely delightful experience. Last time I checked, RDP
       | wasn't able to use your GPU at all unless you had a very
       | specific, no-longer-maintained version of Windows 10 Pro (the
       | Workstation variant).
        
         | waymon wrote:
         | Teradici clients for realtime. For smooth playback they export
         | to a mp4. For real-time smooth playback, they do SDI to a
         | streaming box that provides low latency.
        
         | mikepurvis wrote:
         | I think the gold standard here is what Chromecast does with
         | YouTube-- tell the cast device directly what video to show
         | instead of re-encoding and passing the whole thing over the
         | network.
         | 
         | But of course, doing that requires application level support,
         | or at least low-level hooks into the desktop rendering system
         | to extract video streams and treat them differently from
         | everything else. And if the video is being generated on the fly
         | or is an uncompressed local preview, then you may be looking at
         | an intermediate compression step regardless, but at least it
         | could be compression suited for video instead of for text.
        
           | zamadatix wrote:
           | Well, that works because there is a pre-rendered video that
           | one can just point the Chromecast to most of the time. If you
           | cast a tab or whole desktop to display anything that's not
           | pre-rendered (which is what workstations would fall into)
           | then Chromecast runs into the same problems as anything else.
        
           | dstaley wrote:
           | > But of course, doing that requires application level
           | support, or at least low-level hooks into the desktop
           | rendering system to extract video streams and treat them
           | differently from everything else.
           | 
           | That's exactly what RDP does on Windows! I don't know the
           | extent to which controls it supports rendering on the client,
           | but the concept is that RDP is aware of things like text,
           | buttons, title bars, etc., and is able to tell the client
           | things like "draw a button at (20, 140)" instead of sending
           | the raw pixel data. With enough engineering effort (and some
           | standardization on the application UI side) I totally think
           | Microsoft could make RDP the protocol of choice for virtual
           | desktops.
        
             | mikepurvis wrote:
             | It makes total sense; OTOH I'm kind of cracking up here
             | because isn't this what a web browser has become? A portal
             | for viewing a remote application state, that can be
             | serialized into a blob of semi-structured XML and JSON?
             | 
             | It's hard to imagine that retrofitting such a serialization
             | onto the total wild west that is desktop applications could
             | be any more successful or consistent than what the web has
             | already got.
        
         | zamadatix wrote:
         | > Last time I checked, RDP wasn't able to use your GPU at all
         | unless you had a very specific, no-longer-maintained version of
         | Windows 10 Pro
         | 
         | Not sure where you got that idea. RDP has been able to use the
         | GPU for both encoding and the apps since Windows 7 era and both
         | certainly still work on the current version of Windows 10. The
         | main limitation I've run into is it's still limited to 30 FPS
         | though.
        
           | dstaley wrote:
           | It's my understanding that RemoteFX is only available in
           | Windows Server and Windows 10 Pro for Workstations.
        
             | zamadatix wrote:
             | During the Windows 7 era it required Enterprise or Server,
             | during the Windows 10 era it requires Pro, Enterprise, or
             | Server (or that Pro for Workstations version you mention).
             | Remember you have to actually enable it though, it's not
             | configured by default.
             | 
             | Perhaps you're thinking of the deprecation of the RemoteFX
             | guest vGPU for the Hyper-V role?
        
         | geodel wrote:
         | > I really wish Microsoft would just throw a ton of money at
         | making RDP an absolutely delightful experience.
         | 
         | I suspect they will make VSCode RDP plugin and call it next
         | generation remote desktop.
        
         | piotr_chomczyk wrote:
         | Hey, we launched Renderro - cloud computers for filmmakers,
         | graphic designers and animators in October 2020, and since then
         | we are helping creatives all around the globe. Check us out
         | here: https://renderro.com/
         | 
         | We are supporting up to 4 monitors, each up to 4k res, and the
         | quality we are offering (60 Hz and low latency) allows you to
         | do real-time editing, eg. when using Premiere Pro or Avid MC
         | running on our Cloud Computers. We are also used a lot for 3D
         | design and rendering with Blender.
         | 
         | Independent review: https://www.youtube.com/watch?v=LxM4mC5hwpo
         | 
         | Tutorial showing how to use Renderro:
         | https://www.youtube.com/watch?v=Uw44El8kMxM
         | 
         | If you have any questions, don't hesitate to contact us at
         | info@renderro.com
         | 
         | Please note, that with us you don't need any additional tools
         | to connect to our Cloud Computers (no need for Teradici or
         | Parsec) - we've got VDI solutions built in Renderro. We are
         | focusing on delivering Renderro as super easy to use, out-of-
         | the-box solution with clear pricing and superb quality, so I
         | hope that you will enjoy it ;)
         | 
         | Cheers!
        
         | boulos wrote:
         | You should probably check out Teradici. They even finally
         | published Linux clients.
        
         | nonane wrote:
         | I work on Jump Desktop (https://jumpdesktop.com) that's being
         | used for remote video editing @ 60fps on up to 3 monitors. It
         | supports very low latency remote desktop and even works in
         | environments that have high latency. For example here's Jump
         | connected from the east coast to a physical server in Paris
         | with 150ms ping times between the two:
         | https://twitter.com/jumpdesktop/status/1356679263806763013?s...
         | 
         | It works on Macs and Windows. It works really well with text as
         | well as full motion video.
        
       | Grakel wrote:
       | They should rent this out, I have 3 computers rendering frames
       | right now, and I can't find more graphics cards anywhere.
        
         | Budabellly wrote:
         | Have you looked at RNDR? Sounds like it could be a perfect
         | match for what you are looking for. It's designed for Octane
         | GPU renderer but there are more render engine supports coming
         | soon I believe.
         | 
         | https://medium.com/render-token
        
         | stickydink wrote:
         | (I'm not super informed in this space)
         | 
         | AWS WorkSpaces purports to have machines with meaty GPU's, are
         | they not powerful enough?
        
           | cameronh90 wrote:
           | The problem with AWS Workspaces for us was the price was just
           | too damn high.
           | 
           | Their Graphics Pro instance was okay spec: 16 vCPU, 122 GB
           | memory, 8 GB VRAM. But they charge $1000/month for it.
           | 
           | Our $3000 workstations beat it in every benchmark, sometimes
           | by 3-5x and have a useful lifetime of at least three years.
           | 
           | I'm sure for some people, that is a valid trade off, but for
           | us, it was just too much.
        
           | Grakel wrote:
           | I'm sure they are, I just need to make the leap. :)
        
           | WrtCdEvrydy wrote:
           | Most cloud GPUs are just thin-sliced datacenter GPUs which
           | only have one redeaming quality: high RAM per PCIe slot.
           | 
           | The $100,000 GPU can be beaten by a $5,000 consumer space CPU
           | for raw compute but is basically 3 to 4 times the amount of
           | RAM.
        
           | skytrue wrote:
           | There's a niche in the space for workspaces that support "low
           | intensity 3D applications" with 99.99% uptime. Right now, all
           | the major cloud vendors (AWS/Azure/GCP) are prohibitive with
           | respect to pricing. I can spend $600 to buy a dedicated
           | machine to run my GPU-bound workload 24 hours a day, or spend
           | $600/month to have a dedicated GPU instance. Things like
           | Paperspace don't cater to this particular workload, still.
           | I've honestly thought about creating a new PaaS provider that
           | attacks this niche. Seems doable.
        
       | nickdothutton wrote:
       | Why not just use Citrix HDX Pro? Tech has been around for ages,
       | works well for accessing virtual desktop, providing virtual
       | desktop with big GPU and IO since it is in a central facility. We
       | were doing this 10+ years ago. What am I missing?
        
         | devoutsalsa wrote:
         | I took a Citrix training class on thin clients back in I think
         | 1999. I spent the entire class wondering if I'd be able to use
         | it to play Starcraft. I never got to try it.
        
       | DTE wrote:
       | We (Paperspace, YCW15) also give you the ability to run a high-
       | performance workstation in the cloud (and soon on any remote
       | machine!). Check it out https://www.paperspace.com
       | 
       | Disclaimer: I am the CEO/co-founder
        
         | skytrue wrote:
         | I've tried to use Paperspace in the past, but the leap in
         | pricing (for Core) from $60/month for "2D applications" to
         | $120/month for "3D applications" is too high. I have a low
         | intensity, always-on (24/365) 3D application that doesn't
         | require a powerful GPU, but nonetheless requires compute that
         | allows for graphical workloads. I would love to use Paperspace
         | for our case, but I currently can't. :/
        
         | brasetvik wrote:
         | I tried Paperspace and Blender would just crash the instance
         | when rendering the splash screen demo files.
         | 
         | Would really like a service that works well for casual Blender
         | usage. Vagon didn't crash, but was too slow, even though the
         | ping was in the low tens of ms.
        
       | leetrout wrote:
       | This is really cool! I would love to demo this- I regularly used
       | Maya via Google Remote Desktop at a previous job and it worked a
       | lot better than I expected. We're headed back toward thin clients
       | :D
       | 
       | I've not used Salt very much and went all in on Ansible years
       | ago. I assume they're running it in master-minion mode but the
       | article doesn't say... Maybe not and that's why they need a
       | custom agent as well?
        
       | egypturnash wrote:
       | I'm an artist and I do fine with a 13" laptop and a Wacom tablet
       | that fits next to it in my laptop bag, to be honest. I consider
       | myself to be pretty discerning about my tools.
       | 
       | If I was doing a lot of 3d stuff I might want more horsepower but
       | I'm pretty happy to be able to go sit out in the park and draw
       | with all the Internet off to conserve power.
       | 
       | YMMV, obviously.
        
         | Cullinet wrote:
         | Keyshot and Redshift have put PIXAR level rendering within
         | reach of anyone with a modern GPU and can make great use of 64
         | core AMD now too.
        
         | whywhywhywhy wrote:
         | It's honestly just unpleasant working over a laptop when your
         | work is video or 3D. It's the difference between working over a
         | hot jet engine or just a gently humming tower.
         | 
         | Not to mention some 3D/video workflows are just unfeasible with
         | the lack of power available in laptops.
        
         | Cullinet wrote:
         | actually more horsepower isn't necessarily helpful - I was
         | reading earlier today about a point cloud architecture
         | application that preallocates larger jobs by guessing the
         | capabilities of the multiprocessor available. This isn't a low
         | end application but a banner product for Leica Geosystems. the
         | reviewer could only get it to slice the job into 8 sections
         | despite many more cores available.
         | 
         | in filmmaking, personal equipment rental companies like
         | sendgrid have raised the level of everything now I can buy my
         | own Red and lenses and learn my tools intimately and pay for
         | them (when I'm not paid to bring them on set) by rental
         | customers who are liberated by the freedom from having to
         | deposit the replacement value with traditional rental houses
         | which risk is now insured. nvidia should sell a licence to use
         | the 3090 professionally and rent it out like you can with the
         | cards formerly known as Quadro now RTX A* beginning RTX A6000
         | (48GB VRAM at under $3K list is incredible if you can get one
         | 96GB at 1TB/s+ linked..) if they really want to take the visual
         | graphics industry for good. (equally AMD can grab this business
         | forever just as easily with a turn on a penny worth of
         | engineering)
         | 
         | Architecture and geographical systems are strongly and
         | culturally averse to parallel programming. the complexity
         | involved has only recently been significantly helped by
         | multiple cores - the workstation on my desk in 2005 was dual
         | dual core 3.9GHz xeon. 32GB RAMBUS RAM. early solid state swap.
         | (and four FC drops from a SAN shared array with enough RAM
         | cache to hold geometry models)
         | 
         | and laptops can be made to run single threaded codes very
         | quickly single core, satisfying the (self imposed limits of)
         | potential for most "graphics heavy" programs. But it isn't at
         | all a good situation.
         | 
         | Ed removed confusingly placed sentence
        
           | sbierwagen wrote:
           | >in filmmaking, personal equipment rental companies like
           | sendgrid
           | 
           | Sendgrid rents camera equipment now? The email company?
        
             | ziari wrote:
             | I think they meant to say ShareGrid.
             | 
             | https://www.sharegrid.com/
        
       | beardicus wrote:
       | I'm very curious what remote display tech they're using, as I've
       | had trouble with latency even over a local network. I've tried
       | VNC and NX, mostly with Linux servers and Mac clients.
        
         | jbaiter wrote:
         | Judging from the GIF, they're using AWS NICE DCV:
         | https://docs.aws.amazon.com/dcv/latest/adminguide/what-is-dc...
        
         | mroche wrote:
         | As stated in a sister comment, it looks like Netflix is using
         | DVC NICE for their VMs.
         | 
         | For the vast majority of this industry, the two larger players
         | are Teradici with their PCoIP protocol (also licensed to
         | VMware) and HP Remote Graphics Software (now known as ZCentral
         | Remote Boost). Teradici is definitely the most widely used, but
         | I know a couple of shops using RGS. NICE is an interesting
         | project coming up in AWS, particularly since it's free for
         | systems within the AWS cloud, and available outside of the
         | cloud for a price.
         | 
         | I don't have any HP RGS experience myself, but with Teradici it
         | currently supports Windows and Linux hosts, with software
         | clients for Windows, Linux, and macOS. They also have hardware
         | solutions (zero-clients) available so end-user operating
         | systems aren't needed. Those clients are provided by third-
         | party OEMs (e.g. 10Zig, Dell, etc) using the Teradici supplied
         | chips[0]. macOS host support is coming later this year[1]. The
         | host support is split up into two different categories:
         | hardware with the Remote Workstation Cards which have a
         | hardware encoder on them (uses a PCIe slot) that you plug
         | display output into, and their software agents (standard and
         | graphics) that don't require hardware and encode on the CPU
         | and/or GPU depending on the agent you're using.
         | 
         | Teradici has also been fairly clear that software is the future
         | of the product line, so the hostcards are not a good buy for
         | long-term support. However they will be continuing the hardware
         | clients with a "next-gen" zero-client coming out this summer
         | with a boatload of new features over their older clients, but
         | it's not really a zero-client in the truest definition of the
         | word.
         | 
         | [0] https://www.teradici.com/resource-center/product-service-
         | fin...
         | 
         | [1] https://connect.teradici.com/blog/macos-release
        
         | Cullinet wrote:
         | HP Remote Graphics is very popular and widely deployed in post
         | production and artist graphics suites.
         | 
         | Azure has been taking high end graphics workstation seats in
         | the GFX and compositing and film production industry at a clip
         | for close to a decade now.
         | 
         | the hourly cost of running PS30,000 worth of hardware is below
         | PS5ph, which pays for the hardware running 24/7/365 over 3
         | years, but is clearly a saving serving 5*8/7 active employees.
         | [ed] and that's before factoring in floating licence economies.
        
         | Miraste wrote:
         | I don't know anything about NX, but VNC is terribly slow and
         | outdated. Pretty much anything else will be faster.
        
         | akhilcacharya wrote:
         | At an internship we heavily used Windows RDP over our local
         | network - worked phenomenally well, basically 1:1 when
         | connecting to our "home desk" desktop. I think RDP + very high
         | bandwidth network is the best way to do it still.
        
         | lame88 wrote:
         | I was thinking the same thing as I was reading. Seems like
         | feedback lag while drawing brush strokes would be frustrating
         | for an artist. But then again, we have examples of surprisingly
         | good response time in the game streaming world.
        
           | mroche wrote:
           | Like anything it's going to heavily depend on latency to the
           | workstation in question. With cloud providers that have
           | reasonable region coverage, it's not really that bad. In our
           | tests, ~40ms latency is the upper bound of where you want to
           | be before artists start really complaining, and Teradici
           | themselves suggest 25ms for optimal results. However,
           | depending on the task said artist is doing, you can venture
           | above that, though for Cintiq/pen displays it becomes pretty
           | painful and a "live with it" type deal.
        
       | endisneigh wrote:
       | What I don't understand is how we're able to livestream out video
       | in 4K but can't share our desktop at the same speed and
       | resolution.
       | 
       | Am I missing something?
        
         | wongarsu wrote:
         | Those 4k livestreams have a latency measured in seconds. We
         | have made massive advances in recent years and if you have good
         | bandwidth you can now have 4k livestreams with maybe 2-4
         | seconds of latency. But for moving a mouse that's still one to
         | two orders of magnitude too much.
         | 
         | If you allow less latency you can't do as good a job at video
         | encoding, and can't allow for any buffering (client side or
         | encoder side) to smooth out bandwidth fluctuations or allow for
         | more dynamic bitrates. That drastically lowers the achievable
         | quality.
        
         | thatcat wrote:
         | the video is pre-cached by your ISP
        
         | digikata wrote:
         | Not that it isn't an underdeveloped area, but the needs of the
         | streaming cases are different. One key difference is if there
         | is no feedback/interaction on the livestream, then there could
         | easily be a relatively long time buffer that would not be
         | noticed. Vs moving a mouse pointer around on a desktop where it
         | would get annoying at a few hundred milliseconds. It also leads
         | to differences in how the encoders and decoders need to be
         | applied.
        
       | mcoliver wrote:
       | As mentioned in a few of the comments here the gold standards for
       | desktop streaming are Teradici, HP RGS (now zcentral remote
       | boost), and NiceDCV (acquired by amazon). All are limited to
       | Windows or Linux (sorry macOS you'll have to use something like
       | Jump although teradici did just announce they are working on a
       | macOS product for release this year:
       | https://connect.teradici.com/blog/macos-release). Parsec recently
       | came out with a Teams offering that is pretty slick.
       | 
       | There are also some interesting application streaming services
       | out there like fra.me, cameyo, aws appstream, etc..
       | 
       | Fundamentally remote workstations come down to interactivity. If
       | you assume a 60Hz monitor (60 cycles a second), whenever I move
       | my mouse or interact with an application I need all the
       | computation and draw calls to happen on the computer and
       | return/display the modified pixels in 16ms. So that's all sorts
       | of latency but when you are talking about a remote computing
       | environment the largest component will be your network latency.
       | Now humans are incredibly adaptable and so you can go beyond that
       | but the further you get from 16ms the more your brain will have
       | to compensate for the delay. Data can travel around 122 miles per
       | millisecond (2/3rds C). But with all the network switches and
       | things in between it is not a "google maps" type calculation.
       | 
       | Remote workstations have a few advantages and disadvantages.
       | 
       | Advantages include easy provisioning, secure access to data and
       | assets, centralized backups, access to high speed networking for
       | large data sets, easy system config updates (add storage, add
       | gpus, etc..), opex not capex, global systems deployment without
       | going through customs, immutable desktops, etc..
       | 
       | Cons include expensive ($1000/month for a windows workstation.
       | Linux is much cheaper but Adobe and Avid make Linux hard for some
       | creative workflows), limited to hardware provided by cloud
       | provider (quadro's not geforce, slower clock speeds affect single
       | threaded applications, inability for other addon cards like
       | BlackMagic / AJA), limitations for working with HDR, high
       | resolution (beyond 4k), multichannel audio, multiple monitors, VR
       | headsets, require a stable and consistent connection, etc..
       | 
       | For certain creative applications the flexibility of remote
       | workstations is really great. For example if you need to access
       | 100TB of raw video footage and 3d assets. For others (for example
       | where you are painting on a canvas, need to hook up a peripheral
       | like a wacom tablet, or need to hook up a VR headset), they can
       | be non optimal depending on the user, video drivers, application,
       | etc..
       | 
       | There is also an emerging hybrid coming out which leverages
       | browser native applications so you get the local interactivity
       | but offload the processing and assets to a backend in the cloud.
       | Things like Storyboarder, Photopea, or the Blackbird Editor.
       | 
       | Teradici got there start over a decade ago primarily shipping
       | ASICs that would connect to the video out of your GPU via
       | physical cable, encode, and then stream over IP to a zero client
       | (Wyse, eVGA, 10zig) that had another ASIC that would decode.
       | Popular in the military and other areas where you want to provide
       | a terminal. They excelled over other stuff like NX based tools or
       | RDP for a couple reasons: 1) Full screen video playback 2)
       | Optimized protocol for crisp text gives it an edge over the
       | others that are x264/x265 streaming. Teradici will actually
       | dynamically modify their stream based on what you are doing. So
       | working in a DCC you get their crisp text goodness, but if you
       | switch to full screen video they will switch over to x264/265 3)
       | USB bridging for peripheral support 4) Local termination of the
       | mouse cursor. So while draw calls may be delayed (think a
       | trailing line when drawing), the mouse cursor is snappy which
       | helps with latency adaptation.
       | 
       | The most recent version has gone all software and is actually
       | what powers AWS Workspaces product. They also now support 4k and
       | 10 bit color with their Ultra product. Audio is still limited to
       | stereo.
       | 
       | NiceDCV was acquired by Amazon a while back and while it is
       | "free" for ec2 instances, and does support 7.1 audio on Windows,
       | there are some things it doesn't do as great some of the others.
       | 
       | Parsec was originally focused on the remote gaming space and I
       | remember reading about it on reddit years ago where people were
       | using it with Paperspace to run games remotely. Their newish
       | Teams product is pretty cool moves them from a single user type
       | use case to definitely worth a look for enterprise deployments
        
       | sxp wrote:
       | For people who want to roll their own system, I suggest following
       | https://www.reddit.com/r/cloudygamer/. That's focused on
       | commercial game streaming services, but there are many threads
       | about people who roll their own game streaming system.
       | 
       | The low-latency high-quality requirements for game streaming is
       | very similar to the requirements outlined by Netflix in this
       | post.
        
         | TheRealDunkirk wrote:
         | I don't know who's actually using these kinds of game streaming
         | solutions, or what games they're playing with them. The PSN
         | version of the idea is basically unusable for FPS games, and I
         | have a good broadband connection. Even Steam, over the LAN,
         | Steam introduces a noticeable-enough lag that I just don't
         | bother. But things like Stadia make it seem like enough people
         | are getting mileage out of these solutions to make it
         | profitable to run, so I guess I'm not the target demo.
        
           | Deathmax wrote:
           | There are enough people that just aren't that sensitive to
           | latency. For reference, look at any console player that
           | doesn't go out of their way to enable PC/game mode on their
           | TVs (HDMI 2.1 with Automatic Low Latency Mode is thankfully
           | changing this) that is going to suffer upwards of 100ms of
           | input latency[1]!
           | 
           | [1] https://www.rtings.com/tv/tests/inputs/input-lag
        
       | simon_acca wrote:
       | Sounds like a nightmare for the artist compared to the status
       | quo. Specially, I could imagine that the thought of your
       | workstation being ephemeral does not sit well with somebody that
       | is not used to the sophisticated version control tools available
       | to software engineers
        
       | kbar13 wrote:
       | i'm not really in the workstation provisioning industry so i
       | can't comment on the article, but i've used salt, puppet, etc in
       | provisioning servers and have found that declarative
       | configuration management is surprisingly complex and has a lot of
       | pitfalls and is deceptively hidden behind the "look how simple it
       | is to install a package". declarative languages for config
       | management imo was a mistake, because in order for it to be truly
       | declarative, every part of the system (including preinstalled
       | software) needs to be managed by the config management tool.
       | otherwise stuff like "i pushed a diff to remove the block where i
       | asserted this package to be installed but the package is still
       | installed on the hosts" will continue to surprise newish users.
       | stuff like cdk works because, as stated earlier, it asserts the
       | desired state of every piece of infrastructure it manages
       | 
       | a better alternative would instead be something akin to
       | Dockerfiles where you assert what packages you want to be
       | installed on top of a base system, and if you want anything
       | changed you update the dockerfile and rebuild the whole shebang.
       | 
       | i'm imagining a system where the local filesystem is more or less
       | immutable and work is saved to separate filesystem. but having
       | worked from computer labs in schools, i know how less than ideal
       | that is when i need to use a program that isn't preinstalled to
       | do my work, so maybe this all doesn't work and saltstack is a
       | necessary evil
        
         | outworlder wrote:
         | > a better alternative would instead be something akin to
         | Dockerfiles where you assert what packages you want to be
         | installed on top of a base system, and if you want anything
         | changed you update the dockerfile and rebuild the whole
         | shebang.
         | 
         | Something like Packer?
         | 
         | You are right that there are hidden pitfalls. But they
         | shouldn't be a great deal if servers are cattle. A lot of
         | corner cases are related to trying to manage servers that live
         | forever. This shouldn't really be done in 2021 except in rare
         | cases.
         | 
         | > i'm imagining a system where the local filesystem is more or
         | less immutable and work is saved to separate filesystem
         | 
         | CoreOS worked like that. Immutable filesystem for the system
         | itself. It also had a secondary "mirror" of this filesystem
         | that was used for upgrades (and as a fallback in case boot
         | failed). New stuff would either come when the instance first
         | booted up (cloud-init), or via system upgrades.
         | 
         | They were acquired and so my experience with them ended there.
         | 
         | There's also https://nixos.org/
        
       ___________________________________________________________________
       (page generated 2021-03-08 23:01 UTC)