https://www.nitrokey.com/news/2021/nextbox-why-we-decided-and-against-ubuntu-core * EN * DE * FR * RU Nitrokey | Secure your digital life * Company + About Us + Contact + Career + Press + Community Program + Affiliate Program * News * Products + Nitrokeys + NitroPad + NetHSM + NextBox + NitroChat + Nitrokey Business Subscription + Nitrokey Business Fulfillment + NitroShred * Solutions + Success Stories + Passwordless Login and Two-Factor Authentication + Secure Administration of Servers and IoT With SSH + Phishing Protection + Crypto Currency Exchanges and Bitcoin Startups * Support + Installation + Applications + FAQ + Forum + Download * Shop NextBox: Why we Decided for and Against Ubuntu Core [Ubuntu_log] When we started developing NextBox, we decided to use Ubuntu Core as the operating system for the integrated software. At first glance, Ubuntu Core seems to have many advantages, but unfortunately they disappeared bit by bit and even became problems. Therefore we had to revise this decision and changed the system to Debian. This article describes the reasons that led to this. Requirements for the Operating System The NextBox is a hardware device with Nextcloud pre-installed, which is also aimed at users with little to no technical understanding, but who still want to run a secure system under their control. This results in the main requirements for the operating system used in it: * Open source * Automatic updates and high robustness for minimal administration effort * Sole control by the user. No remote access by us should be necessary or possible. * As long as possible security updates and ideally the updatability of the entire operating system. * A system as small as possible to provide a minimal attack surface. * The possibility to add further functions (e.g. status LED, reset button, secure element). At the same time we don't want to reinvent the wheel by developing an operating system (or distribution) from scratch, but we want to use an existing operating system that meets the above requirements. Why we Chose Ubuntu Core The open source landscape for operating systems is enormously diverse. Not entirely uninfluenced by our own positive experiences with Ubuntu as a reliable, low-administration operating system, Ubuntu Core appeared on our radar relatively early. The overview of Ubuntu Core lists exactly the features we need for NextBox: "Security first", "Tamper-resistant and hardened against corruption", "10 year security update commitment", "Minimal core, minimal risk, minimal bugs". In addition, support for the hardware platform of our choice: The Raspberry Pi 4. Of course, the classic criticisms were already present here, especially the fact that the Global Store is the only way to get snaps onto the device. For NextBox, the associated security mechanisms even appeared to be an advantage. Obviously, the restriction of Ubuntu Core to exclusively install snaps is a double-edged sword; For our requirement profile, however, basically good and desirable in terms of security and robustness. At this point, however, we had already noticed that a so-called Brand Store is an (apparent) option for using Ubuntu Core for such a project. The documentation describes a Brand Store as an additional feature for larger projects and software vendors. Since we offer NextBox without subscription costs, such a Brand Store with an annual cost of $30,000 is not economical for us and therefore not an option. By means of custom images we seemed to be able to do without the Brand Store, thus problem solved. Of course, we had also investigated other options at this point, but more about that later... Our Experience With Ubuntu Core As soon as the decision was made, the development of a proof-of-concept began immediately. In the process, numerous advantages of the snap ecosystem quickly became apparent: * There was already a Nextcloud snap, as well as other snaps we needed. * Building an Ubuntu Core image went well, including the required snaps. * Images work fine on our target hardware * Ubuntu builds from our own snaps can also be used on other platforms Accordingly, we had a proof-of-concept developed and successfully tested in just a few days. We thought, "Wow, this is going to be good, 10 years of security updates, nice ecosystem, perfect!" Of course, it was already visible at this point that we have to adapt to this ecosystem, but we were happy to accept this in view of the rapid progress. The euphoria did not last long and the first setbacks were not long in coming: * NextBox must have the ability to integrate external hard-disks into the system. For this, udisks2 was proposed as a solution until at least the end of 2020, but this did not work without problems. (Today's Ubuntu Core 20 documentation no longer recommends udisks2.) * A snap is understandably subject to numerous restrictions regarding system permissions. The solution to this are so called interfaces, which at first sight seem to be a quite charming solution: A snap announces the interfaces it needs and can use them as soon as they are connected. At this point, we did not perceive this as a real problem, but rather as a typical limitation of such an ecosystem. * The ecosystem and the freely available tools did not allow us to create a deliverable image. Connecting the interfaces turned out to be a bigger problem. The plan was to use the tools provided by Canonical to create an Ubuntu Core NextBox image that would be written to an SD card and shipped with the Raspberry Pi 4 in the NextBox. To realize this, it was necessary to add a gadget snap to the image. No sooner said than done: forked the appropriate gadget snap for the Raspberry Pi, made adjustments, posted it to the Ubuntu Global Store for building. The aforementioned gadget snap was successfully built, but it could not be added to the Global Store: "manual review required". Shortly after, the first contact between us and Canonical occurred. Two salesmen told us in a phone call that this was only possible with a paid Brand Store. We pointed out that we, like Canonical, are an open source company and do not earn enough money with the product to pay the costs for the Brand Store of $30.000 per year. Unfortunately, we couldn't find a compromise and were shrugged off. "Hey, this is a setback, but this is open source, so let's find a solution!" we said to ourselves as we developed a solution we called a "hack." Essentially, the idea was to use our snap package to get the needed functionality into the system. The generated image is then booted once and a systemd service is installed which runs the functions brought in by the snap as a root process. This essentially removes the snap restrictions and allows us full access to everything you can otherwise only use with a "Brand Store". Basically, this was our forced solution for a long time and we had pushed the product with it, not quite as intended, but "hey, we get 10 years of updates" and we did our best to reduce the risk of the hack as much as possible. With this approach we were able to successfully implement all the essential features such as DynDNS, USB access, and backups. Canonical Takes Notice A few weeks later, NextBox was a successful Kickstarter project. Canonical contacted us and wanted to discuss the project technically. Of course we agreed immediately, hoping to work together to improve the product and maybe even get rid of the hack. A short time later, we talked to two software engineers from Canonical. The main findings were the following: * The promised 10 years of support is just "good marketing" and actually there are only 5 years of support. Another 5 years are only available with the paid Brand Store. This would mean that we could easily get only 3 years of security updates for Ubuntu Core from today. * Ubuntu Core 18 cannot be upgraded to Ubuntu Core 20 without already preparing it specifically (undocumented "feature"). We could upgrade our software to Ubuntu Core 20 before delivery, but the Nextcloud snap package we use only exists for Ubuntu Core 18. To upgrade to Ubuntu Core 20 later, we would have to work on a complicated device upgrade in the field. * Canonical wants to support us to remove the hack and motivates us to do so as well. * The hack has the central problem that all devices in the Global Store are identified as the same device and "this can cause problems". Oops, that was a hard blow for us! Still, there was a constructive "we can do this" mood. Information was exchanged and we felt there was a way: without hack, with Ubuntu Core 20 and a bunch of work. Knee-deep in the Ubuntu Core ecosystem, we decided, "Ok, we're going to do this right. It'll be fine, support is there, let's go!" With the new architecture envisioned by Canonical, the first hurdles were not long in coming: * There is no way to mount an external hard drive (Deja Vu!). * Interfaces can only be connected automatically with a gatekeeping process maybe. Or you just have a gadget snap, that is a paid Brand Store. * Interfaces like GPIOs, I2C etc. can apparently only be used in a Brand Store at all. This is relevant for us to be able to realize further hardware features in the future. Fortunately Canonical answered us that they are checking if we are allowed to place a gadget snap in the Global Store for free. That would solve the problems. Why we Decided Against Ubuntu Core Since then we haven't heard anything from Canonical. Of course, Canonical never made us any promises or reliable commitments. Nevertheless, the tenor of the conversation was that Canonical wants to support us to make the NextBox with Ubuntu Core right. Unfortunately, this has not been realized so far. Finally, only one conclusion remained: It does not work without Brand Store. This paywall keeps popping up everywhere. Even if we got a free gadget snap in the Global Store, Canonical would certainly not give us any assurances as to how long they would allow this. Accordingly, the risk for NextBox users would be that at some point in the future, Canonical would revoke this privilege from us, making NextBox un-updatable from one day to the next, or at worst, unusable. The fact that the 10 years security updates advertised on the front page is, to say the least, well-placed marketing, and without $30,000 per year is simply not true, was the biggest disappointment. This is simply not what we expect from one of the open source flagship companies. Even the datasheet on Ubuntu Core only hints this limitation with "up to 10 years". In addition, it became apparent that we had not selected sufficiently strict according to open source criteria. Assuming Canonical would eventually cease to exist or discontinue Ubuntu Core, it would be nearly impossible with Ubuntu Core for the open-source community to ensure that NextBox would continue to be usable in a meaningful way. The restrictive idea of Ubuntu Core shows its dark side here. In addition it should be mentioned that gatekeeping for the Global Store restricts the inherent freedom of open source. Even if we would own a Brand Store, it would be nearly impossible to keep the devices up-to-date or to develop them further through the community. In the end, we had no other option but to turn our backs on Ubuntu Core in order to realize a NextBox that delivers on its promise. The King is Dead, Long Live the Debian So, with this new experience, we again analyzed and compared the available options. As before, the requirements described at the beginning had to be met. Additionally, we took care to avoid a gatekeeper that could potentially make the system unusable (or unmaintainable). Among others, we considered the following systems: Buildroot, Yocto, OpenEmbedded Using Buildroot, a specially tailored system can be built, which consequently has a minimal size. This would provide a minimal attack surface and thus allow for a secure system. However, the configuration effort for a Buildroot image is far from trivial for these very reasons. Likewise, the flexibility of the resulting system is limited and a luxury such as a package manager is deliberately absent. Yocto, or more generally OpenEmbedded, additionally provide a package manager, but the variety and fast availability of security updates fall short of Debian (more on this later). Therefore they are to be classified similarly: Initially very well adaptable, but later losses in flexibility and security updates. The biggest challenge, however, would be to reliably update the entire system. Since with Buildroot we would have the responsibility to evaluate, install and test security updates for each software component used, this would be a high maintenance effort. It is therefore not surprising that many consumer products based on such small systems are inadequately supplied with security updates. balenaOS BalenaOS is clearly on the opposite side of the spectrum in terms of flexibility, simplicity and control. In principle, container orchestration is used here, in addition to a supervisor container. So it's a similar approach to what Ubuntu Core is doing with snap. The ecosystem is also very extensive, especially the broad support of different hardware stands out very positively. In addition, openbalena is a self-operating alternative to the commercial balenaCloud, which is slightly limited, but already offers the most important functions as an open source solution. This shows most clearly why NextBox is not an IoT product, at least not in the interpretation of Canonical or Balena: openbalena allows, among other things, complete control over the devices in the field via SSH login. This is clearly not what we want to offer our customers. In the end, the NextBox should be a device with minimal administration effort, but by no means controlled by us. Debian Debian offers the advantage of an enormously large community and complete openness. It has high similarity to Ubuntu Core and thus is closest to the user experience of Ubuntu Core, without the limitations described above. In terms of system size and flexibility, we rank Debian pretty much between balenaOS and Buildroot: We can use Debian packages and additional containerization to ensure that the device is robust and secure. At the same time, we do not have direct control over the device. Still, it is possible for the user to access the device and install additional packages. Compared to Ubuntu Core, Debian is certainly very stable to run over a longer period of time, including automatic security updates. Specifically, Long-Term Support for Debian Buster is scheduled until June 2024, and usually another 2 additional years until 2026 can be expected for Extended-Long-Term Support. Accordingly, without our intervention, NextBox will be provided with security updates for about 5 years from today. This is 2 years more than with Ubuntu Core 18. Our goal is to offer a mostly automated possibility to switch to the next Debian version latest at the end of the Long-Term-Support and thus to get another 5 years of security updates. With Ubuntu Core a corresponding upgrade possibility from Ubuntu Core 18 to Ubuntu Core 20 would be unclear, since this is not planned by Canonical! We use Docker as the core technology for deploying Nextcloud and its dependencies, which gives us similar control and confinement as Ubuntu Core's snap approach. At the same time, we are far more flexible and there is no gatekeeper to use GPIOs for example. We build a similarly robust system as Ubuntu Core by following similar concepts. In doing so, the user benefits from the flexibility introduced with Debian: * Minimal write access to the SD card by offloading write-intensive directories to the internal hard drive or SSD. * Specific version combinations of the containers used allow version jumps to be tested in advance and rolled out as a whole with the NextBox Debian package. * Different NextBox packages represent different levels of stability: stable, testing, unstable. * Unattended upgrades provide security updates and update the NextBox Debian package. Due to the good integrability of apt, we could also offer to install system software directly from the Nextcloud NextBox app in the future. Ubuntu Core is advertised as "minimal core, minimal risk, minimal bugs". A comparison with Debian is difficult because it is not clear which system areas are comparable. Nevertheless, we were able to build a comparably lean system with Debian: The base systems of both Ubuntu Core and Debian each have a total size of about 1 GB. The entire functionality of NextBox can now be provided by a Debian package, which brings numerous advantages such as testability and platform independence. While the latter is also possible with a snap-based approach, it is only possible with a Brand Store. From a developer's point of view, it is an enormous relief to no longer squeeze into the Ubuntu Core ecosystem. But of course this also means more responsibility, since there is more freedom in general and of course it has to be used properly. Since our system architecture, the so-called hack, follows common Unix methods, the conversion to Debian took only a few days. Summary Ubuntu Core and balenaOS seem to be aimed more at high-priced systems where the manufacturer (better: operator) retains control. Presumably, this is only suitable for high volume consumer products with a subscription pricing model. These systems are better suited to e.g., industrial systems and machines. Specially tailored systems, which can be built using Buildroot, for example, seem to make more sense for small hardware. In addition, the maintenance effort is not insignificant. On a relatively powerful hardware like the Raspberry Pi (the NextBox), the additional effort seems unnecessary to us. With Debian, we have a system where we get the services promised by Ubuntu Core with little effort, which we need for the NextBox: small system size and high security, robustness, many years of security updates, good hardware support and full freedom. 10.3.2021 Comments Submitted by Jorn on 10. March 2021 - 21:15 Ich hatte es gut gefunden, wenn Ihr euch fur eine eurpaische Distribution entschieden hattet, z.B. openSuSE. Klar habt Ihr einige valide spezielle Anforderungen, aber vll. kann man mit solchen Partnern besser arbeiten. Ausserdem gibt es halt europaische Rechtssicherheit und europaische Software. Die Updatefahigkeit fur die nachsten 10 Jahre halte ich Allgemein fur unrealistisch. Besser ware die Hardware nach 5 Jahren zuruckzunehmen und zu recyclen und dann muss gegen neue Hardware und OS getauscht werden. * reply Submitted by Anonymous on 10. March 2021 - 23:03 Welchem Staat ordnest du debian denn zu? Was ist ,,europaische Software" im Kontext von freier Software? Sollte der Kernel (Linux) ausgetauscht werden, weil Torvalds (offizieller Hauptentwickler) in den USA lebt? Jetzt zu planen, alle NextClouds in 5 Jahren zu verschrotten, halte ich fur mindestens fragwurdig. Klingt nicht besonders nachhaltig, direkt zu sagen ,,mehr als 5 Jahre uberlebt es sowieso nicht". Verschrottung und Ersetzung in 5 Jahren klingt auch sehr unokologisch. * reply Submitted by Jorn on 10. March 2021 - 23:18 Vielleicht war es etwas missverstandlich ausgedruckt. Es geht um einen europaischen Geschaftspartner - nicht darum das alles in Europa entwickelt werden muss. Klar Debian und etwas eigenes machen ist auch vollkommen valide, wenn garantiert ist das es weiterhin aktualisiert wird. Es ist wie gesagt nichts gegen die Entscheidung selber, die sehr gut im Blog begrundet wurde und auch vollkommen ok ist. Recyclen ist ebend nicht verschrotten, sondern wiederverwenden! Und alte Hardware ewig laufen zu lassen kann auch zu Nachhaltigkeitsproblemen fuhren. Ich finde man muss da nicht auf die 10 Jahre fixiert sein. Auch hier gilt, dass dies alles im Blog gut begrundet wurde. * reply Submitted by Thomas McWork on 12. March 2021 - 14:28 @Jorn: Den Aspekt eines Geschaftspartners konnte ich aus dem Text gar nicht erkennen. Viel mehr wurde ja auf die eigene Verantwortung verwiesen. Bei tatsachlich auftretenden Problemen konnte man sich ja immernoch an einen deutschen oder europaischen Partner wenden. Wurde mich wundern, wenn es da nicht mindestens eine Hand voll Systemhauser gibt, die das anbieten. Und beim Thema Nachhaltigkeit wurde ich eine moglichst lange Nutzung einer Nachverwertung vorziehen. Es wurde in Studien mehrfach nachgewiesen, dass eine lange Lebensdauer fast immer nachhaltiger ist als eine Neuanschaffung. Grund dafur ist der hohe Energie und Ressourcenverbrauch bei der Herstellung. @nitrokey Ich finde, dass das ein schoner Artikel ist, weil ihr damit genau eure Hoffnungen und Schmerzpunkte beschreibt. Ich hoffe, dass ihr damit die richtigen Leute erreicht, die vor einer ahnlichen Entscheidung stehen. * reply Submitted by Thomas on 14. March 2021 - 13:38 Ich finde den Ansatz gut und danke fur den ausfuhrlichen Artikel. Evtl. konnte man sich im Docker-Umfeld auch noch einmal mit Podman auseinandersetzen. Hier gibt es auch die Moglichkeit einen Pod (wie in Kubernetes) allerdings im User-Mode laufen zu lassen. Eine Verwaltung konnte man z. B. mit Cockpit erreichen. Das Letztere wurde das System allerdings ein wenig aufblahen, wobei man dem Nutzer die Installation auch selbst uberlassen konnte. Als kleinere Schaltzentrale ist das ein nettes Werkzeug und man konnte die Installation neuer Anwendungen daruber ein wenig vereinheitlichen. So liesse sich z. B. auch die GPIO/I2C-Anwendung in Podman von NextBox getrennt entwickeln und bereitstellen. Wenn man dann noch alles im User-Mode z. B. unter dem nextbox-User laufen lassen konnte, ware eine super Trennung vom restlichen OS-Umfeld erreicht. * reply Add new comment Your name [ ] Comment * [ ] [ ] [ ] [ ] [ ] First name of the actor Chaplin? * [ ] Fill in the blank. [Save][Preview] [footer_log] Newsletter [ ] [Subscribe] Keep in Touch TwitterMastodonGitHubFacebookNewsletterBlog FeedYouTubeLinkedIn Contact/Legal * Terms and Conditions * Data Privacy Policy * Legal Information * Contact * Withdrawal Be The first to know Stay informed about the Nitrokey project, new models and firmware updates. Enter your E-mail address: [ ] [subscribe] Confirm e-mail address Please re-enter your email address [ ] [Ok] * Company + About Us + Contact + Career + Press + Community Program + Affiliate Program * News * Products + Nitrokeys + NitroPad + NetHSM + NextBox + NitroChat + Nitrokey Business Subscription + Nitrokey Business Fulfillment + NitroShred * Solutions + Success Stories + Passwordless Login and Two-Factor Authentication + Secure Administration of Servers and IoT With SSH + Phishing Protection + Crypto Currency Exchanges and Bitcoin Startups * Support + Installation + Applications + FAQ + Forum + Download * Shop Nitrokey - Made in Germany [piwik]