From /etc to database reddit / BSD ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 15:45 I know that it’s not the Unix way, but has anyone tried storing all system settings in a database & have a database driver load at boot? This would eliminate the need for /etc. If anyone has done this, I’d be very interested in hearing about it. ---------------------------------------------------------------------- losthalo7 · 2023-07-08 15:54 What's the advantage to this approach? ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 15:59 Here are the benefits that I can identify (I’m not sure what others would see as benefits): 1. A uniform standard of how settings are stored 2. A uniform way of accessing them. 3. A setup that prefers settings to be changed programmatically, rather than manually. ---------------------------------------------------------------------- xzk7 · 2023-07-08 16:50 I could see the benefit in an embedded system or some kind of appliance where you want to provide an admin with a unified dashboard for setting configuration. One struggle, if you ported the entire system to using the database would be maintaining it and syncing changes with upstream. You'd essentially be forking the OS at that point. Once the base system has been changed to use a DB you'd then probably want to do the same with any 3rd party apps too (web servers, etc.) which may also be a pretty big effort and would be a lot of work to stay in sync with upstreams which is why I suspect most systems just generate config files. ---------------------------------------------------------------------- jantari · 2023-07-08 16:58 Windows has done it and calls it the registry. It works very well. A big advantage you didn't mention is native support for data types. The windows registry supports strings, 32-bit ints, 64-bit ints, binary data and a few more less frequently needed ones: https://learn.microsoft.com/en-us/windows/win32/shell/hkey-type This means when you query the configuration, you get back strongly typed values. Technically some of this is possible with structured text as well, but not as efficiently especially for binary data blobs. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 16:58 Sure, but who’s done it in the BSD world? ---------------------------------------------------------------------- TheFlean · 2023-07-08 17:58 Windows has it. It’s called the registry. Do you like it? ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 17:59 It’s odd that some of you can’t comprehend the obviousness of me asking about whether or not it’s been done on a BSD system -NOT Windows. ---------------------------------------------------------------------- TheFlean · 2023-07-08 18:02 Services accessing /etc expect to interact with files and only know their flavor of syntax. How do you want to do that if your own driver? Most things are just files and I believe that it’s better than Windows concept of a registry. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 18:05 Any BSD system typically comes with its own services, since it’s usually a complete OS. If any BSD project has adopted this idea (even if it’s just a research project), then they undoubtedly would have modified their services to use this setup. ---------------------------------------------------------------------- jantari · 2023-07-08 18:30 Noone that I'm aware of. Frankly if I can imagine any project making a change this drastic it's systemd under Linux. ---------------------------------------------------------------------- TheFlean · 2023-07-08 19:11 But every application knows how to read conf files. Why would you want to change that. Conf files are awesome and readable. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 19:16 1. No, not every application knows how to do this. And those that do normally aren't doing it in a uniform way. 2. "Awesome" is highly subjective & has no objective value. 3. Readable by who? Humans? Humans only need to read config files to fix mistakes that humans made to the config files. Configurations managed through APIs are less error prone. ---------------------------------------------------------------------- sylgeist · 2023-07-08 19:31 Check out AIX (IBM’s Unix). Not to say you would copy the implementation, but the concept does make it very easy for documentation and predictability ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 19:33 That's a great idea. Looking outside of the BSD community might be my best chance at seeing such a thing working successfully. ---------------------------------------------------------------------- sylgeist · 2023-07-08 19:39 AIX is definitely odd, but I did like working with the consistency aspect of it. I would love to see some of those elements come over. I think it would be an improvement! ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 19:40 I agree. ---------------------------------------------------------------------- m1k3e · 2023-07-08 19:43 Took the words out of my mouth. My suggestion would have been to call it the Registry 🙂 Teasing obviously, but it’s always good to think outside of the box. ---------------------------------------------------------------------- papageek · 2023-07-08 20:08 I think TrueNAS does this for many settings, stored in sqlite db. ---------------------------------------------------------------------- liveoneggs · 2023-07-08 20:19 https://man.netbsd.org/NetBSD-9.3/pwd_mkdb.8 ---------------------------------------------------------------------- TheFlean · 2023-07-08 20:44 How do config files get configured? By a darn human. So it better be readable. Awesome was my take on it, that’s why it is the last sentence…. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 21:05 I'll try to make this a bit clearer: Under this system, the configurations would be done via an API, so the config files would not be hand written. ---------------------------------------------------------------------- gumnos · 2023-07-08 21:14 A couple observations: - the Windows registry is roughly what you're describing, and is known for its fragility issues (having corrupted more than one in my life even when doing just basic stuff); Gnome has a similar registry that works within that ecosystem but doesn't seem to have much support outside of Gnome applications - while a tree-like registry/database is a consistent interface, some problem-spaces for configuration don't model well with a tree; and sometimes a custom DSL makes it much clearer. I'm trying to envision a situation in which a collection of database tables is as readable as my `pf.conf` file - and stemming from that, text files often incorporate the ability to comment sections — things like why certain decisions were made, particular gotchas, etc. Commenting database tables/relationships poses a complex a challenge - you could go with a full database like `sqlite`, but then you have issues of schema migration which is one of the main pain-points I experience when working with enterprise & web software (changing the sizes of columns, adding columns, splitting columns, normalizing a single one-to-one value into a one-to-many or many-to-many foreign-key relationship, dealing with NULL values, etc) - when uninstalling, it takes special care to clean up one utility's subset of a database without breaking other programs' areas of the database - text files are easy to version-control, `diff`, `grep`, and merge; databases are notably less so - you'd be looking at modifying every application to read (and some to write) to this new format. That's a **lot** of work. Even if you limit yourself to applications/services in the base system (i.e, just sticking to `/etc` rather than relegating package-configuration to `/usr/local/etc`) There are some conceivable benefits to such a model, but I'm not sure they out way the costs. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 21:18 This is the kind of thing I’m looking for. I didn’t know about this usage. ---------------------------------------------------------------------- infursec · 2023-07-08 21:25 Some other *nix operating systems have endeavored to accomplish the above by migrating to XML-based configuration files in standardized locations. XML has the advantage of existing tools for editing, validation, diffing, pretty formatting, programmatic access, and version control. It also makes atomic rollback of configuration changes more trivial than a database that has multiple programs' configurations in one database file. Another approach that accomplishes some of these is modifying programs' configuration parsers to support "include" directives. Sane defaults can be stored in the default /etc locations, and then /etc/{program name}.d/ contains individual configuration files with exceptions to the default configuration. Both of the above have the advantage of a good chance of being accepted upstream, especially in the second case of adding "include" directives. Reinventing the Windows registry under *nix has a whiff of the "there are too many standards, so let's invent a new one" mentality, requiring new tools and added complexity for admins and developers. Admins and devs will need to know the old standards and one more new one during a lengthy transition period. It could also lead to port maintainers having patches that forever have to be developed and applied independently of upstream project sources. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 21:32 This sounds like another great approach. ---------------------------------------------------------------------- TheFlean · 2023-07-08 21:46 I understand it. But what would be the benefit. You’d have to write a system that stores the files differently but still expose them in the old gashed way. There’s no benefit I can see and it would definitely be slower. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 21:49 I’ve already written the benefits above under another comment. I wouldn’t expose them in the same way, because there wouldn’t be anymore config files -that’s precisely what I’m trying to get away from. I doubt it’d be slower. ---------------------------------------------------------------------- compuwar · 2023-07-08 22:54 ODM issues were the *worst* part of dealing with AIX IMO. ---------------------------------------------------------------------- compuwar · 2023-07-08 23:00 You doubt reading through an API into a database through a driver into a file system would be slower that reading from a file system directly, let alone a cached one? ---------------------------------------------------------------------- balr · 2023-07-08 23:02 The GNOME project did something similar with [gconf](https://en.wikipedia.org/wiki/GConf). ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 23:03 I called it a driver, but it’s not a kernelspace driver it’s really more of a userspace driver -as the db itself is a userspace component. As such, reading/writing shouldn’t be much different than using libc functions for reading/writing files. Undoubtedly, any timing difference will be unnoticeable to end users. ---------------------------------------------------------------------- compuwar · 2023-07-08 23:09 So now you have to overlay and manage a security schema as well? That’s a lot of work and patching. Userspace DBs for a mix of privileged and unprivileged processes without kernel involvement sounds like a security nightmare to me. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-08 23:12 As pointed out above, there’re commercial Unix systems that’ve already implemented the system that I was asking about. If it’s not a security nightmare for them, then there’s no reason to expect it to be a security nightmare in a system initially based on BSD. ---------------------------------------------------------------------- compuwar · 2023-07-09 00:06 I don’t believe you really understand AIX, which is a System V derivative. I’ve professionally administered AIX systems, so I’ll just make a few points here: 1. Even with the ODM, AIX *still* has a /etc directory with files in it. 2. Even with the investment in the ODM and SMIT, there are still text file configurations after decades of time. 3. AIX has a *lot* of built-in security and privilege functionality, it’s not just throwing everything into a user-space driver, crossing their fingers and hoping for the best. 4. A single entity maintains all the software that interacts with the ODM. 5. Rebuilding an AIX system with a corrupt ODM is an order of magnitude more difficult that repairing a corrupt root file system. Almost as bad are recurring objects that are only corrupt enough to re-emerge at boot time. From AIX 3.x to 4.x, it was our standard production practice to immediately un-ODM all the configuration things we could like addresses, routes, resolvers, etc.) because the grass was most certainly not greener. It’s BSD, you can build it if you wish, but I doubt you’ll see any significant adoption. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-09 02:53 I’m not looking for significant adoption. My post had absolutely nothing to do with adoption. I asked if anyone had seen an example of such a thing & said that I’d be interested in hearing about it. As for AIX having /etc, that’s really irrelevant to what I asked & why I was asking it. It’s nothing more than a historical holdover present in an old system. As such, it’s not really germane to this post. ---------------------------------------------------------------------- catonic · 2023-07-09 04:27 then you wind up recreating the problem as now you need atomic access and to define users, groups, policies, ACLs, etc. and properly constructed databases to hold said information but also now the kernel needs it as well. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-09 11:27 Now that I’ve seen other Unix systems with similar functionality, I doubt that it would be as bad as you’re describing. ---------------------------------------------------------------------- self · 2023-07-09 12:44 it's not purely bsd, but nextstep/openstep had [netinfo](https://en.wikipedia.org/wiki/NetInfo) for this. ---------------------------------------------------------------------- nmariusp · 2023-07-09 21:59 No way. The Windows registry is used for less than one percent of the settings of Windows OS, Microsoft app and non Microsoft apps. There are maany other things used for storing configuration, settings, defaults, schemas etc. E.g. the directory "C:\\ProgramData", SQLite databases, JSON, XML or INI files, binary proprietary formats etc. ---------------------------------------------------------------------- TheFlean · 2023-07-09 23:49 Yeah cus people realized that it’s stupid. Nothing comes over plain files. ---------------------------------------------------------------------- fragbot2 · 2023-07-10 05:04 I was hoping someone would mention AIX as it's the only Unix I know of that does this. I still remember the first time I added something to _/etc/services_ and was baffled when it disappeared after reboot. ---------------------------------------------------------------------- volci · 2023-07-11 13:26 The *only* thing I dislike about the Registry is that keys are so rarely named *well* that if you don't "know where to look", you'll never find a setting to update Sure, the *locations* of keys are pretty consistent - but the names? They weren't even named as badly as some people camelCase, snake_case, or Hungarian notate their variables and functions! Want to change the color of the title bar? Easy! It's in `HKEY_USER\LOOK_HERE\NOT_WHERE\Y OU_WOULD\THINK_TO_LOOK\UNTIL_YOU_GET_HERE_AGAIN\BUT_NOT_NOW\h01ffwt\(D efault)` It's kinda like the horrendous class hierarchy in Java (or, for that matter, Python) - `org.company_name_from_4_acquisitions_ago.some.path. that_kinda.sorta.made_sense_then.but_not_now` But once you name/create that class path, you can't ever get rid of it ... because *everything* that uses it expects it to be there :| Now. If you had a Registry-like kvp backend for most of your settings, and you could *search* it - that could be really nice Something like MongoDB (doesn't have to be Mongo - just something like it) with a simple search language atop it ([a subset of] SQL, AQL, GraphQL, SPL, basic grep/awk/sed, something nominally RESTful...) that could both *find* and *update* stuff (and a way to revert to defaults (because often enough we have ***ALL*** made a change that turned out to break something because of some unexpected interaction (or fat fingering... set the width of the scrollbar to 1000 pixels instead of 10 ... oops! (though, that should be automatically correctable programmatically, too - you can set any value you want, but if you're outside the acceptable range, set it to minval or maxval, depending on which way you overshot the range)))) ... you've got me *highly* intrigued This brings up a few absolute core requirements for this theoretical Registry-but-better config store, which for now I will call "Ark": - the kernel has to know how to access the Ark *very early* in the boot process (probably sometime during init 1 before switching to the correct init level (which, of course, it would look up in the Ark)) - the kernel needs to expose (or enforce) a perms-appropriate view to the Ark - the Ark *must* be 100% reliable (some kind of journaled backup, perhaps? almost like it was its own highly-stable filesystem) - an Ark should be [almost] 100% transferrable/mergable to another Ark-based system (make for fast cloning of servers and/or settings across an environment) - any "unused" settings would just be ignored on the new system - merging more than one Ark into a new one must - be possible - be simple - have clear priority when settings collide ---------------------------------------------------------------------- volci · 2023-07-11 16:47 Someone on HN (https://news.ycombinator.com/item?id=36682595) suggested etcd (https://etcd.io) ---------------------------------------------------------------------- demetrioussharpe · 2023-07-11 17:07 That's a very interesting approach. Thanks! ---------------------------------------------------------------------- vermaden · 2023-07-11 20:31 AIX ODM is only comparable with Windows Registry. This is NOT the way and definitely not the UNIX way. ---------------------------------------------------------------------- demetrioussharpe · 2023-07-12 11:15 This is a great idea! ---------------------------------------------------------------------- volci · 2023-07-12 12:04 Getting this *right* (ie actually learning from previous failure and at least trying to only make new mistakes) is gonna be the hard part On the surface, it sounds like something a motivated 3rd- or 4th-year CS/SE major could do over a semester. Or maybe a grad school capstone project. But it's probably really more like a 4-5 person team working for a couple years to get even a basic releasable version done ---------------------------------------------------------------------- reallokiscarlet · 2023-07-13 17:02 It's definitely possible to do, especially if you're willing to mount the database as /etc and store conf files as really big keys in the interim while you update your software to use the database natively. This type of structure does come with drawbacks, however. Firstly, it will rely on a database driver. You'll be reading a database from a something like a file or a server, and if that driver fails, your settings are gone. Second, it will depend on what type of database you want to use. Some database software is just a driver to translate requests to file contents in a database file. Some database software may have severe demands for resources like CPU or RAM. And one last part I'll mention, though that's not where the issues end, is many programs and services actually benefit from having their own structure and syntax for configuration. If you're willing to face these issues head-on, get ready to apply directly to the forehead. You'll need to fork a lot of software, and release your own distribution. I would highly recommend, even after your implementation matures, that you still use /etc for compatibility purposes, even if it means the files in /etc will be keys in a part of your database. Unix benefits from being able to address everything as a file. Plenty of Unix/Linux systems have databases that are addressed as a filesystem. Devfs is an example of a pseudo-filesystem that works like this, letting you access hardware as if all your computer's parts were files on the hard drive. Linux has sysfs as another example of this. Some files are read-only keys to tell you what's happening like your address bits and your kernel version, some are read-write and you can change some settings in the running kernel by outputting something to them. Operating systems that hide their settings in a database usually do this specifically to separate themselves from Unix. The Windows Registry is an example of this. In the DOS days, you had ini files all over the place, sometimes in centralized directories, sometimes in the directory of what you're configuring. Microsoft wanted to reduce that, which is where the Windows Registry comes into play. You might be able to get away with having a system registry table and a compatibility table, mount the compatibility table as /etc and the system registry table as /etc.d, since the common convention for itemizing configuration is to have a conf and a conf.d for itemized conf files.