2026-10-30 Sysadmin day ======================= > My wife is looking at graphs of our electricity use and our warm > water use, and I'm looking at the Munin graphs of our server. Is there something I could learn from looking at my public Munin graphs, even without an incident, perhaps over longer time periods? Let's see. I'm counting Apache connections because at one point I was trying to understand whether my server was being overwhelmed by many connections and if I needed to fiddle with the many configuration options of the web server. Apache connections over one month. Two weeks ago the number seems to have stabilized at 10 or less. I'm counting the hits on the garbage server because the idea is that bots get lured into the maze of pages generated by the garbage server so it's a proxy for detected bots that fall for it. A high number of hits should not be a problem as it answers requests without disk access so it should be fast. The hits on the garbage server appear to be stable at about 1 hit/second for the last three weeks. I'm also counting requests per status code because I'm interested in my server's health and the quality of my links. What I generally look for is whether I get a lot of server errors (500+), a lot of missing pages (404), a lot of redirects (301–303) a lot of local cache hits (304), etc. Hits per seconds, per HTTP status code. The high number of 301 responses ("moved permanently") is a sign, I think, of bots that keep accessing old URLs that are being redirected and never updating their data. In other words, these days they are suspected AI scrapers. I'm also counting hits per domain to see if there are any spikes for a particular domain only. Domains I fear for are the GoToSocial subdomain, the git subdomain, and so on. Everything seems calm, at the moment. I also host three search engines and I'm just realizing that one of them has seen spikes nearly twice as high as the maximum hits on Emacs Wiki. That is a bit disconcerting. What happened? I'm assuming the spikes for the GoToSocial server happen when it comes back online after some downtime as the servers it is connected to start working on the queues. And the spikes on the git server happen when somebody checks out a project. All the various domains served, with hits per second. I run an open video-conferencing server called Galène. I want to keep an eye on it in case somebody is starting to use it a lot. At the moment it looks as if it's just me and my friends. Last wee had a handful of spikes with up to four people connected. This is fine. I run an IRC server networked with the IRC servers of two friends. I want to keep an eye on the number of users and channels. I wouldn't want the server to end up being a control server for malware. The number of channels and users remained stable. Temporary drops reflect on the quality of our network, if anything. Now we're moving on to resources. Disks first. I think the main fear is running out of disk space because logs are out of control. This is where the systemd journal shines. If log files are written to disk, logrotate needs to know about them. So installing a piece of software that is missed by logrotate is a problem. As long as you stay within the software of your distribution, and as long as any additional software you install uses systemd, you should be fine. The graph is pretty stable, showing logs growing over each day and getting truncated again. And a huge jump two weeks ago, probably the backup of the GoToSocial database file (10G). I guess these days people don't think about inode usage all that much any more. Back when servers carried all of USENET and every news article was a file, it was possible for a system to run out of inodes. It's basically a limit on the number of files that can exist on a system. Still, I track them. Just to be sure. The number of inodes is at less than 8%. Nothing to worry about. If the number of inodes keeps hovering around 8%, then the exact number of inodes doesn't matter that much. This seems like a graph that I could remove. The number of inodes used remains stable, too. There were two strange, temporary drops two weeks ago and I have no idea why this could have happened. Another strange graph is the one with the number of open files. Linux has a limit on the number of open files at any given point in time so I guess it's important to see whether we're close? lsof | wc -l shows nearly 20,000 right now. Then again, sysctl fs.file-max says 9223372036854775807 … so I guess this graph is also one I could remove. The number of open files is stable at about 3000. I don't understand how that relates to the 20,000 I mentioned above. This machine runs a mail server. Of course it does. But it's local only. The daily mail you see is probably the net news server status. I guess I'm most afraid of some situation sending a gazillion mails and me not noticing it. A small spike once per day, that's it, for almost all days. The mail queue is important if mails cannot be sent. But if nobody is sending mail on this system, the queue should be empty, and it is. Unsurprisingly, the mail queue itself is empty, too. The number of users is a measure in the same vein. It's useless on a single user machine. On a shared machine, this graph would make sense. I guess I'm waiting for a second user to show up that I don't recognize. I'll know that the machine has been hacked, then. 😬 Most of the time nobody is logged in. Sometimes one person is logged in (me). Sometimes more, all of them being me. Measuring the resources spent by Munin to produce the graphs is a kind of insurance against writing a plugin that hurts. I guess we're doing fine? Munin processing time stays at about 18 seconds per month, with a tiny day cycle. I'm assuming when the current log file is short in the morning, some plugins finish faster than at the end of the day when the logs are long. The firewall throughput measures the amount of data that is actually sent, via all ports. This could be interesting to detect exfiltration or bots repeatedly downloading huge files. That would have to stop, of course. Firewall throughput shows a small number of spikes last week. I did not investigate. Whatever it was, it's now over. I'm not quite sure what to do with net stats. I guess some sort of denial of service might show up here? I have no idea. The net stats seem to be stable even if I don't know what it means to me. The failed and reset TCP connections are very low. So I guess we're fine. We're getting to one of the important graphs regarding the bots, I think. Ever since the server got an upgrade, and perhaps ever since I installed the botcheck defence, average system load has been under control. What hasn't been under control is the number of autonomous systems that I'm banning. For more about this, see my Butlerian Jihad page. Note: Each autonomous system manages multiple IP address ranges and each IP address range contains many IP addresses. Each autonomous system belongs to an organisation involved in managing Internet traffic. Sometimes an organisation is an Internet service provider (ISP), either for residential or commercial subscribers. Sometimes an organisation belongs to an even bigger company (who else but Amazon, Meta, Google, Microsoft and all the other tech giants). In any case, banning an autonomous system can hit many innocent people: an autonomous system that serves both users of virtual private networks (VPNs) and bots; big companies hosting both individuals and bots; and so on. The solution is forced upon me, however, since the bots are trying to evade blocks by using a gazillion IP addresses. Blocking them individually results in huge firewall sets and it's inefficient because they don't get reused all that often. Four weeks ago, over 8000 IP address ranges were blocked, now it's less than 3000 IP address ranges. Back when the scraper bots overwhelmed the server on a regular basis, the problem the load average going up to 80 on a system with two virtual cores. Now I have four virtual cores and load stays far below that. The load average is under control, safely below 1. I'm not sure what the fork rate means for me. When heavy processes fork, other resources spike immediately. So I guess we're doing OK. Fork rate is pretty constant at 4/second. If the fork rate is pretty much constant, the number of processes should also be pretty much constant. And that's what I find, so I'm happy. Apparently four weeks ago there were a number of zombie processes that were hanging around. I'm happy they are gone, now. Everything else seems to be constant, so good. I'm never quite sure about threads. Are they even used these days? "Workers" are not necessarily threads. If I'm counting processes, I should also keep an eye on threads. Hence this graph. Since there are no weird spikes, I guess we're doing OK. A stable number of threads that goes up and down over the course of each day. I have no idea why three and four weeks ago the number seemed to be higher. There used to be a time when my web apps based on the Mojolicious framework would spawn more workers as load went up. I guess now that load is under control, that rarely happens any more. A list of processes, but almost all of them show just a single process being counted. Surprisingly, the max number of processes for Emacs Wiki was more than 12. Another measure of resource use is compute. In other words, CPU usage. CPU usage is extremely low these days. The numbers can go up to 400% because I have four (virtual) cores available. I also try to identify the processes that use the most resources. I'm looking for spikes. Most of the time it's GoToSocial that's using the most CPU. That seems fine. Apache is acting as the reverse proxy for all web services. Fail2ban is managing the firewall. Everything seems fine. A huge list of processes with GoToSocial showing daily spikes. And yet another resource is, of course, memory. Specially with RAM prices being what they are! In my case, I think my machine is doing pretty well. The memory usage shows about half a gigabyte of eight unused and nearly four gigabytes used for caching. This system is far too powerful for my use. The graph for memory use is used to find applications that seem to have memory leaks. Sadly, once you've identified the application, it's not entirely clear how to fix those memory leaks. Right now, I don't see any suspicious candidate, though. I do get the impression that some applications don't want to relinquish the memory that they have, like Traveller. Do I still have circular data structures resulting in memory leaks? A log list of processes, with GoToSocial clearly dominating memory use, and surprisingly, The Lounge on second place. Above, I used percentage of memory for processes to see. And here's the RSS. Measuring memory consumption under Linux is hard. Another long list of processes. A least the memory use here does not contradict the MEM graph above. And yet! And yet. Everything looks great but at the same time the logs still show the out of memory (OOM) killer striking every now and then. These instances should all get investigated. I am slacking. Multiple instances of the out of memory killer striking. Back in the old days, computers never had enough memories and so unused memory was swapped to disk. As this system is doing so well, that isn't necessary. So I guess what I'd expect to see is that if memory starts running out, swap in and swap out should pick up. If the numbers are close to zero, that's good. Swap shows some activity but I don't know what it means. We're going to end the tour with the most useless of graphs: uptime. When I was young, long uptime was a sign of stability. Your system kept running for months and years. So much better than Windows! These days, I think a long uptime means that you're not upgrading the kernel when security issues come calling and that's bad. The graph shows that I rebooted the machine twice in the last four weeks. I'm guessing this means I missed a few security updates! And that's it! System administration when nothing is happening. Let's hope it stays that way. #Administration 2026-10-31. Sean Connor wrote in to tell me about SYN flooding from Brazil and how to block it (I still don't understand this SYN attack, but now I can block it easily) and how it was a Brazilian tech firm ( For now, one last time with the Brazilian SYN attacks). He said one should never expect too many connections in SYN state. When I just checked, I had 15. Now I have 6. So the list is small. netstat -an|grep SYN I don't think I want to install mod_limitipconn which limit the maximum number of simultaneous connections per IP address. And he also mentioned a strange HTTP header rewrite he does (Oh, it's a bug on my side that prevents full conditional requests) to allow feed readers to do conditional requests (the ones where an unchanged resource doesn't get downloaded but a 304 is returned).