Server crashing (on periodic jobs?), how to diagnose? reddit / freebsd ---------------------------------------------------------------------- dddddavidddd · 2020-07-29 20:53 My FreeBSD server frequently stops responding overnight (no SSH, HTTP, other services) and needs to be rebooted in the morning. I'm looking for ideas to diagnose what's happening. I have a `cron` job to record system temperatures every three minutes, and use this to determine when the system crashes. The timing has me suspecting that overnight `periodic` jobs may be involved (crashes this month: 3:51, 3:57, 5:24, 4:45, 3:48, 3:51, 4:09). There are no messages in `/var/log/messages` around these times. Suspecting resource exhaustion, I set up `cron` jobs to log swap use (never above 10%), max CPU process (often `find` or `sendmail`, 30-60% usage) and max RES ram usage (normal services, e.g. `syncthing`, about 100 MB). All disks are ZFS RAID 1, and I've limited the ARC cache to 1.5 GB (8 GB RAM total). I've been considering getting a KVM to take a look at the server after a crash, since currently it runs headless. What else could I do to figure out what's going on? ---------------------------------------------------------------------- rysto32 · 2020-07-29 22:25 You might try enabling crash dumps and watchdogd. Tools such as ps and netstat can be run against a crash dump, which would hopefully give a clue as to what was running at the time of the crash. ---------------------------------------------------------------------- bsdbro · 2020-07-29 22:32 What version are you running? Is the debug.debugger\_on\_panic sysctl set to 1? ---------------------------------------------------------------------- dddddavidddd · 2020-07-30 01:02 Thank you for the pointers! I've set up watchdogd and enabled crash dumps and will hopefully be able to get more information. ---------------------------------------------------------------------- antiduh · 2020-07-30 03:13 Have you run memtest86 and prime95 on the hardware? It could be marginal hardware. ---------------------------------------------------------------------- oh5nxo · 2020-07-30 06:46 > no messages in /var/log/messages around these times Try \*.\* @otherhost in /etc/syslog.conf to catch more messages. ---------------------------------------------------------------------- avgapon · 2020-07-30 11:04 Add monitoring of wired memory as well. ---------------------------------------------------------------------- darkempath · 2020-07-30 11:17 I had a similar problem which I solved by turning off power management in UEFI. I was running a headless FreeBSD 11.x system flawlessly, but after a binary upgrade to 12, I was getting random "freezes". I couldn't ssh in, and I couldn't access Roundcube or Nextcloud (Apache). I couldn't tell what was triggering it, I'm not a particularly skilled sysadmin :-p Once I disabled power management, it seemed to work fine again. I assumed it was a config glitch after the upgrade, maybe you have a similar problem? ---------------------------------------------------------------------- frenchiephish · 2020-07-30 11:19 For troubleshooting, if you're pretty confident it's related.to periodic you can manually kick off periodic by running `periodic daily`. The other suggestions here are all good for checking for a host crash. I've had similar symptoms from a flakey power supply and from a motherboard with bad power delivery. In those instances you get a precious shortage of info and they're not fun to diagnose. ---------------------------------------------------------------------- daemonpenguin · 2020-07-30 13:41 I used to have this happen a bunch of times. I found a few things helpful: 1. Check the access times of your daily periodic scripts (/etc/periodic/daily). Assuming you have access time enabled on your filesystem, this will tell you which was the last script run, and therefore the problem. Chances are the same job is crashing your machine every night. Since jobs are run in numeric order, just look for the last one in the list that was accessed using "ls -l". 2. In my experience periodic jobs bringing down the server are almost always related to memory issues (not enough RAM, broken RAM). Especially on FreeBSD boxes running ZFS. There are some periodic jobs that use a lot of RAM when doing ZFS checks, such as the scrub check. Moving or removing the script(s) that do ZFS checks will typically correct the issue on my systems. Or replace a broken RAM module. 3. OP already did this, but limiting the ARC size for ZFS can help keep the system alive. ---------------------------------------------------------------------- dddddavidddd · 2020-07-30 16:42 Good point, I'll take a look at this. ---------------------------------------------------------------------- dddddavidddd · 2020-07-30 16:43 Thanks for the tip, sometimes my laptop (20 GB RAM) has issues with this, so it seems more likely with the server's limited RAM. ---------------------------------------------------------------------- dddddavidddd · 2020-07-30 16:45 Thanks for the tip about access times, I'll check this out next time it crashes. ---------------------------------------------------------------------- dddddavidddd · 2020-09-20 22:27 I tried the watchdog but unfortunately the machine wouldn't reboot when it became unresponsive. I ended up adding another 8GB of RAM and haven't had a crash since.