Troubleshooting random Redis service shutdowns and Postgres not
accepting incoming connections?
reddit / freebsd

----------------------------------------------------------------------
ajm3232 · 2020-10-05 05:44

To give some context really quick. I have two production servers both
running FreeBSD using AWS on 12.0-RELEASE-p13. One running ONLY
Postgres and the second server running the application and Redis.

The main issue I'm concerned with the Postgres server is in the very
early morning my application failed to connect to the Postgres server
due to tcp i/o timeout and came back online after two hours. Of course
checked the Postgres logs with no error found, and also the
\`/var/log/messages\` with no content.

I have logging enabled, of course set to level warning and see from
htop the server has been running the entire time. It almost seems like
nothing went wrong on the servers end the while time the application
couldn't connect for two hours. I have my connection string supplied a
direct IP address. So no host lookup would be needed. Is there a way
to help diagnose what it could have been to determine it doesn't
happen again? Both servers are using their own private network on AWS.

My second issue is my Redis caching layer keeps shutting down for no
reason almost once a month. I have verbose logging enabled and have a
very conservative cache limit. Of course, \`/var/log/redis/redis\`
doesn't show any errors or any odd activity. Is there a way I can
troubleshoot the init system on why it may have shut it down or at
least what error code it may have returned when exited?

Thanks!

TL;DR Ver: Postgres wasn't accepting connections for two hours.
Everything on the server appeared fine. No idea what it could have
been. Redis keeps shutting down w/o error. No idea if FreeBSD is
shutting it down or the init system and only happens once a month
after conservative caching rules. Both are on their own separate
FreeBSD servers in the same AWS network.