[HN Gopher] Observe a Service; Not a Server
___________________________________________________________________
Observe a Service; Not a Server
Author : kiyanwang
Score : 14 points
Date : 2021-08-02 05:44 UTC (17 hours ago)
(HTM) web link (blog.last9.io)
(TXT) w3m dump (blog.last9.io)
| pmlnr wrote:
| Right. Do this and you'll never be able to find that single
| machine that believes it's on a 100mbit line instead of 10G and
| keep wondering why every X user has a problem.
|
| Monitor both, please.
| imwillofficial wrote:
| A billion times this. The cattle still need love.
| cotillion wrote:
| Yes monitor. You really want to find that server which is busy
| encrypting all that nice data your service needs.
| derefr wrote:
| The article is talking about "monitoring" in the sense of
| alerting + stakeholder incident reporting. Things that reach
| out and tell ops staff / users that the service is having a
| problem.
|
| You do want to be capturing _metrics_ about your servers --
| _observability into_ their configuration and behavior -- but
| capturing that type of data is not precisely the same thing as
| "monitoring" the server. "Monitoring" usually implies having a
| defined acceptance threshold on the metric, and triggering some
| (probably human) response when the metric is in an unacceptable
| range.
|
| If you don't have either a clear threshold for the metric, or a
| clear "thing that should happen" when the metric goes out-of-
| tolerance, then you aren't "monitoring" the metric per se; or
| at least, you _shouldn 't_ be.
|
| As the article says:
|
| > Here's an example: A disk consuming 100% at 3 AM may or may
| not be actionable depending on its impact on service KPI. For
| instance, if it's a logout service and the KPI says 99.999%
| availability despite a disk failure, It can wait till the
| morning.
|
| If you have set up alerting for server disks being full, where
| that alert often _doesn 't_ translate into any problem with
| your running services, then that alert is a useless/noise
| alert, that will only serve to make you treat notifications
| from your alerting infrastructure with less urgency. It's the
| "check-engine light" of SaaS.
|
| It's fine to have a dashboard that holds all that stuff. But it
| shouldn't be reaching out and telling you/users about it. You
| can go look at that dashboard when _investigating_ why the
| service KPIs you 're alerting on are down.
|
| In your example, the appropriate service KPI to alert on would
| be "[service] 99th percentile end-to-end response latency."
| Upon receiving an alert that that number is out-of-tolerance,
| you'd then start to investigate your individual -- or,
| hopefully, aggregated! -- server _metrics_ , to see what-if-
| anything is acting weird.
| nix23 wrote:
| >"Monitoring" usually implies having a defined acceptance
| threshold on the metric, and triggering some (probably human)
| response when the metric is in an unacceptable range.
|
| Like the Smart data on your Hard-drives? The increasingly
| (over month) thrown away packets of a dying Network card? Or
| that one Fan that increasingly spins slower then all others?
|
| Look Hard is not different from Software, both need to be
| Monitored having a close look on "trends" and want to be
| Maintained.
| nix23 wrote:
| Full hit! If one think in services, the Hardware is the first
| service one should observe, just thinking about the running
| Software ON IT as a service is ridiculous. Good luck
| "debugging" your performance problems if your Hardware is
| struggling to keep the Raid up, trowing packets away a hardware
| level...or hey, running at 600Mzh because the CPU is boiling,
| two fan's reported errors, hell even fans are services (by
| providing airflow)
___________________________________________________________________
(page generated 2021-08-02 23:03 UTC)