[HN Gopher] Accuracy of Commercial Sleep-Trackers Compared to Re...
___________________________________________________________________
Accuracy of Commercial Sleep-Trackers Compared to Research-Grade
Tools
Author : PaulHoule
Score : 62 points
Date : 2024-01-27 14:12 UTC (8 hours ago)
(HTM) web link (www.mdpi.com)
(TXT) w3m dump (www.mdpi.com)
| simonebrunozzi wrote:
| > The Fitbit Inspire, Fitbit Versa, and Oura Ring estimated light
| sleep with less bias than deep sleep. Conversely, the Withings
| Mat and Garmin Vivosmart estimated deep sleep with less bias than
| light sleep. Across devices, estimates for REM sleep were less
| biased than estimates of light and deep sleep. Despite low
| absolute biases, it should be noted that some estimates were
| still up to 250 min different than their PSG derived measures.
|
| In other words: these devices are all terrible, and shouldn't be
| trusted. Pretty different from the final conclusion, which is
| probably "soft" to get additional grants from the commercial
| entities selling these devices.
| marcinzm wrote:
| I mean, given the study they shouldn't be trusted as a single
| night measure of absolute sleep quality however I don't know of
| anyone who uses them as such. You use them over long periods of
| time to either get aggregate statistics to compare against
| baseline or track personal changes over time.
|
| edit: Also, the last time I did a sleep study at home it had
| rather little correlation to how I normally sleep except in
| terms of apnea. Having wires, electrodes, a chest strap and a
| finger O2 monitor strapped to me did not lead to a pleasant
| night of sleep.
| pessimizer wrote:
| > You use them over long periods of time to either get
| aggregate statistics to compare against baseline or track
| personal changes over time.
|
| Doesn't that assume that the errors will be consistent and
| precise enough that any trends don't get totally wiped out?
| GIFtheory wrote:
| I think the conclusion is that /total sleep time/ is actually
| very accurate, while deep/light/rem measurements are crap.
| 2devnull wrote:
| All they have to do is measure time in bed, which for most
| people most of the time will correlate closely with total
| sleep. Insomniacs are outliers so you don't need to worry
| about measuring them and most are taught to just get up when
| they awaken anyway.
| ilaksh wrote:
| You skipped the section where the measurement of total sleep
| for most devices was found to be more than adequate.
|
| Detecting light versus deep sleep accurately with wristwatches
| or devices under the mattresses might not be possible.
| Especially not when accuracy is supposed to be comparable to
| multiple electrodes places directly on the head to measure
| brainwaves.
| yodon wrote:
| These plots definitely align with my personal anecdotal
| experience.
|
| I own both a Fitbit Vivo Active 4S and a Garmin VivoSmart 4.
|
| In the daytime, as a watch, I much prefer the Garmin, but at
| night, as a sleep tracker, which is the main thing I want from
| these watches, I only trust the data from the Fitbit.
|
| If I'm concerned about actual total sleep times, I do generally
| need to edit the Fitbit data in the morning. Over the years, my
| Fitbit watches have fairly consistently confused idle time
| sitting on a couch watching videos or lying in bed reading on my
| phone before or after sleeping as sleep. The data edit tools in
| the app fortunately make it easy to correct this (I'm not someone
| who falls asleep with the video playing, so I generally know when
| I turned off the activity at night and when I woke for the
| morning).
|
| The data during the night, which is what I care about, I've come
| to trust on Fitbit watches over the years. I can't know if it's
| precisely accurate, but it's definitely directionally correct (if
| there's an overall scale factor error or similar from true sleep
| quality and time numbers I don't really care as the measurements
| are consistent enough that I can use them and benefit from them).
|
| My Garmin I view as a random number generator as far as sleep
| tracking. My perception is it's probably a bit better about not
| misclassifying idle time before and after sleep, but not accurate
| enough to be actually correct, and I have not been able to find
| any simple data editing feature in the app to fix the
| misclassifications, so the wrong data stays wrong.
|
| The Garmin data during the night I have zero confidence in, as
| anecdotally it feels far less "believable" than what I see
| reported by my Fitbit.
|
| Edit added: another commenter just mentioned that Garmin uses a
| learning algorithm that needs about two weeks of data to hit good
| data quality. I've done a couple periods of sleep tracking with
| my Garmin, but I'm guessing I've abandoned it due to poor data
| quality each time at around two weeks. Sounds like I should give
| it another try and force myself to wait a month or so before
| making any assessments.
| onetimeuse92304 wrote:
| I don't have experience with Fitbit, but I have been using
| Garmin Fenix 7x for the past two years and I can confirm that
| at least in my case, its measurement of my sleep quality is
| only loosely connected to reality.
|
| As I am an avid runner and I train daily, I pay close attention
| to all sleep, heart rate, stress, and running performance
| numbers.
|
| Unfortunately, it will frequently say I had good night of sleep
| when I definitely feel groggy or the opposite.
|
| As to misclassifying idle time before and after sleep, I
| recognise that simple lifestyle adjustment could fix it (just
| don't browse on your phone before sleep!) But that of course
| won't fix it if you try to lie still but can't fall asleep.
| lloeki wrote:
| My anecdata is different: sleep is mostly accurate, if
| sometimes a bit fuzzy, and occasionally misses the mark.
|
| Regarding grogginess or lack thereof, what I noticed is that
| sometimes I feel that way even though the watch says I had
| good rest, but it's only so at the beginning of the day;
| afterwards I'm okay.
|
| Conversely I may have had a bad night and still feel okay,
| but what I noticed is that if I don't pay attention to the
| watch I'm actually plowing through and don't notice I'm
| wearing myself out, which I then have to repay the following
| days.
|
| Also the watch gives a) objective measurements (whereas my
| feeling is subjective) and b) measures only a bunch of things
| but not everything (e.g I noticed eating pasta or something
| weighting on my mind and morale can both make me experience
| brain fog that the watch does not account for)
| petre wrote:
| Drinking one or two cups of green tea helps with my
| grogginess. Or some light cardio exercise like 3 minutes of
| rope jumping if you're more inclined to do that rather than
| stimulents. I also drink a shot of espresso sometimes, but
| the tea does a much better job without risking a headache.
| It's Chinese green tea with low caffeine content, Chun Mee or
| other blends from Yunnan or Anhui. I like their method of
| processing more than what is used in Taiwan or Japan.
| smallerfish wrote:
| I've been sleep tracking with my Garmin Fenix 7 for about 18
| months, and to me it seems bang on. The HRV stat is
| particularly good - a decline over about a week predicted the
| onset of flu symptoms a couple months ago, for example. Body
| battery is also a really useful feedback measurement. Sleep
| onset and wake times are right; REM seems right; and
| deep/shallow measurement at least correlates with how I feel.
| lloeki wrote:
| From the paper:
|
| > There are limitations to consider in the interpretation of
| these results. First, devices were worn for one night.
|
| That is odd.
|
| What I noticed is that my Garmin watch may be wrong at times, but
| overall provides a fairly good and very useful assessment,
| _especially comparatively day vs day and overall for trends_.
|
| In addition it takes a bunch of time for the watch to work out
| enough data to tune the algorithms for a particular person: some
| UI bits explicitly call out that for accurate results the watch
| should be worn continuously for ~2 weeks.
|
| I've now been wearing this particular one (Fenix 7) for 6 months,
| a serious upgrade from a Forerunner 735xt continuously worn for 5
| years. 95% of the time the watch is stupidly accurate in ways it
| has no business being as a simple wrist-worn set of sensors.
| 99094 wrote:
| My Garmin (255s) is pretty inaccurate when it comes to sleep.
| It often mistakes REM for awake and vice versa. Honestly,
| nowadays I only wear it for my runs. Even if it was accurate, I
| can just tell by feel. All the metrics such as sleep, resting
| HR and HRV are pretty bad compared to your nervous system.
|
| The metrics aren't really actionable either. I got a bad night
| of sleep, so what? I know it. I have low HRV because I drank
| alcohol/over-trained/am ill. I don't need a watch to tell me
| that. I know it.
|
| It's just collecting data for the sake of collecting data.
| csdvrx wrote:
| > It's just collecting data for the sake of collecting data.
|
| No, sleep is one of the most important parameters for health!
|
| If in 2024 you are not collecting data on your sleep to take
| action (and for ex, stop overtraining, reduce drinking etc)
| to reduce the risks of future brain problems, you are doing
| it wrong.
|
| > Even if it was accurate, I can just tell by feel
|
| I can't, and I fear it may be the same "overconfidence" issue
| that cause car accidents when drunk.
|
| I collect logs with my sqlite based bash history collection
| that I've been running for 5 years:
| https://github.com/csdvrx/bash-timestamping-sqlite
|
| Thanks to my garmin, I have identified a pattern where I feel
| ok, but my garmin reported less REM sleep than usual: in the
| day, I run more commands than usual but I have way more
| mistakes (as can be seen by the non-zero return values)
|
| So now, when my watch reports low REM sleep, I try to be very
| careful to avoid negative-productivity days
| ocharles wrote:
| Sleep isn't the metric to track here, there are much better
| things to analyze such as HRV. For reducing drinking you
| shouldn't be doing that because of your sleep, you should
| be doing it because of your overall health. If you care
| about future health, take blood tests, weigh yourself,
| evaluate your physical fitness.
| csdvrx wrote:
| Personally, I don't drink, and I've noticed sleep to be
| the best indicator of my coding performance but YMMV
| petre wrote:
| I've changed my 250 to an Insight 2. It's an improvement and
| it helped me be more attentive with my sleep, exercise. I
| don't do runs but walks, hiking, cycling and several water
| sports during summer. It's also used as a HR sensor for my
| cycling Garmin device.
| Aachen wrote:
| If the software compares behavior between nights and can then
| say "that was relatively good", then you're right and the
| limitation matters
|
| If the software behaves no differently from one night to the
| next, that is, if you can take any random night's assessment
| and it would be as likely to be accurate as the first, then it
| does not matter that they had participants only wear it one
| night. You'd get more data (from multiple nights) without
| needing to recruit more participants, but you'd also be biasing
| towards whether the tracker works well for this set of people
| (probably a worthwhile trade-off, up to a point)
|
| Your watch says to use it for a while. It could be that that's
| just because it's not flawless and they want people to see it's
| often correct even if they got unlucky the first night(s). Or
| maybe your watch works differently from the ones used in this
| study
|
| I'm also surprised they only did one night, both because you
| get a lot more data per amount of effort by having participants
| keep it for a month, and because I'd not be surprised if the
| software is more comparative than absolute. But I don't know if
| we can draw real conclusions from that
| HumblyTossed wrote:
| Unless they're spewing out random garbage (which I doubt they
| are) they're good for tracking trends. My overnight RHR is
| about 55. If I'm starting to get sick, even before symptoms set
| in, it raises to about 60. By looking at trends I can tell when
| I'm about to start feeling bad.
| LikelyClueless wrote:
| The YouTube channel, 'The Quantified Scientist', does a great job
| of comparing sleep tracking wearables to a reference EEG. He
| collects a lot of data and presents it very well.
| CharlesW wrote:
| Thank you for posting this! I recently started using my Apple
| Watch for sleep tracking and have been dismayed at my quality
| and quantity (vs. "bed time") of sleep. Searching the channel
| for "sleep" I instantly found 3 videos for my "to watch" list.
| smnplk wrote:
| A used CPAP machine and overnight oximeter device from aliexpress
| are much better tools for tracking sleep quality than any of
| these watches.
| salad-tycoon wrote:
| I used to use this [1] software about 7 years ago when I first
| got my CPAP due to a lack of attention from my sleep medicine
| team. I could dive deep into my downloaded data and set my
| precise measurements perfectly.
|
| Haven't used it in years.
|
| Do you have any recommendations for how to find a dependable
| overnight pulse oximiter on alibaba? I've never shopped there.
|
| 1 https://sleepyhead.software.informer.com/
|
| Edit: apparently sleepyhead was shutdown and replaced by Oscar.
| https://www.apneaboard.com/sleepyhead/
| hn_throwaway_99 wrote:
| Semi-related question. I recently took an at-home sleep apnea
| test using the WatchPAT ONE (https://www.itamar-
| medical.com/professionals/disposable-hsaw...). It includes three
| main pieces you can see in the image there: a wristwatch-like
| sensor, and finger sensor, and a sensor you attach just above
| your sternum (primarily to track breathing events for sleep
| apnea). There is an associated phone app.
|
| Of course, thanks to capitalism, the device is marketed as
| disposable (I love how this is somehow portrayed as a bonus in
| the marketing materials). You're literally supposed to throw it
| away after a single use.
|
| So questions:
|
| 1. I've searched online but haven't found anything about people
| successfully hacking the device so it can be used multiple times.
| Has anyone seen any succesful hacks of this? If not, can anyone
| point me to some resources where I would even start to try to
| hack it? It communicates with the app over bluetooth.
|
| 2. Does anyone know of other medical-grade sensors that are
| usable by consumers that are _not_ disposable?
|
| I ask this because in the past I've just used the "Sleep as
| Android" app with a smartwatch and it was horribly wrong. It just
| checks for movement, and there are lots of times when I an
| exhausted, staring up at my ceiling not getting any sleep, but
| I'm not moving because I'm stuck in this exhaustion-but-not-
| sleeping state.
| Bud wrote:
| This is cool, but it'd be nice if they had evaluated Apple Watch,
| which is far more common than the devices they measured for. I've
| been tracking sleep for years with mine. My sense is that it's
| pretty accurate, but the problem is, I'm asleep during the time
| in which I could attempt to verify this! ;)
| justaguyonline wrote:
| The inability of wrist trackers to identify deep vs light sleep
| might cause frustration to some, but their automated sleep
| journaling is actually more than enough to help change the lives
| of people with insomnia.
|
| For some reason we had to slap complicated sleep scores and sleep
| classification on these things to take them seriously when all
| you needed was simple data gathered consistently over time.
| seventytwo wrote:
| No Apple Watch? Cmon. That's gotta be the most common one out
| there.
| Projectiboga wrote:
| Shh, this is marketing funded "research".
| AJ007 wrote:
| I was considering trying a sleep tracker again last year and
| before that happened solved my sleep problems. The amount of
| things I tried back in my 20s when it all was trivially simple: I
| just need to get an adequate amount of strength training and
| cardio exercise during the day and I sleep straight through the
| night.
|
| I was a hardcore flu.x/nightshift user. I left most lights off
| once the sun set. Now I'm a lot less convinced that lighting
| stuff matters for sleeping (but I would still follow it given the
| the whole cancer-light link thing.)
|
| Of course, all of the common sense stuff still applies: staying
| up all night, sleeping erratic schedules, drinking a gallon of
| water before going to bed, drinking a bunch of caffeine late at
| night, medications, stimulants, cocaine, trying to sleep in a
| room that's 90F+, having sleep apnea, someone drilling a
| jackhammer outside your bedroom window etc. If you do a
| combination of those things there is probably zero you can do to
| sleep consistently and well.
|
| I really wish I had understood this decades ago. Getting older it
| hits me how trivially simple solutions to certain problems are
| despite most running around in circles trying to solve them the
| entirety of their lives. To be fair, for sleeping poorly, I got
| an incredible amount of work done sitting in front of a PC 12
| hours a day. That's a number you probably can't hit if you are
| getting adequate exercise, quality food, and consistent deep
| sleep.
| sublinear wrote:
| > Getting older it hits me how trivially simple solutions to
| certain problems are despite most running around in circles
| trying to solve them the entirety of their lives.
|
| Oh, but you can't (can?) sell a self help book that way. It
| would only be a few pages long and everyone in the media would
| shit all over the author if it ever got traction. Hordes of
| bots on Reddit would be having fake conversations in threads
| that cry "science!" in an attempt to protect book sales based
| on TED talks and trending morning show style topics.
| slothtrop wrote:
| I really, really don't need a tracker and I doubt most people do.
| Even to satisfy curiosity they seem woefully lacking.
|
| What I do feel I need is an effective way (including cost-wise)
| to mitigate thermoregulation issues at night, i.e. to avoid
| waking up hot or cold, dialing in the correct bedding temperature
| at any moment.
|
| I thought a wool comforter might be the answer given its
| reputation, but I still fuss around with adding or removing
| layers. I basically sleep with a stack that changes depending on
| the weather.
| salad-tycoon wrote:
| I got a Fitbit in my mid twenties, noticed my heart rate going
| to 40. Had no idea of what that meant and asking around it
| turned out "it could be sleep apnea." Long story short, it was.
| I no longer fall asleep driving and I assume my heart is
| healthier than had I not known for many more years when I
| finally got married and had a sleeping partner (if I hadn't
| died on a highway first).
|
| My point is, I think these types of devices are wonderful
| screening tools. Cheap and imagine all the prevention we could
| have by screening widely and early.
| pedalpete wrote:
| Have you looked at Eight Sleep or ChiliPad?
| RecycledEle wrote:
| Activity trackers have to learn your behaviors, and that takes
| about a month.
|
| Testing them for 1 night without having worn them for a month or
| two first is silly.
| CharlesW wrote:
| Apple Watch was not tested, so for anyone interested, here are
| two videos by "The Quantified Scientist" creator Rob ter Horst, a
| postdoctoral researcher who specializes in bioinformatics
| (studying biology using machine learning and statistics).
|
| Apple Watch: Scientific Sleep Lab Test (Ultra vs 8) (2023) -
| https://www.youtube.com/watch?v=BQMIAkt4P48
|
| Apple Watch: Scientific Sleep Test (2022) -
| https://www.youtube.com/watch?v=LPqtfC70QTU
|
| (Thanks to _@LikelyClueless_ for the channel recommendation!)
| quinncom wrote:
| Kagi Summarizer key points of the second video:
|
| - WatchOS 9 beta introduces sleep stage tracking to the Apple
| Watch using a machine learning algorithm trained on
| polysomnography and Apple Watch heart rate and movement data.
|
| - The author tested the Apple Watch's sleep tracking against an
| EEG headband over 18 nights and found it agreed very well,
| especially for deep sleep tracking.
|
| - The Apple Watch detected 89% of deep sleep, 84% of light
| sleep, and 67% of REM sleep correctly compared to the EEG
| device.
|
| - Overall sleep stage tracking accuracy was better than the 37
| other devices the author has tested.
|
| - The Apple Watch also detected sleep cycles and awake periods
| reasonably well compared to the EEG.
|
| - Falling asleep and waking up times mostly differed by less
| than 10 minutes from the EEG.
|
| - Future improvements could include sleep coaching based on
| metrics.
|
| - Results may vary for other populations not well represented
| in the algorithm's training data.
|
| - The author used a beta version and polysomnography would
| provide better validation.
|
| - The Apple Watch's performance makes it a current leader for
| sleep and heart rate tracking accuracy.
| pedalpete wrote:
| After I was using an Oura ring for about a week, I came to the
| conclusion that knowing the data of how I slept provided very
| little value.
|
| I personally believe that the consumer grade devices are "good
| enough" because the micro-data is not that valuable anyway, and
| the way PSG works, there is no "correct" score. Get 5 different
| polysomnography's, you'll get 5 different measures of the same
| sleep.
|
| Once you're doing the sleep hygiene stuff, and experimented with
| all the magnesium stacks, ashwaganda, CBD, etc, etc, what do you
| do to actually ensure you are doing the most for your sleep?
|
| This is why I got into the sleeptech/neurotech space, and for the
| last few years we've been building a headband that monitors brain
| activity and increases the efficiency of deep sleep. The tech is
| backed by peer-reviewed and published research from discoveries
| which began in 2013. https://soundmind.co/research
|
| Yes, people like data and want to "understand" our sleep, but the
| quality of sleep is more than just the number of minutes you
| spend in each stage of sleep, just like fitness is more than the
| number of minutes you spend in each cardiovascular training
| stage.
|
| Our understanding of sleep today is at the same place our
| understanding of fitness was in the 70s, and sleep is just
| beginning to get is due importance. What we'll learn in the next
| 20 years, I believe, will lead to significant changes in health
| and longevity.
___________________________________________________________________
(page generated 2024-01-27 23:01 UTC)