https://eclecticlight.co/2023/10/07/sonomas-log-gets-briefer-and-more-secretive/

Skip to content
[eclecticlight]

The Eclectic Light Company

Macs, painting, and more

Main navigation

Menu

  * Downloads
  * M1 & M2 Macs
  * Mac Problems
  * Mac articles
  * Art
  * Macs
  * Painting

hoakley October 7, 2023 Macs, Technology

Sonoma's log gets briefer and more secretive

If you used Macs before Sierra, chances are that you were happy to
browse their logs when necessary, for example to check whether your
Time Machine backups were working correctly, or quietly throwing
errors. At that time, the logs were traditional text-based records
like those in other variants of Unix, and browsed using the Console
app in /Applications/Utilities.

Browsing your logs using Console is a vital part of diagnosing many
problems.Traditional text logs in OS X 10.10 Yosemite before they
changed in Sierra.

I started to become irritated by them as long ago as El Capitan,
writing:
"Having excised most of the old dross and accumulated extensions, and
running a fairly clean El Capitan system, my All Messages [in
Console] clocks up around 4000 messages every 8 or 9 hours. At its
worst - straight after the OS X 10.11.5 update, for example - it can
fill those 4000 message slots in a minute or two, but even overnight
when the system is not asleep it will seldom stretch to much longer
than 12 hours. A traditional Unix wizard would have the screaming
ab-dabs at this. We used to tweak, tune, and pare down systems until
they barely coughed into their logs. It used to be the mark of a
well-loved system."

Little did we realise then that Sierra was going to change all that,
and by Mojave we'd be enduring 4,000 and more log entries in a
second, when our Macs were feeling loquacious. That was because Apple
introduced the Unified log, with its entries written not in plain
text but compressed binary format. This was the death-blow for the
casual reader of logs: for a start, the replacement Console app was
unable to access any log entries made in the past, and its tools
were, and remain, woefully inadequate for tackling the increasing
torrent of log entries.

Despite its many great strengths, the Unified log has suffered two
problems that are limiting its usefulness in Sonoma: its diminishing
period of coverage, and censorship.

Period of coverage

Rather than rolling its log files by their age, and keeping the last
five days of logs, macOS maintains them by size. Like its
predecessors, that means logd, its maintainer, only keeps the last
520 MB or so of regular Persist log entries collected at any time,
and purges older entries. Some older log entries are retained, but
it's pot luck as to whether those you're interested in are among
them, and even if they are, other related entries may have been
removed already. If you're interested to see how this works, logd
keeps records in the logdata.statistics.[n].txt files in /var/db/
diagnostics, but those too are purged so you can only look back a few
days.

In the early days of the Unified log, I was amazed to discover that
it kept log records for as long as 20 days or more. In Sonoma, my
main working Mac is struggling to keep them for as long as 24 hours.
There's no way for the user to increase that period to compensate for
the increased rate of writing log entries by macOS. If I don't catch
an event well within 24 hours of it happening, chances are that its
log entries will already have been purged when I go to look for them.

Privacy

Censorship to maintain privacy is an important feature in the Unified
log, and is responsible for so many of its entries being peppered
with <private> instead of revealing potentially sensitive data. The
mechanism for this is robust: when collecting entries for the log,
sensitive content is excluded, then displayed with that marker. As
that sensitive content isn't stored in the first place, there's no
way to unmask that content once it has been collected and written to
the log.

Currently, the only way to see unmasked log entries is through a
special profile. Once that has been enabled, and until it's disabled
again, all privacy should be removed from the log, and those
<private> markers replaced with real data.

Only Apple has now decided that some log content is too private to
reveal even when privacy has been removed. I came across this
recently, in multiple log entries, when researching Web Apps, in
entries like
ShareKit Start performing service <private> with items: <private>
ShareKit filteredItemsFromItems:<private> [-1]--> <private>
ShareKit Final shared items for <private>: <private>
ShareKit Perform custom action (<private> [<private>])
ShareKit Custom action (<private>) does not support async, running
sync block

Of course, it may be possible to turn this additional privacy off,
but the documentation leads us once again to a guessing game. Apple
documents the Enable-Private-Data profile key here, where it refers
to it controlling "private data logging". But in its account on log
privacy settings, Apple refers to two distinct privacy settings that
can be applied to the content of log entries: the "standard option"
'private', and sensitive, described as "the option to always redact
interpolated values that contain sensitive information." Does this
mean that setting Enable-Private-Data to false doesn't unmask content
designated sensitive, only that set as 'private'?

Like so many great ideas, the Unified log is suffering from
profligate overuse. Who really needs to see full dictionary dumps
from RunningBoard? Is there no better way to record that information
than in the log?

Share this:

  * Twitter
  * Facebook
  * Reddit
  * Pinterest
  * Email
  * Print
  * 

Like this:

Like Loading...

Related

Posted in Macs, Technology and tagged Console, log, macOS 10.12,
macOS 14, privacy, Sierra, Sonoma. Bookmark the permalink.

16Comments

Add yours

 1. 1 [bca931598b2f]
    Udo Thiel on October 7, 2023 at 10:08 am
    Reply

    This stuff is getting out of hand. On a single-user machine it is
    plain useless, and on a multi-user machine, every non-privileged
    user has to trust admin anyway, there is running and hiding,
    especially if it is the Bastard Operator from Hell (does anyone
    remember him?)

    At the minimum, Apple should add log customisation functionality
    to Console.app directly.

    BTW, I came across your profile you mentioned only a few weeks
    ago, and since comments were closed for that article, I couldn't
    ask my question, which is: given that I have an Apple developer
    account myself, how do I create my own profile? Thanks in
    advance.

    LikeLiked by 1 person

      + 2 [87cc8acbb0b9]
        hoakley on October 7, 2023 at 4:45 pm
        Reply

        Although you can write a profile in XML and sign it, the
        simple way is to use Apple Configurator 2. The process is
        described in its Help book.
        Howard.

        LikeLike

 2. 3 [95c92c6c33c2]
    SarahB on October 7, 2023 at 12:43 pm
    Reply

    Sorry to hear how things have been going but I felt it in so many
    other ways that affect me also with each macOS update. Mostly how
    it keeps putting priority over apple's services like Apple TV and
    stuff, but then keeps taking away the functions that were useful
    for me.

    Like for people like me who have their own media, apple keeps
    taking away things we use in the apps like TV, Music and stuff
    are so focused on if you use apple's paid apps like TV+ and Apple
    Music. Like even in the TV app shows I purchased show up proper,
    but if I go to my computer on the Apple TV for shows I purchased
    so often the cover art is missing or broken. Never used to be
    this way.

    I miss how the user had more control on macOS with things. Or
    just access to things. I know it seems this often helps how the
    apps you create work. I remember Mike at Bombich software kept
    getting more frustrated even with each macOS release and changes
    that made it more difficult to backup.

    I even had to remove OCLP from my mac's that use it since this
    1.0.1 release seems bad. All my mac's using it and running Sonoma
    were running too hot. So I put them back to the last macOS they
    can run and now running 20C cooler.

    LikeLiked by 2 people

      + 4 [87cc8acbb0b9]
        hoakley on October 7, 2023 at 4:46 pm
        Reply

        I'm sorry to hear that you've had problems with the OCLP
        update. Have you informed the developers?
        Howard.

        LikeLike

          o 5 [95c92c6c33c2]
            SarahB on October 7, 2023 at 5:07 pm
            Reply

            Maybe was too soon to try it but it was released. Just
            found 3 different mac's I was using it with all ran 20C
            hotter than when on OCLP and Ventura. Also they ran so
            slowly. Booting took so long. Just felt like it really
            slowed down those mac's. 2012/2014 mini's and 2014
            MacBook Air. The mini's all have 16gb of ram but the air
            8gb. Still they ran ok on Ventura.

            I kind of prefer to run the OS native though since then I
            can turn on file vault and SIP and other things which I
            can't do on OCLP. Also the Mac runs a lot better even
            though it's not on the current OS anymore. I think maybe
            it's just time to stop since these are getting too old I
            guess. Even the new wallpapers would kill the Mac and
            have the fans on max.

            On the air though was terrible, I would click something
            then a long delay before anything happened. Working on
            the last Mac now the 2014 mini to put it back on Monterey
            and no OCLP. Then won't be using OCLP on any Mac here
            anymore.

            People I told said they were not having these issues but
            not sure if people use them like I do. It's ok though.
            Can't expect miracles.

            LikeLiked by 1 person

 3. 6 [db74b6f3c006]
    John Gilbert on October 7, 2023 at 11:16 pm
    Reply

    Some time ago, I got irritated by verbosity and by losing logs.
    This was even though I don't use them very often.
    So I used sudo log config -mode "persist:off" commands to exclude
    excess entries clogging up the logs. (I think you taught me how
    to do that).
    And then created a simple launch daemon and script to create
    daily log archives and keep them for 3 months. I store about 1 GB
    every day.

    LikeLiked by 1 person

      + 7 [87cc8acbb0b9]
        hoakley on October 8, 2023 at 6:34 am
        Reply

        Thank you. Yes, I too have turned off some subsystems from
        remaining in the Persist logs. It's reaching the stage where
        I wonder if I need to turn off most of macOS, though, which
        defeats the whole point, surely.
        Howard.

        LikeLike

 4. 8 [265da7155736]
    Thomas on October 8, 2023 at 1:14 am
    Reply

    System logs have become unusable for me. I find it extremely
    confusing. Recently for example I spent a day trying to find a
    way to configure the logging system to show syslog debug logging
    of a process I had no control over in the unified log. Nothing I
    did to asl.conf seemed to have any effect. I had to give up and
    debug it some other way... If anyone knows how to do that I would
    appreciate any suggestions.

    LikeLiked by 2 people

      + 9 [87cc8acbb0b9]
        hoakley on October 8, 2023 at 6:38 am
        Reply

        I have dozens of articles here about the unified log, and
        free log browsers, including Ulbow. Perhaps if you can
        explain what you were trying to do, I can help.
        Howard.

        LikeLike

          o 10 [265da7155736]
            Thomas on October 8, 2023 at 8:23 pm
            Reply

            Thanks for offering to help  - The problem is not the
            filtering, the problem is that syslog level >5 is simply
            not showing up in the logging system anymore. (syslog -s
            -l 5 test5 does but -l >5 does not). I looked at /etc/
            asl.conf which seems to be the cause because of the line
            "? [<= Level notice] store" - however changing this to ?
            [<=Level debug] has no effect, neither creating things in
            the asl folder. My question: How can I get syslog -s -l 6
            or 7 to show in the logging system? Thanks!

            LikeLiked by 1 person

              # 11 [87cc8acbb0b9]
                hoakley on October 8, 2023 at 8:39 pm

                Syslog and asl were replaced in macOS 10.12 by the
                Unified log. No code should be still trying to write
                to logs that were superseded in 2016.
                Howard

                LikeLike

 5. 12 [cropped-imag]
    Week 41 - 2023 - This Week In 4n6 on October 8, 2023 at 11:18 am
    Reply

    [...] Howard Oakley at 'The Eclectic Light Company'Sonoma's log
    gets briefer and more secretive [...]

    LikeLike

 6. 13 [39a89951c96a]
    aONe on October 8, 2023 at 4:57 pm
    Reply

    Thanks for the reading . I miss the days where I didn't need to
    filter the console for it to be useful.

    LikeLiked by 1 person

      + 14 [87cc8acbb0b9]
        hoakley on October 8, 2023 at 7:50 pm
        Reply

        Thank you. So do I.
        Howard.

        LikeLike

 7. 15 [265da7155736]
    Thomas on October 8, 2023 at 9:21 pm
    Reply

    I am not sure I understand.... clearly syslog on notice level and
    above makes it into the new logging system. I was trying to debug
    pureFTP (without changing its implementation) and like many unix
    open source tools it uses syslog. So are you saying there is no
    way to change the configuration so that other levels show up?

    LikeLiked by 1 person

      + 16 [87cc8acbb0b9]
        hoakley on October 8, 2023 at 9:56 pm
        Reply

        I'm sorry: I suggest that you ask someone who maintains such
        open source tools on macOS. I have absolutely no idea whether
        syslog still works, or how it does, as I don't think any apps
        written for macOS have used it for seven years.
        Howard.

        LikeLike

Leave a Reply Cancel reply

 [                                             ] 
 [                                             ] 
 [                                             ] 
 [                                             ] 
 [                                             ] 
 [                                             ] 
 [                                             ] 
D[                                             ] 

This site uses Akismet to reduce spam. Learn how your comment data is
processed.

Quick Links

  * Downloads
  * Mac Troubleshooting Summary
  * M1 & M2 Macs
  * Mac problem-solving
  * Painting topics
  * Painting
  * Long Reads

Search

Search for: [                    ] [Search]
Monthly archives

  * October 2023 (19)
  * September 2023 (77)
  * August 2023 (72)
  * July 2023 (79)
  * June 2023 (73)
  * May 2023 (79)
  * April 2023 (73)
  * March 2023 (76)
  * February 2023 (68)
  * January 2023 (74)
  * December 2022 (74)
  * November 2022 (72)
  * October 2022 (76)
  * September 2022 (72)
  * August 2022 (75)
  * July 2022 (76)
  * June 2022 (73)
  * May 2022 (76)
  * April 2022 (71)
  * March 2022 (77)
  * February 2022 (68)
  * January 2022 (77)
  * December 2021 (75)
  * November 2021 (72)
  * October 2021 (75)
  * September 2021 (76)
  * August 2021 (75)
  * July 2021 (75)
  * June 2021 (71)
  * May 2021 (80)
  * April 2021 (79)
  * March 2021 (77)
  * February 2021 (75)
  * January 2021 (75)
  * December 2020 (77)
  * November 2020 (84)
  * October 2020 (81)
  * September 2020 (79)
  * August 2020 (103)
  * July 2020 (81)
  * June 2020 (78)
  * May 2020 (78)
  * April 2020 (81)
  * March 2020 (86)
  * February 2020 (77)
  * January 2020 (86)
  * December 2019 (82)
  * November 2019 (74)
  * October 2019 (89)
  * September 2019 (80)
  * August 2019 (91)
  * July 2019 (95)
  * June 2019 (88)
  * May 2019 (91)
  * April 2019 (79)
  * March 2019 (78)
  * February 2019 (71)
  * January 2019 (69)
  * December 2018 (79)
  * November 2018 (71)
  * October 2018 (78)
  * September 2018 (76)
  * August 2018 (78)
  * July 2018 (76)
  * June 2018 (77)
  * May 2018 (71)
  * April 2018 (67)
  * March 2018 (73)
  * February 2018 (67)
  * January 2018 (83)
  * December 2017 (94)
  * November 2017 (73)
  * October 2017 (86)
  * September 2017 (92)
  * August 2017 (69)
  * July 2017 (81)
  * June 2017 (76)
  * May 2017 (90)
  * April 2017 (76)
  * March 2017 (79)
  * February 2017 (65)
  * January 2017 (76)
  * December 2016 (75)
  * November 2016 (68)
  * October 2016 (76)
  * September 2016 (78)
  * August 2016 (70)
  * July 2016 (74)
  * June 2016 (66)
  * May 2016 (71)
  * April 2016 (67)
  * March 2016 (71)
  * February 2016 (68)
  * January 2016 (90)
  * December 2015 (96)
  * November 2015 (103)
  * October 2015 (119)
  * September 2015 (115)
  * August 2015 (117)
  * July 2015 (117)
  * June 2015 (105)
  * May 2015 (111)
  * April 2015 (119)
  * March 2015 (69)
  * February 2015 (54)
  * January 2015 (39)

Tags

APFS Apple AppleScript Apple silicon backup Big Sur Blake bug
Catalina Consolation Console Corinth diagnosis Disk Utility Dore El
Capitan extended attributes Finder firmware Gatekeeper Gerome HFS+
High Sierra history of painting iCloud Impressionism iOS landscape
LockRattler log logs M1 Mac Mac history macOS macOS 10.12 macOS 10.13
macOS 10.14 macOS 10.15 macOS 11 macOS 12 macOS 13 malware Mojave
Monet Monterey Moreau MRT myth narrative OS X Ovid painting Pissarro
Poussin privacy realism Renoir riddle Rubens Sargent scripting
security Sierra SilentKnight SSD Swift Time Machine Turner update
upgrade Ventura xattr Xcode XProtect

Statistics

  * 15,603,767 hits

Blog at WordPress.com.

Footer navigation

  * About & Contact
  * Macs
  * Painting
  * Language
  * Tech
  * Life
  * General
  * Downloads
  * Mac problem-solving
  * Extended attributes (xattrs)
  * Painting topics
  * Hieronymus Bosch
  * English language
  * LockRattler: 10.12 Sierra
  * LockRattler: 10.13 High Sierra
  * LockRattler: 10.11 El Capitan
  * Updates: El Capitan
  * Updates: High Sierra and later
  * LockRattler: 10.14 Mojave
  * SilentKnight, silnite, LockRattler, SystHist & Scrub
  * DelightEd & Podofyllin
  * xattred, Metamer, Sandstrip & xattr tools
  * 32-bitCheck & ArchiChect
  * XProCheck, T2M2, Ulbow, Consolation and log utilities
  * Cirrus & Bailiff
  * Taccy, Signet, Precize, Alifix, UTIutility, Sparsity, alisma
  * Revisionist & DeepTools
  * Text Utilities: Nalaprop, Dystextia and others
  * PDF
  * Keychains & Permissions
  * LockRattler: 10.15 Catalina
  * Updates
  * Spundle, Cormorant, Stibium, Dintch, Fintch and cintch
  * Long Reads
  * Mac Troubleshooting Summary
  * LockRattler: 11.0 Big Sur
  * M1 & M2 Macs
  * Mints: a multifunction utility
  * LockRattler: 12.x Monterey
  * VisualLookUpTest
  * Virtualisation on Apple silicon
  * LockRattler: 13.x Ventura
  * System Updates
  * LockRattler: 14.x Sonoma

Secondary navigation

  * Search

Post navigation

 
Painting on the edge: Spinning wheels
Saturday Mac riddles 224
Search for: [                    ] [Search]
Begin typing your search above and press return to search. Press Esc
to cancel.

  * Follow Following
      + [croppe] The Eclectic Light Company
        Join 3,360 other followers
        [                    ]
        Sign me up
      + Already have a WordPress.com account? Log in now.
  * 
      + [croppe] The Eclectic Light Company
      + Customize
      + Follow Following
      + Sign up
      + Log in
      + Copy shortlink
      + Report this content
      + View post in Reader
      + Manage subscriptions
      + Collapse this bar

 

  

Loading Comments...
 
Write a Comment... [                    ]
Email (Required) [                    ] Name (Required)
[                    ] Website [                    ]
[Post Comment]

 
%d bloggers like this:

[b]