[HN Gopher] Osquery: An sqlite3 virtual table exposing operating...
___________________________________________________________________
Osquery: An sqlite3 virtual table exposing operating system data to
SQL
Author : signa11
Score : 167 points
Date : 2024-02-25 14:58 UTC (8 hours ago)
(HTM) web link (osquery.io)
(TXT) w3m dump (osquery.io)
| jeffbee wrote:
| Osquery: a privileged daemon written in an unsafe language that
| allows your junior sysadmins to take down all your machines at
| once or cause mystery performance blips that someone else has to
| diagnose for a year before they figure it out.
|
| First-person testimony.
| signa11 wrote:
| if possible, may you please elaborate, that would be so useful.
| thank you !
| VWWHFSfQ wrote:
| You should explain what you're talking about if you're going to
| publicly trash somebody else's project
|
| > allows your junior sysadmins to take down all your machines
| at once or cause mystery performance blips that someone else
| has to diagnose for a year
|
| Sounds like you have extremely poor administrative management
| of your computer systems.
| jeffbee wrote:
| I don't feel like I need to be nice to Facebook.
|
| The _raison d 'etre_ of this thing is to allow interactive ad
| hoc exploration of large-scale systems. It is, in other
| words, a thoroughly bad idea.
| convolvatron wrote:
| there may be problems with the implementation, or its
| usage, but I don't see why that's fundamentally a bad idea
| at all
| formerly_proven wrote:
| OS state introspection can get expensive quick and often has
| a performance impact beyond the process doing the sampling,
| because you're locking tons of kernel structures to read from
| them.
|
| Case in point, a simple "select * from processes" takes a
| solid 3 seconds of kernel time on my laptop.
|
| Now you might say, "well that's clearly a dumb idea because
| osquery certainly relies on vtab's colUsed field to avoid
| querying all sorts of expensive stuff when it doesn't have to
| so you really should only query what you need" and that's of
| course 100% true. But it's also a senior developer thought.
| Easy to see how an inexperienced person might make mistakes
| like this with any one of the dozens or hundreds of tables
| offered by osquery and cause performance issues.
|
| In terms of security, well it is clearly a kitchen sink
| project (there's a prometheus client in there, for example:
| https://osquery.io/schema/5.11.0/#prometheus_metrics), so
| there's a huge breadth of interfaces it talks to and files
| controlled by all sorts of people it parses, and the default
| does seem to be privileged usage, which is the general
| ballpark where AV engines and their highly dubious track
| record live.
| iJohnDoe wrote:
| Or they are a righteous developer who likes to trash people
| outside of development. It was a popular fad for a while,
| which worked both ways, such as, "Developers think they are
| experts in everything but couldn't figure out how to plug in
| a computer."
| foobiekr wrote:
| Everyone who has experienced osquery in production when
| mandated from above and used by less qualified admins knew
| exactly what he was getting at and the simple truth of it.
| w0de0 wrote:
| I've used it. At large enterprises with sophisticated
| deployments. I disagree.
| anotherhue wrote:
| Haha, yeah I've seen this. Highly tuned production server? No
| match for a junior security admin whose only tool is a hammer.
|
| It really really messes with windows defender too.
| w0de0 wrote:
| The senior sysadmins neglected to configure osquery's client
| performance safeguards and apply properly scoped access
| (observer mode!) on the server.
|
| Systems administration is the art of protecting everyone with
| less access from footguns while still and especially enabling
| their use of effective tools.
|
| First-person testimony.
| WhatIsDukkha wrote:
| https://osquery.readthedocs.io/en/latest/deployment/performa...
|
| Seems pretty straightforward as a start.
| wkat4242 wrote:
| I like osquery.
|
| Airwatch the Endpoint management system was working to
| incorporate this. It was pretty good. Unfortunately they were
| acquired by VMware which tried to maneuver the product into
| pretty unrelated VDI tech (Horizon). And of course VMware is
| completely in the shitcan now with everything being discontinued.
| We moved to Intune in the end. Which was not better by the way
| (nobody buys Microsoft because they're great, just because of the
| network effect). But it does have better long term outlook (no
| pun intended).
|
| But it was a good product while it lasted. The osquery
| integration was really useful for custom scripts. AirWatch had
| linux support 4 years ago even though Intune is only starting
| with that (and it hardly works).
| w0de0 wrote:
| Deploy your own, stand-alone osquery instance - it's open
| source.
|
| Airwatch/Workspace One was a terrible product even before the
| acquisitions. You may want to try FleetDM, an MDM product that
| deeply integrates osquery (an intersecting set of people are
| responsible for the two).
| wkat4242 wrote:
| I don't agree, I really liked airwatch and we managed tens of
| thousands of mobiles with it. It had its issues but every
| product does. The problem was that when we had to move to
| Intune some features we used in airwatch weren't even
| supported yet!
|
| It really made me laugh when Gartner put Intune in their
| magical Quadrant but not airwatch as Intune wasn't even
| feature complete at that point for basic mobile uses. I'm
| sure those Gartner guys just talk to the sales suits but
| don't actually try the products.
|
| But now we're stuck with Intune due to decisions made at top
| level.
| w0de0 wrote:
| When I used it, I was managing mostly Macs - and I never
| used it pre-vmware. This probably encouraged my flippancy!
|
| Is there even an established, reliable alternative to
| intune for Windows? I don't know.
| wkat4242 wrote:
| Yeah the Mac side was the worst. Many features didn't
| work reliably or were not updated quickly enough to keep
| up with OS updates. It was all very beta unfortunately.
|
| For Mac JAMF is pretty much the gold standard and I tried
| to get it but the leadership preferred a "single pane of
| glass" sadly.
|
| For Windows we always just used SCCM and even now we're
| only in hybrid mode with most functions in traditional
| management.
| bolle wrote:
| VMware Carbon Black uses osquery for the 'Cloud Audit and
| Remediation' part of the software.
| lolinder wrote:
| Definitely a tangent, but it's always interesting to see which of
| {a,an} people use for SQL and derivatives. It never occurred to
| me until today to pronounce "sqlite" as anything other than
| "sequelite".
| simonw wrote:
| D. Richard Hipp pronounces it Ess-Queue-El Ite.
| quickslowdown wrote:
| When I have to say SQLite out loud, my brain decides in the
| moment between "ess-cue-lite" and "see-cue-lite." I have no
| idea what criteria/heuristics I use in that moment to decide
| which way to pronounce it, but for whatever reason that's how
| it works for me.
|
| In my head I feel like I say "ess-cue-lite" pretty much every
| time.
| chrisweekly wrote:
| huh, I get the "ess-cue-lite" but not (at all) the "see-
| cue-lite" -- for me, the "see" sound only makes sense in
| the context of "sequel" (see-kwell-ite), which is how I
| instinctively tend to pronounce it.
| smaudet wrote:
| I think the former pronunciation is more correct, but...
|
| I see the point of the later if you are prone to
| verbalize acronyms:
|
| SQL => Verbalized Acronym: "See-Quell" vs Structured
| Query Language: "Ess-Queue-Ell"
|
| So then See-Cue-lite is a concatenation of the
| verbalization with the "lite". Whereas Ess-Cue-Lite is a
| concatenation of the acronym pronunciation with the
| "lite".
|
| This probably solve the former question of why some
| people say both and don't understand why. If the
| conversation/person is saying "SeeQuel" and "Sass" or
| "less" etc. then "See-Cue-Lite" fits that dialect. In a
| dialog with a lot of acronyms e.g. SQS, VPN, AWS, etc.
| you are probably more likely to go the acronym route i.e.
| "Ess-Cue-Lite".
|
| The term itself is a bastardization of an acronym and the
| "lite" term, I'd argue there's no correct pronunciation.
| samatman wrote:
| "like a mineral"
| o11c wrote:
| "Sequel" is explicitly wrong; it was a predecessor of SQL, not
| a pronunciation of it.
| euroderf wrote:
| Can't it just be pronounced "sclite", rhyming with (e.g.)
| "sprite" ?
|
| And when you apply FUSE (or FUSE-T or NFS or WebDAV) to its
| archive format "sqlar", getting a file system called "sqlarfs",
| then pronounce it as "sclarfs", rhyming with
| snarfs/scarfs/barfs.
| woodruffw wrote:
| osquery is a cool project, with a _lot_ of outstanding issues. It
| has a great deal of technical debt, including performance and
| security debts that don 't receive adequate attention. It also
| has a huge user community around it, but only a handful of active
| recurring contributors and companies actually funding development
| on it (and, even then, the bulk of the development is feature
| work rather than debt burndown).
|
| (Source: I worked on osquery's core and tables for 2-3 years for
| various clients, and my company employs several of the current
| maintainers.)
| PenguinCoder wrote:
| Agreed with that first statement. Some of the things osquery
| does especially with regards to Linux disk space usage, is
| frustrating to deal with.
| mikermcneil wrote:
| What did you run into on Linux?
| neodymiumphish wrote:
| It's also the base layer for Sophos' EDR platform.
| woodruffw wrote:
| It's the base layer for a _lot_ of EDRs :-)
| lopkeny12ko wrote:
| Huh, I thought osquery was a Facebook open source project.
| Surprised to see it under the Linux Foundation now. I wonder if
| Facebook still uses osquery in its current form or if they re-
| forked their own project to evolve separately in-house.
| woodruffw wrote:
| I don't know the exact dates, but Facebook turned osquery over
| to the community back in 2019 or 2020, after a period of
| uncertainly over (reasonable!) conflicting development
| interests[1]. It became a LF hosted project sometime after
| that.
|
| (I don't know whether they still use it in-house.)
|
| [1]: https://blog.trailofbits.com/2019/04/18/announcing-the-
| commu...
| mikermcneil wrote:
| History (tldr): https://fleetdm.com/handbook/company#history
|
| There's a podcast episode on it, if you're curious to hear it
| in Zach's and Mike's own words
| ciupicri wrote:
| From https://osquery.io/blog/osquery-foundation
|
| > 5 years ago
|
| > Today we are excited to announce the creation of the osquery
| foundation. Please read the Linux Foundation's official
| announcement. We have created an osquery/foundation repository
| on GitHub to host the technical charter.
|
| > The official repository has been renamed from
| facebook/osquery to osquery/osquery.
| gooseyard wrote:
| Have wondered for years whether Akamai's Query system
| (https://www.usenix.org/legacy/events/lisa10/tech/full_papers...,
| sadly never open sourced) had any influence on the creation of
| osquery. They are quite different architecturally but
| functionally similar.
| infogulch wrote:
| Sqlite is embedded, so how do you actually build it into your
| app? Is there a C file somewhere, an amalgamation published? I
| haven't been able to find it, but surely it exists.
|
| I'd like to build a Go binary with mattn/go-sqlite3 with osquery
| vtables included.
| w0de0 wrote:
| https://github.com/osquery/osquery/tree/master/osquery/sql
| tehlike wrote:
| is a shared library you can embed into your app.
|
| or you can use it in a server/client setting through some other
| project, but probably not you wanted
| archsurface wrote:
| I like the idea, but the queries seem to be longer than the shell
| commands, so the only advantage is not having to know the shell
| commands, and I'm not convinced you want people who don't know
| the system mucking about on the system.
| riku_iki wrote:
| its also useful when you are building some telemetry agent,
| since you don't need to parse output of each individual command
| tonymet wrote:
| I've been working on "life query" which is like osquery but all
| your life indicators: social, finance, health, business,
| productivity.
| russell_h wrote:
| Is it usable? Depending on exactly what it can access I might
| be interested to try it.
| tonymet wrote:
| Not at all because the data sources are collected with
| scraping tech. I like your enthusiasm let me think of how to
| better scale the product. Is this something you've been
| interested in for a while?
| 29athrowaway wrote:
| Ansible is my prefered way to do this.
| debarshri wrote:
| Few projects in the same realm that you should also checkout -
|
| [1] Steampipe (https://steampipe.io/)
|
| [2] InfraSQL (https://iasql.com/)
| michelpp wrote:
| system_stat is another one specific to PostgreSQL:
|
| https://github.com/EnterpriseDB/system_stats
| getvictor wrote:
| I started fixing some outstanding osquery issues recently. There
| are a lot of outstanding issues on GitHub that I haven't gone
| through.
|
| What's an issue preventing you from deploying osquery in your
| org?
| worewood wrote:
| This sounds like having a complete toolbox and saying "you know
| what would be cool? Using a drill to drive a nail into wood" and
| start banging the nail with it
___________________________________________________________________
(page generated 2024-02-25 23:00 UTC)