[HN Gopher] Launch HN: Patchwork (YC S24) - Automatically add st...
___________________________________________________________________
Launch HN: Patchwork (YC S24) - Automatically add structured logs
to your code
Hey HN! We are Sam, Alex and Ben from Patchwork Technologies
(https://getpatchwork.io). Patchwork automatically adds actionable
structured logs to your code. Big picture: we are building a next
generation logging and observability platform which gives engineers
the rich debugging context they need. There's a demo video
(https://youtu.be/ObIepiXfVx0), as well as a demo^ instance with
some results for you to explore at
https://hackernews.getpatchwork.io (no sign-in required!) The demo
shows the analysis and improvement (against a style guide) of
existing logs. It's set up for two OSS repositories,
https://github.com/elastic/cloud-on-k8s and our awesome friends at
Glasskube (https://github.com/glasskube/glasskube). We welcome
suggestions of other OSS Go repositories that you would like to see
added! Why we are building this: At our previous company, we
relied heavily on actionable, context-rich structured logs. They
were the unsexy but critical tool for managing complex software at
scale. When they're implemented well, they allow you to understand
the application state when things break. Structured logs are easier
to search and run analytics on. The trouble is that they are time-
consuming to implement properly - if you're reading this, you know
it's a chore. We all know it's usually an SRE (like Alex) that
finds out at 2 am that logs were added as an afterthought.
Unstructured and spammy - _Error: Failed to do task._ Datadog made
metrics easy; we are creating that moment for logs. We've grappled
with some tough technical challenges to build our product. First,
identifying log statements in codebases where logger symbols are
inconsistently named or output directly to stdout/stderr. We solved
this with SCIP, indexing symbol references at compile time. Next,
we needed to provide the LLM with method context and variable types
to ensure we accurately understand what the code is doing. This is
achieved using tree-sitter for parsing and SCIP for repo
navigation. Finally, we addressed the critical need to reason
whether specific data can/should be logged before bringing this all
together in an improved logging statement. We're staring down the
next challenge - where should logging statements go in new code?
Refactoring the logs in existing code bases today is a manual slog.
Even finding the existing logging and _printf_ statements is tricky
at scale. It's possible to rely on Copilot in VS Code or Cursor to
help construct logs as you write new code. However, there are many
more reasoning steps required than these tools are designed to do.
They are general purpose, zero-shot machines. Our first goal is to
reliably improve existing logging statements (as in the demo). In
the next couple of weeks we aim to prove-out adding logs to new
applications. Structured logs provide downstream benefits when it
comes to storage and query time - our final goal is to build a
storage layer that exploits this. Your honest feedback would mean
a lot to us. We have a lot of conviction that observability needs a
shake up, and that going back and getting the basics right is
valuable. We'd love to hear what you think works and doesn't work
in current approaches, and whether Patchwork solves a problem for
you. Chat with us on Discord: https://discord.gg/fkVTgX5s. ^If
you'd like to enrol additional repos and run analysis on them,
there's a magic link step.
Author : benjaminfh
Score : 70 points
Date : 2024-08-29 15:37 UTC (7 hours ago)
| btown wrote:
| This looks really, really cool. In the Python world, we can rely
| on tools like Sentry to be able to say "on any exception or
| logger.warning, crawl the stack and show me all the variables at
| every level." And we've also set up Honeycomb to record log
| entries as OpenTelemetry spans, which lets us see logs in context
| of database queries etc., and do structured queries about when
| and how certain things occur. Which means that we can just
| instruct our team to "err on the side of adding simple log
| statements, even if it's just a static string without variables;
| we can dig for those reactively."
|
| But in a codebase where far more of the complexity is in native
| code and doesn't cross network boundaries, I imagine folks feel
| like they're flying blind without those tools that we Python devs
| can take for granted. So Patchwork is desperately needed. I'd be
| very curious to see how your larger observability platform
| integrates with the broader world of OpenTelemetry, where the
| native code might be part of a broader distributed system. It's
| cool to see this space moving forward so quickly!
| benjaminfh wrote:
| A big part of our journey is trying to learn the different ways
| people generate, collect and then retrieve context, so we
| really appreciate you sharing how you're doing it - thanks! Am
| I right in understanding that you have a two-part approach:
| Sentry with simply messages and all variables up the call stack
| + Honeycomb/OT spans. What do you collect with the spans?
|
| Regarding the platform, transparently, it's early days for us
| and we're focusing on vanilla structured logs (generating
| better ones and then later storing them efficiently in
| something like ClickHouse), rather than tracing. Are there
| particular things you'd expect to see from a platform like
| this?
| benjaminfh wrote:
| Going to double tap my own reply here and come back to the
| OTel peice. Do you find there's a lot of effort required in
| adding attributes to spans through your code? A lot of people
| we speak to who are all-in on tracing are (we think) getting
| by with minimal/auto span implementation, without attributes.
| We're trying to keep digging and learning about how this
| looks at different orgs / figure out if this approach holds
| as systems get more complex. Would our workflow as applied to
| adding attributes and messages to spans be useful to you?
| anushnaidu wrote:
| It looks awesome..
|
| Does it checks the Log Levels and whether it's appropriate or not
| benjaminfh wrote:
| Great catch - we didn't include that rule in the demo but it's
| in the works. To share our thinking a little more: analysing
| the contents of the statement is easiest (but still quite hard)
| so we're nailing that down. Analysing the level requires a bit
| more careful understanding of what the method/code around it is
| attempting and where this logging statement sits (harder
| still). Soon we hope to nail adding statements to fresh code
| (where we'd have to understand where to put them, what the
| level should be, and then the contents).
| ucarion wrote:
| This is really cool. Is the "standards" thing basically some sort
| of LLM prompt? Like, do I just say in words "use slog, discourage
| logrus"?
| benjaminfh wrote:
| Right now it's where we specify which rules to apply to a repo
| when analysing and improving, plus any user-specific tweaks
| which we do splice in to prompts (still working on exactly how
| to balance freedom + necessary rails here). The rules
| themselves are sets of logic in their own right - essentially a
| mix of static code analysis to carefully prepare context for an
| LLM to reason over. This makes me think the UI for the
| standards page probably needs to be more binary with respect to
| the rules (then freeform for user prefs) - would that make more
| sense in your opinion?
| ucarion wrote:
| I don't really have much an opinion on the what, I think I
| just know how I want this to feel. My preference is whatever
| works, and then making it obvious how it works. Mystery meat
| config files feel like the opposite, but I totally
| acknowledge that's where you have to start when building
| this.
| benjaminfh wrote:
| That's very fair feedback, we appreciate it!
| aviperl wrote:
| On the topic of structured logs, can anyone point me towards
| where I might learn more about what people have learned over
| time?
|
| I'm new to the world of querying through my logs, but I can
| already see a benefit to logging with JSON...
|
| This is what I've defined as a solution for a need of mine for
| the moment: https://gist.github.com/avi-
| perl/b173fdc30219155eb9ee4bb3a21...
| benjaminfh wrote:
| Structured/JSON saves a tonne of time building regex parsers.
| The regex parsing at query-time is also pretty expensive. This
| is where Splunk excels - dealing with the noise with powerful
| querying. ClickHouse is also very performant at this, we hear.
| It's an expensive task though (computationally and cost wise)
|
| I thought this was well put together from Better Stack:
| https://betterstack.com/community/guides/logging/logging-bes...
|
| Charity, CTO of Honeycomb has strong views (which we enjoy a
| lot): https://charity.wtf/tag/observability-2-0/ - they come at
| it from a tracing/OT angle which is Honeycomb's forte, but we
| agree a lot on the intended outcomes - actionable (not
| spammy/noisy) + make it easy to gather the variable/state
| context in the context of a single event.
| brap wrote:
| Your landing page sucks, I don't care about your team or your
| marketing text, I want to see the code!
| benjaminfh wrote:
| Understood. The landing page so far is taking a back seat
| relative to making the product better for our early customers,
| which I'm sure you can appreciate. We hope to open source
| components of the product. In the mean time are there specific
| parts of the technical approach that you'd like us to see
| better described?
| elwell wrote:
| If the landing page doesn't lead me to eventually try the
| product, then this is a bad tradeoff. I would listen to this
| feedback more seriously.
| carlsverre wrote:
| Really cool idea! You may want to update the explainer video to
| use a different example of incorrect logger usage though. The
| current example
| (https://youtu.be/ObIepiXfVx0?si=-lumV_0ShKLpNl_h&t=203) claims
| that the use of Println is incorrect and logger.Info should be
| used instead. However, that would break the command in question
| as the command is outputting the current configuration as YAML.
| By using logger.Info, the additional logging prefix would make
| the user's life more difficult when they presumably output the
| command into a file or attempt to copy and paste the output. I'm
| sure that over time Patchwork will learn how to filter out these
| false-positives, but perhaps it's not the best thing to show off
| in the demo.
| benjaminfh wrote:
| Thanks for catching this and providing the feedback. We're
| iterating very hard to make sure the fixes are indeed always
| fixes!
|
| I won't waste the chance to engage a logging connoisseur -
| could you see yourself using this if the hit rate was 99%?
| carlsverre wrote:
| No problem. :)
|
| Possibly yes! If I had a project running in production with a
| team this seems very valuable as another layer of defense.
| With these kinds of scanning tools however, they are quickly
| ignored if they tend to emit noise. If at all possible (from
| a cost perspective) I'd focus on launching a free version for
| open source projects complete with a badge to add to the
| readme. Similar to other Linting/CI projects. Might help gain
| a bunch of traction in the short term, and if it's free folks
| may be likely to add it and leave it even if the accuracy
| isn't perfect which may be nice marketing for y'all.
| carlsverre wrote:
| Also... can you expand this to comments? Doesn't seem too
| different from logs. And then add in the fairly obvious
| auto-PR logic with the ability to respond to PR comments
| and now you're eliminating maintenance burden. Caveat: I
| dunno if anyone else is doing this, but this seems like a
| promising step.
| benjaminfh wrote:
| We're thinking alike. While we're iterating with
| customers, we're also thinking about how we can use it to
| contribute beneficially to projects (and in doing so
| prove it works reliably). One thing we are keeping a list
| of is popular OSS repos with notoriously spammy logs - if
| you know any, please let us know! We are planning to
| start with some of our YC batch mates' OSS projects once
| we get the quality right. I won't say exactly when we
| started showing the "fixes" but it was _very_ recently.
| Ensuring accuracy on identifying exceptions to the rules
| (aka the appropriate statements to touch in the first
| place) is the first thing we are perfecting.
|
| We could expand to comments. The code maintenance
| direction is a possibility but the reason we get out of
| bed right now is to make a worthy contribution to logging
| -> debugging -> SRE sleep :)
| andrewshadura wrote:
| Sorry, but the name's already taken: Patchwork is an online patch
| review tool.
| benjaminfh wrote:
| Thanks for flagging that one. It has turned out to be a very
| busy namespace. We're focusing on making sure the product
| sticks first... then figuring out if the name needs changing is
| a good problem to have! :D
| fnord123 wrote:
| > Refactoring the logs in existing code bases today is a manual
| slog.
|
| Nice.
| benjaminfh wrote:
| Glad to hear that resonates. Goes without saying that we get
| the best reception with people who've literately done manual
| surges on this. Speaking from experience?
___________________________________________________________________
(page generated 2024-08-29 23:00 UTC)