[HN Gopher] Show HN: CodSpeed - Continuous Performance Measurement
___________________________________________________________________
Show HN: CodSpeed - Continuous Performance Measurement
Hi HN! We're Arthur and Adrien from CodSpeed. We're building a tool
measuring software performance before any production deployment,
catching performance regressions before they hit production
environments and reporting performance changes directly in Pull
Request comments. It's kind of like Codecov but for performance
measurement. Today, the go to solution to measure performance is
probably to use an APM(DataDog, Sentry, ...), continuously
analyzing your production environment. However, since those
solutions are operating on real environments they need real users
to experience poor performance in order to report issues and
unfortunately, performance remains an afterthought appearing only
at the end of the development cycle. Another possibility to
measure performance is to create benchmarks while developing and to
run them on a regular basis to have an idea of the performance
trend of your project. However, with this approach, the variance in
the results creates a lot of noise and it's rarely possible to
compare your results with the ones from a co-worker or a production
environment. To make consistent performance measurement as easy as
unit testing and fully integrated in CI workflows, we chose a
benchmark based solution. And, to eliminate the usual variance
associated with running them, we measure the number of instructions
and memory/cache accesses through CPU instrumentation performed
with Valgrind. This approach gives repeatable and consistent
results that couldn't be obtained with a time based statistical
approach, especially in extremely noisy CI and cloud environments.
We have been in closed beta for a few months, already being used by
popular open-source projects such as Prisma and Pydantic. Notably,
CodSpeed helped Pydantic through their Rust migration, empowering
them to make the library 17x faster:
https://docs.pydantic.dev/latest/blog/pydantic-v2/#performan...
Today, we're super excited to finally make the product available to
everyone. We currently support Python, Node.js and Rust and are
looking forward to integrate with more languages soon. The product
is and will be free forever for open-source projects. Also, we have
a per-seat pricing for private repository usage. We have a lot of
exciting features planned regarding additional integrations, such
as Database and GPU integrations that should come in upcoming
months. Don't hesitate to try out the product and give your honest
feedback. We're looking forward to your comments!
Author : art049
Score : 26 points
Date : 2023-07-11 15:02 UTC (2 days ago)
(HTM) web link (codspeed.io)
(TXT) w3m dump (codspeed.io)
| [deleted]
| foota wrote:
| Seems interesting... how did you land on valgrind vs some other
| means of simulation? Looking at valgrind, it sounds like
| cachegrind? Imo seems like the biggest gap is non instruction,
| non cache sources of latency, like mutex contention or kernel
| slowness? (Or does it capture kernel delays?)
|
| Pur most recent performance issues have been from someone
| accidentally creating a new thread pool in a request, from
| generating tons of stack traces in an error handling path, and
| from some thread hop delays. Sounds like the first two would
| probably be caught but maybe not the third?
| jeremiec wrote:
| [dead]
| art049 wrote:
| [dead]
| jammycrisp wrote:
| > we measure the number of instructions and memory/cache accesses
| through CPU instrumentation performed with Valgrind. This
| approach gives repeatable and consistent results that couldn't be
| obtained with a time based statistical approach, especially in
| extremely noisy CI and cloud environments.
|
| This is a neat approach! I'm curious how well it maps to actual
| perf degradations though. Valgrind models an old CPU with a more
| naive branch predictor. For low-level branch-y code (say a native
| JSON parser), I'd be curious how well valgrind's simulated
| numbers map to real world measurements?
|
| My probably naive intuition guesses that some low-level branchy
| code that valgrind thinks may be slower _may_ run fine on a
| modern CPU (better branch predictor, deeper cache hierarchy). I
| 'd expect false negatives to be rarer though - if valgrind thinks
| it's faster it probably is? What's your experience been like
| here?
| art049 wrote:
| >This is a neat approach! I'm curious how well it maps to
| actual perf degradations though. Valgrind models an old CPU
| with a more naive branch predictor. For low-level branch-y code
| (say a native JSON parser), I'd be curious how well valgrind's
| simulated numbers map to real world measurements?
|
| We didn't try it on this specific case but on we found that on
| branchy code valgrind does aggravate the branching cost.
| Probably, we could mitigate this issue by collecting more data
| relative to the branching and incorporate those in our reported
| numbers to map more accurately to the reality and more recent
| CPUs.
|
| >My probably naive intuition guesses that some low-level
| branchy code that valgrind thinks may be slower may run fine on
| a modern CPU (better branch predictor, deeper cache hierarchy).
| I'd expect false negatives to be rarer though - if valgrind
| thinks it's faster it probably is? What's your experience been
| like here?
|
| Totally! We never encountered a false positive in the reports
| yet. But as you mentioned since valgrind models an old CPU,
| it's likely to happen. But even though the cache simulated has
| a quite old, it still improves the relevance of our measures.
| When we have some time, we'd really enjoy refreshing the cache
| simulation of valgrind since it would probably eliminate some
| edge cases and reflect memory accesses more accurately.
___________________________________________________________________
(page generated 2023-07-13 23:02 UTC)