[HN Gopher] Launch HN: Haystack (YC W21) - Engineering analytics...
___________________________________________________________________
Launch HN: Haystack (YC W21) - Engineering analytics that don't
suck
Hey HN! I'm Julian, co-founder of Haystack
(https://usehaystack.io). We're building one-click dashboards and
alerts using Github data. While managing teams from startups to
more established companies like Cloudflare, my cofounder Kan and I
were constantly trying to improve our team and process. But it was
pretty tough to tell if our efforts were paying off. Even tougher
to tell where we could improve. We tried messing around with JIRA
which gave us story points and tickets completed but it didn't help
us dig deeper on where we could improve. We found a few tools that
integrated with Github measuring # of commits, lines of code, and
even comparing engineers using these metrics! - but we didn't like
that approach. We wanted to know (1) how quickly we deliver as a
team (2) what bottlenecks tend to get in the way and (3) as we make
adjustments, are they helping us improve? We scoured the internet
looking for every piece of research on the topic we could find,
talked to >500 engineering leaders working everywhere from startups
to FAANG, and started to learn which metrics helped answer our
questions and which ones just sucked. Once we had a clear picture
of what that looked like, we built Haystack. Haystack analyzes
pull requests on the team level, giving you "northstar" metrics
like cycle time, deployment frequency, change failure rate and 20+
more to help you improve delivery. Teams use Haystack to quickly
find bottlenecks like code review, experiment with changes like
smaller pull requests or automated tests, and see the result. Using
this feedback loop, the top 70% of Haystack users have increased
production deployments by 58% and achieved 30% faster cycle times
on average. We're lucky enough to work with some awesome teams at
Microsoft, Robinhood, and The Economist. As we continue to build
out our product, we'd love to hear any of your experiences with
engineering metrics, your thoughts about how to actually get them
right, and of course your disaster stories :)
Author : lovedev
Score : 67 points
Date : 2021-03-10 16:54 UTC (6 hours ago)
| dsiroker wrote:
| I just signed up and I'm blown away by how great the entire
| experience was. Their product is fantastic (better than Screenful
| or LinearB), their Slack community is awesome, and they even made
| me a personalized 5min onboarding video as soon as the data was
| ready to be presented. I'm a huge fan!
| lovedev wrote:
| Thanks for the kind words :)
| throw03172019 wrote:
| Heads up: site is not loading. iOS Safari & macOS Chrome.
|
| Mixed Content: The page at 'https://usehaystack.io/' was loaded
| over HTTPS, but requested an insecure favicon
| 'http://www.usehaystack.io/favicon.ico'. This request has been
| blocked; the content must be served over HTTPS.
|
| Edit: It appears to be an issue on the naked domain only.
| pedalpete wrote:
| Also won't load on Windows with Brave
| lovedev wrote:
| Looks like it was an issue with our Webflow set up. Nice
| catch! It should be up and running now :)
| jacques_chester wrote:
| Likewise not loading on MacOS + Firefox.
| lovedev wrote:
| Thanks for the heads up, looking into this now! :)
|
| In the meantime, we've opened up our team Slack (link below) if
| you'd like more info on Haystack (or just want to jam about
| metrics with other eng leaders from companies like Microsoft
| and The Economist).
|
| I believe this link should work:
| https://www.usehaystack.io/haystack-slack-community
|
| And sorry for the technical difficulties. Should be resolved
| shortly.
| throw03172019 wrote:
| Looks good now. Thanks!
| rememberlenny wrote:
| I absolutely love this. Congrats on building such an obviously
| missing problem. The efficiency/productivity obsession in me
| dances for joy at the idea of this.
|
| The last company I worked at used the "Accelerate: The Science of
| Lean Software and DevOps: Building and Scaling High Performing
| Technology Organizations" book as their bible. It was a wonderful
| way to structure team rituals and review cycles.
| lovedev wrote:
| Awesome! Every person on our team has Accelerate on our their
| desk :) highly recommended read and is a huge part of how we've
| shaped our product
| throwaway43432 wrote:
| I don't mean this critically but your product alarms me. I can
| imagine these reports being gamed and used to stack rank teams.
|
| I am trying to be more open minded about your work but I can't
| understand how developer productivity could be measured with any
| of these metrics. I'm not aware of any prior work or research
| that shows that either. Writing software is a complex task and
| quantising it in this way seems reductive and problematic.
|
| I would welcome being wrong about this. What am I missing?
| lovedev wrote:
| TBH I love the skepticism here. It's important to think about
| metrics critically in this way to make sure we're building
| truly useful applications of data.
|
| We've found that evaluating individuals, stack ranking teams,
| and measuring an individual's 'productivity' to be an
| ultimately futile effort.
|
| It's more important to focus on team delivery as described in
| many research studies. My favorite being the book Accelerate by
| nicole forsgren. If you haven't, I do encourage you to check it
| out!
|
| As the founder of Haystack I can be seen as biased so its
| probably more helpful to checkout their research across
| thousands of teams on what makes an 'elite engineering team'
| and how elite engineering teams ultimately help drive
| successful organizations across profitability, market share,
| and customer satisfaction :)
| throwawayr34234 wrote:
| Where is this product used at Robinhood?
| cbsudux wrote:
| This is awesome! Gitlab support will be nice :)
| lovedev wrote:
| Loving all the Gitlab requests! If you sign up and select
| "Gitlab" as your tool of choice, it will put you on our
| waitlist and help us prioritize that work :)
| pm wrote:
| Seems like a potentially useful product, but the branding looks
| like you're already preparing for an Atlassian acquisition.
| lovedev wrote:
| Our main goal is to build eng analytics that are genuinely
| useful for teams :)
|
| But I suppose we will be building a Bitbucket integration
| shortly?
| stigz wrote:
| Gitlab first, please.
| pm wrote:
| It was just an observation. The logo typeface and the blue
| are very similar to Atlassian's branding.
| colinchartier wrote:
| Hey, this looks really cool.
|
| In my line of work [1] I talk to a lot of developers about
| productivity, here are some questions about what you've built in
| the context of that experience:
|
| 1. I think (as you mentioned) "number of commits" or "number of
| pull requests" merged are proxies for productivity at best. Most
| teams that I've seen care more about "are the tasks for the epic
| being chipped down" - is that something your tool can help
| estimate as well? The progress bar for the epic? It'd be nice to
| get a table of the metrics and where they come from.
|
| 2. In the same vein, "scrum with epics" might not always be how
| teams decide on units of work, do you support other
| methodologies?
|
| 3. You mention deployment frequency a lot, but that seems tangent
| to developer productivity - Isn't getting things deployed quickly
| mostly for the benefit of the product folks? (getting feedback
| faster)
|
| 4. How does maintenance work fit into this? Does a developer who
| spends most of their time refactoring show up with "worse
| metrics" than someone pushing copy changes to the landing page?
|
| [1] CEO @ https://layerci.com (YC S20)
| lovedev wrote:
| Great question! I'll try to cover them :)
|
| 1. While we do use pull requests as a unit of measure, we don't
| actually consider # of commits and # of pull requests as great
| proxies for productivity. We mainly track Cycle Time,
| Deployment Frequency, Change Failure Rate, and Throughput which
| help encapsulate speed vs. quality of an engineering teams.
| It's important for engineering metrics to correlate to
| engineering work. Specifically, that the engineering team has
| full control over those metrics. When looking from an Epic,
| Story, or Task perspective - this is often controlled by other
| parties, making it more difficult to know if engineering itself
| is improving or not. In addition to this, Epics often don't
| reflect actual engineering work in the same way that pull
| requests in Github do making them problematic to focus on.
|
| 2. Because we focus on engineering work itself from Github.
| Haystack can be used with any methodology so long as the team
| is using Git!
|
| 3. From an engineering perspective, the main goal is '# of
| successful iterations'. Product's main goal is that those
| successful iterations are pointed in the right direction.
| Looking at # of successful iterations (aka speed, deployment
| frequency, and quality) allow us to see how often we deploy
| value to customers and what is the quality of those
| deployments. Focusing on these metrics allow engineering teams
| to improve 'how they deliver'.
|
| 4. We actively don't look at individual engineers and we
| especially don't compare them using any metric since we've seen
| that lead to some bad takeaways like the one you mentioned. We
| help measure at the team-level to simply highlight improvements
| and bottlenecks rather than cross-compare any team or
| individual which is like comparing apples to oranges
| g_delgado14 wrote:
| How is Haystack different from Pluralsight Flow (formerly
| GitPrime)?
|
| https://www.pluralsight.com/product/flow
| dsiroker wrote:
| They published a blog post comparing themselves to GitPrime and
| other competitors: https://www.usehaystack.io/blog/top-
| alternatives-to-gitprime...
| lovedev wrote:
| You beat me to it :)
| salsakran wrote:
| What pricing? Can't find it anywhere on the site.
| lovedev wrote:
| There's a free 14 day trial, then billed based on active devs
| (aka # of devs who create pull requests). Also, here's a link
| to the pricing section!
|
| https://www.usehaystack.io/#Pricing
| sdesol wrote:
| Hey it is great that engineering analytics is getting some love.
| I have something similar and I've found that you can generate
| some interesting insights from commit history if you look at how
| things are changing from a historical point of view.
|
| I'm not sure if my public server can withstand it, but you can
| play with the public version at https://public-001.gitsense.com
|
| If that doesn't survive, you can install GitSense (my product)
| with a single docker command, which you can find at
| https://gitsense.com
| lovedev wrote:
| Looks cool!
| sdesol wrote:
| Would love to chat. You can contact me in my bio.
| Robin_Message wrote:
| Just out of interest really, I've done some work for a company
| called BlueOptima (https://blueoptima.com) which does this in the
| enterprise space.
|
| github integration sounds good for getting up to speed quickly
| and if I was still a CTO I'd be trying it out.
| whoisjuan wrote:
| GitLab support in your future plans?
| lovedev wrote:
| Definitely! We support Github at the moment with plans to
| expand into Bitbucket and Gitlab shortly :)
___________________________________________________________________
(page generated 2021-03-10 23:01 UTC)