[HN Gopher] Launch HN: Roark (YC W25) - Taking the pain out of v...
___________________________________________________________________
Launch HN: Roark (YC W25) - Taking the pain out of voice AI testing
Hey HN, we're James and Daniel, co-founders of Roark
(https://roark.ai). We built a tool that lets developers replay
real production calls against their latest Voice AI changes, so
they can catch failures, test updates, and iterate with confidence.
Here's a demo video: https://www.youtube.com/watch?v=eu8mo28LsTc.
We ran into this problem while building a voice AI agent for a
dental clinic. Patients kept getting stuck in loops, failing to
confirm insurance, or misunderstanding responses. The only way to
test fixes was to manually call the agent or read through hundreds
of transcripts, hoping to catch issues. It was slow, frustrating,
and unreliable. Talking to other teams, we found this wasn't just
a niche issue - every team building Voice AI struggled to validate
performance efficiently. Debugging meant calling the agent over and
over. Updates shipped with unknown regressions. Sentiment analysis
relied only on text, missing key audio cues like hesitation or
frustration, which often signal deeper issues. That's why we built
Roark. Instead of relying on scripted test cases, Roark captures
real production calls from VAPI, Retell, or a custom-built agent
via API and replays them against your latest agent changes. We
don't just feed back text, we preserve what the user said, how they
said it, and when they said it, mimicking pauses, sentiment, and
tone up until the conversation flow changes. This ensures your
agent is tested under real-world conditions, not just synthetic
scripts. For each replay that we run, Roark checks if the agent
follows key flows (e.g. verifying identity before sharing account
details). Our speech based evaluators also detect sentiments such
as frustration and confusion, long pauses, and interruptions -
things that regular transcripts miss. After testing, Roark
provides Mixpanel-style analytics to track failures, conversation
flows, and key performance metrics, helping teams debug faster and
ship with confidence. Instead of hoping changes work, teams get
immediate pass/fail results, side-by-side transcript comparisons,
and real-world insights. We're already working with teams in
healthcare, legal, and customer service who rely on Voice AI for
critical interactions. They use Roark to debug AI failures faster,
test updates before they go live, and improve customer experiences
- without manually calling their bots dozens of times. Our product
isn't _quite_ ready yet for self-service, so you'll still see the
dreaded "book a demo" on our home page. If you're reading this,
though, we'd love to fast-track you, so we made a special page for
HN signups here: https://roark.ai/hn-access. If you're working on
Voice AI and want to try us out, please do! Would love any
feedback, thoughts, or questions from the HN community!
Author : zammitjames
Score : 29 points
Date : 2025-02-17 16:54 UTC (6 hours ago)
| Closi wrote:
| It looks great! Although the demo shows horrible security
| practices...
|
| Clearly authentication shouldn't rely on prompt engineering.
|
| Particularly when at the end of the demo it says "we have tested
| it again and now it shows that the security issue is fixed" - No
| it's not fixed! It's hidden! Still a gaping security hole.
| Clearly just a very bad example, particularly considering the
| context is banking.
| zammitjames wrote:
| Appreciate the feedback! Completely agree - authentication
| should be handled at the system level, not just in prompts.
| This demo is meant to showcase how teams can build test cases
| from real failures and ensure fixes work before deployment.
| We'll consider using a better example.
| oxidant wrote:
| Your post suggests authorization as a feature:
|
| > For each replay that we run, Roark checks if the agent
| follows key flows (e.g. verifying identity before sharing
| account details)
|
| I don't know if AI will be more susceptible or less
| susceptible to phishing than humans, but this feels like a
| bad practice.
| zammitjames wrote:
| Appreciate the feedback! To clarify, Roark isn't handling
| authentication itself - it's a testing and observability
| tool to help teams catch when their AI fails to follow
| expected security protocols (like verifying identity before
| sharing sensitive info).
|
| That said, totally fair point that this example could be
| clearer--we'll keep that in mind for future demos. Thanks
| for calling it out!
| Cilvic wrote:
| Super cool to see this just now. We are building in the space of
| computer screen analysis and started to experience something
| similar, hence want to build something similar for Pixels instead
| of voice.
|
| Would love to chat to you, jan@kontext21.com
| zammitjames wrote:
| Thanks, Jan! Would love to hear more about what you're working
| on! Just sent you an email.
| mercurialsolo wrote:
| As someone who's building a personal work assistant for voice - I
| see the merit in automating test case generation and validation.
|
| All products in this space by YC teams are targeted at scaled
| voice agent startups or teams.
|
| - Roark (https://roark.ai/)
|
| - Hammin (https://hamming.ai/)
|
| - Coval (https://www.coval.dev/)
|
| - Vocera (https://www.vocera.ai/)
|
| How do you differentiate - who is this for? Voice agent devs
| paying $500/mo. for early stage software?
| zammitjames wrote:
| Great question! There's been a lot of movement in this space,
| but most existing solutions focus on simulation-based testing--
| generating synthetic test cases or scripted evaluations.
|
| Roark takes a different approach: we replay real production
| calls against updated AI logic, preserving actual user inputs,
| tone, and timing. This helps teams catch failures that scripted
| tests miss--especially in high-stakes industries like
| healthcare, legal, and finance, where accuracy and compliance
| matter.
|
| Beyond replays, we provide rich analytics, sentiment & vocal
| cue detection, and automated evaluations, all based on audio--
| not just transcripts. This lets teams track frustration, long
| pauses, and interruptions that often signal deeper issues.
|
| Would love to hear more about your assistant - how are you
| thinking about testing and iteration?
| echelon wrote:
| Wow, YC has so much conviction in this space that they invested
| in four of the same company.
|
| If anyone ever doubts if YC will invest in direct competitors,
| this should be your answer.
|
| Roark: Best of luck. Talk to customers, talk to customers, talk
| to customers!
| NewUser76312 wrote:
| Looks somewhat useful for voice AI QA.
|
| But I wonder if a company is deploying voice AI, wouldn't they
| have their own testing and quality assurance flows?
|
| Is this targeted at companies without an engineering department
| or something? In which case I find it surprising they're able to
| slot in some voice AI assistant in the first place.
| zammitjames wrote:
| Great question! Most teams deploying voice AI do have testing
| and QA flows, but they're often manual, brittle, or incomplete.
| Unlike traditional software, voice agents don't have structured
| inputs and outputs -- users phrase things unpredictably, talk
| over the bot, or express frustration in subtle ways.
|
| Some engineering teams try to build internal testing
| frameworks, but it's a massive effort - they have to log and
| store call data, build a replay system, define evaluation
| criteria, and continuously update it as the AI evolves. Most
| don't want to spend engineering time reinventing the wheel when
| they could be improving their AI instead.
|
| The teams that benefit most from Roark are the ones with strong
| QA processes -- they already know how critical testing is, but
| they're stuck with brittle, time-consuming, or incomplete
| workflows.
| jnovek wrote:
| I noticed this on your website regarding transcription --
|
| "More accurate than Deepgram, supporting 50+ languages with a
| word error rate of just 8.6%."
|
| Can you explain how this helps me? At the end of the day you are
| not my transcriber, wouldn't I want to test using transcriptions
| produced by the transcriber that I'm actually using in
| production?
| aeternum wrote:
| This seems useful for issues early in the convo but what if the
| AI responses diverge from the recorded convo prior to the issue
| being hit?
___________________________________________________________________
(page generated 2025-02-17 23:00 UTC)