[HN Gopher] Launch HN: Meticulate (YC W24) - LLM pipelines for b...
___________________________________________________________________
Launch HN: Meticulate (YC W24) - LLM pipelines for business
research
Hi HN, Meticulate gives finance professionals easy access to world
class business research--think competitive landscapes, market
sizings, customer segmentation etc. Here's a video of a competitive
landscape generation in action (https://youtu.be/aJ-slHcp32c)and
we've taken down the signup wall today so you can try it directly
at https://meticulate.ai/. Some background on "business research":
investment and consulting teams sink many hours a week into
researching companies, markets, and products. This work is time-
sensitive and exhausting, but crucial to big decisions like company
acquisitions or pricing model changes. At large financial services
firms, much of this work is offshored to external providers, who
charge thousands of dollars per project and are often slow and low-
quality. Small teams lack the budget and consistent flow of work to
employ these resources. We're building an automation solution that
brings a fast, easily accessible, and defendable research resource.
Meticulate uses LLMs to emulate analyst research processes. For
example, to manually build a competitive landscape like this one:
https://meticulate.ai/workflow/65dbfeec44da6238abaaa059, an analyst
needs to spend ~2 hours digging through company websites, forums,
and market reports. Meticulate replicates this same process of
discovering, researching, and mapping companies using ~1500 LLM
calls and ~500 webpages and database pulls, delivering results 50x
faster at 50x less cost. At each step, we use an LLM as an agent
to run searches, select and summarize articles, devise frameworks
of analysis, and make small decisions like ranking and sorting
companies. Compared to approaches where an LLM is being used
directly to answer questions, this lets us deliver results that (a)
come from real time searches and (b) are traceable back to the
original sources. We've released two workflows: building
competitive landscapes and market maps. We designed it with an
investor running diligence on a company as the target use case but
we're seeing lots of other use cases that we didn't originally have
in mind--things like founders looking for alternative vendors for a
product they're purchasing; sales reps searching for more prospects
like one they've already sold to; consultants trying to understand
a new market they are unfamiliar with, and more. The main
challenges we've been overcoming are preventing quality degradation
along multi-step LLM pipelines where an error on one step can
propagate widely, and dealing with a wide range of data quality.
We're working hard on our next set of workflows and would love for
you to give it a try at https://meticulate.ai and would appreciate
feedback at any level!
Author : JPalakapilly
Score : 57 points
Date : 2024-03-14 16:51 UTC (6 hours ago)
| cubecul wrote:
| Congrats on the Launch HN!
|
| I've heard a few investor types say something like "You know
| what's surprisingly fun? Popping an edible and making market
| maps"
|
| Here is an example output:
| https://meticulate.ai/workflow/3b3fe891f16fc437acca87c0
|
| It was really nice to go away for a few minutes and come back to
| this. Output is not perfect, but I wouldn't expect it to be at
| this stage.
|
| I assume slide deck output is on the way?
| JPalakapilly wrote:
| Hahaha that reminds me of Erlich tripping in the desert trying
| to come up with one-liners for Pied Piper. Yeah definitely know
| our results are not perfect. We're going for being as
| exhaustive as possible right now so results can be noisy. And
| yes, slide generation is on the very near-term (next few days)
| roadmap :)
| rgbrgb wrote:
| This is so cool, congrats on the launch!
|
| I have a friend who looked into doing something similar but they
| couldn't figure out a way to get the cost low enough. This was
| like a year ago so I'd guess it's much cheaper now and you could
| do something like fine-tuning a smaller domain specific model on
| GPT-4 outputs.
|
| Any ballparks on pricing / cost? What models are ya'll using?
| wilburli wrote:
| Thanks! That's fascinating, do you remember what kind of tasks
| they served?
|
| Using GPT-4 and GPT-3.5 currently, and costs can be $1.50+ per
| request right now (have been benefiting from YC cloud
| credits!). Definitely steep at the moment but we expect costs
| to come down at least 10x over time.
|
| Not super clear on pricing yet (only a few weeks post-launch)
| rgbrgb wrote:
| I think it was similar... an agent finds a bunch of info on a
| private/public company to evaluate investment (automating
| some associate analyst work). TBH I'm not sure where they
| ended up but I know they had an interesting distribution
| channel lined up. Happy to connect if you want.
|
| > $1.50
|
| Thanks for sharing! I think they were ~10x that but hadn't
| done a ton of optimization yet. To me, having a swag at cost
| makes these tool demos a lot more interesting because you can
| start figuring out what types of businesses you can/can't
| build with them.
| sidcool wrote:
| Looks very promising. Will try
| wilburli wrote:
| thanks! let us know what you think
| mfrye0 wrote:
| Congrats on the launch. This looks awesome.
|
| I'm actually working with a number of companies who are exploring
| this space. Many of them are in the current YC batch. We're
| helping to provide the core business data, then we're exploring
| how we can leverage our scraping infrastructure in other ways to
| bring costs down.
|
| I'm open to chat if you're interested: michael (a t)
| bigpicture.io
| wilburli wrote:
| thanks! great to know, just emailed you!
| swatcoder wrote:
| > delivering results 50x faster at 50x less cost
|
| What about _quality_ of results? Are you measuring that too? Did
| you do so for the traditional reference practice? Using what sort
| of methodology? How did it your technique compare in quality?
| What kind of errors was it most likely to make? What techniques
| have you devised for spotting those errors? Are they the same
| kind of errors that users would experience when outsourcing? Are
| the errors easier or harder to spot for one than the other? Are
| they faster to remediate with one?
|
| I see a clever _concept_ but given the state of LLM 's and the
| nature of how they work, I don't know that nominal cost and speed
| differences are really enough to sell on. Not for something
| "crucial to big business decisions." I'd want to know that my
| failure/miss rate is no worse than when outsourcing and that my
| _net_ cost and time (including error identification and recovery)
| still end up ahead. I don 't see either of those vital issues
| touched upon here.
| JPalakapilly wrote:
| This is a great point. We completely agree that high-quality
| results is essential for adoption. It's basically table stakes
| for any tool like this to be useful. We've had several versions
| of this tool that weren't quite "good enough" and never saw any
| real use. Our latest version seems to meet the first quality
| threshold for actual work use.
|
| Our method of evaluating quality is not super systematic right
| now. For this competitive landscape task, we have a "test
| suite" of ~10 companies and for each we have a sort of "must-
| include", "should-include", "could-include" set of competitors
| that should be surfaced. We run these through our tool and
| others and look at precision and recall on the competitor sets.
|
| In terms of errors, right now our results are a little noisy,
| since we're biased towards being exhaustive vs selective. There
| are obviously irrelevant companies in the results that no human
| would have ever included. Our users can fairly easily filter
| these out by reading the one sentence overviews of the
| companies but it's still not a great UX. Actively working on
| this.
| andy99 wrote:
| I wonder if it's more about convincing yourself that it
| faithfully follows the same workflow an analyst would follow.
| It's always possible to miss stuff, so the best a person or a
| machine can do is be demonstrably methodical, it sounds
| like... and that is easier to test. Unless there is really
| some magic tacit step that human analyst perform to get
| better answers.
| madiator wrote:
| I am sure most people never asked these questions to a human
| doing this research.
| spaceman_2020 wrote:
| Most B2B focused AI tools will do 10x better if they pretend
| to be a normal human run company but just have the AI at the
| back.
|
| Their clients want to know that the research report was
| written by a real person and not a bot.
|
| But that doesn't mean it _actually_ has to be written by a
| real person and not a bot.
| internet101010 wrote:
| Yeah when it comes to anything involving numbers there is
| absolutely no room for hallucination. It must be 100% accurate
| 100% of the time. No exceptions.
| babyshake wrote:
| Do you plan to offer an API as an alternative to the UI linked?
| chintler wrote:
| Congrats on the launch!
|
| If I have want to help you in your roadmap- specifically around
| Find Companies, how can I contribute?
___________________________________________________________________
(page generated 2024-03-14 23:00 UTC)