[HN Gopher] Launch HN: Meticulate (YC W24) - LLM pipelines for b...
___________________________________________________________________
Launch HN: Meticulate (YC W24) - LLM pipelines for business
research
Hi HN, Meticulate gives finance professionals easy access to world
class business research--think competitive landscapes, market
sizings, customer segmentation etc. Here's a video of a competitive
landscape generation in action (https://youtu.be/aJ-slHcp32c)and
we've taken down the signup wall today so you can try it directly
at https://meticulate.ai/. Some background on "business research":
investment and consulting teams sink many hours a week into
researching companies, markets, and products. This work is time-
sensitive and exhausting, but crucial to big decisions like company
acquisitions or pricing model changes. At large financial services
firms, much of this work is offshored to external providers, who
charge thousands of dollars per project and are often slow and low-
quality. Small teams lack the budget and consistent flow of work to
employ these resources. We're building an automation solution that
brings a fast, easily accessible, and defendable research resource.
Meticulate uses LLMs to emulate analyst research processes. For
example, to manually build a competitive landscape like this one:
https://meticulate.ai/workflow/65dbfeec44da6238abaaa059, an analyst
needs to spend ~2 hours digging through company websites, forums,
and market reports. Meticulate replicates this same process of
discovering, researching, and mapping companies using ~1500 LLM
calls and ~500 webpages and database pulls, delivering results 50x
faster at 50x less cost. At each step, we use an LLM as an agent
to run searches, select and summarize articles, devise frameworks
of analysis, and make small decisions like ranking and sorting
companies. Compared to approaches where an LLM is being used
directly to answer questions, this lets us deliver results that (a)
come from real time searches and (b) are traceable back to the
original sources. We've released two workflows: building
competitive landscapes and market maps. We designed it with an
investor running diligence on a company as the target use case but
we're seeing lots of other use cases that we didn't originally have
in mind--things like founders looking for alternative vendors for a
product they're purchasing; sales reps searching for more prospects
like one they've already sold to; consultants trying to understand
a new market they are unfamiliar with, and more. The main
challenges we've been overcoming are preventing quality degradation
along multi-step LLM pipelines where an error on one step can
propagate widely, and dealing with a wide range of data quality.
We're working hard on our next set of workflows and would love for
you to give it a try at https://meticulate.ai and would appreciate
feedback at any level!
Author : JPalakapilly
Score : 95 points
Date : 2024-03-14 16:51 UTC (1 days ago)
| cubecul wrote:
| Congrats on the Launch HN!
|
| I've heard a few investor types say something like "You know
| what's surprisingly fun? Popping an edible and making market
| maps"
|
| Here is an example output:
| https://meticulate.ai/workflow/3b3fe891f16fc437acca87c0
|
| It was really nice to go away for a few minutes and come back to
| this. Output is not perfect, but I wouldn't expect it to be at
| this stage.
|
| I assume slide deck output is on the way?
| JPalakapilly wrote:
| Hahaha that reminds me of Erlich tripping in the desert trying
| to come up with one-liners for Pied Piper. Yeah definitely know
| our results are not perfect. We're going for being as
| exhaustive as possible right now so results can be noisy. And
| yes, slide generation is on the very near-term (next few days)
| roadmap :)
| rgbrgb wrote:
| This is so cool, congrats on the launch!
|
| I have a friend who looked into doing something similar but they
| couldn't figure out a way to get the cost low enough. This was
| like a year ago so I'd guess it's much cheaper now and you could
| do something like fine-tuning a smaller domain specific model on
| GPT-4 outputs.
|
| Any ballparks on pricing / cost? What models are ya'll using?
| wilburli wrote:
| Thanks! That's fascinating, do you remember what kind of tasks
| they served?
|
| Using GPT-4 and GPT-3.5 currently, and costs can be $1.50+ per
| request right now (have been benefiting from YC cloud
| credits!). Definitely steep at the moment but we expect costs
| to come down at least 10x over time.
|
| Not super clear on pricing yet (only a few weeks post-launch)
| rgbrgb wrote:
| I think it was similar... an agent finds a bunch of info on a
| private/public company to evaluate investment (automating
| some associate analyst work). TBH I'm not sure where they
| ended up but I know they had an interesting distribution
| channel lined up. Happy to connect if you want.
|
| > $1.50
|
| Thanks for sharing! I think they were ~10x that but hadn't
| done a ton of optimization yet. To me, having a swag at cost
| makes these tool demos a lot more interesting because you can
| start figuring out what types of businesses you can/can't
| build with them.
| wilburli wrote:
| would love the connect to learn more about what they tried,
| I'm at wilbur at meticulate.ai
|
| Definitely... we're lucky to be in an industry where
| there's a lot of money at stake
| sidcool wrote:
| Looks very promising. Will try
| wilburli wrote:
| thanks! let us know what you think
| sidcool wrote:
| I like it so far. The report's readability can be refined a
| bit. It's too noisy.
| wilburli wrote:
| definitely, appreciate the feedback!
| mfrye0 wrote:
| Congrats on the launch. This looks awesome.
|
| I'm actually working with a number of companies who are exploring
| this space. Many of them are in the current YC batch. We're
| helping to provide the core business data, then we're exploring
| how we can leverage our scraping infrastructure in other ways to
| bring costs down.
|
| I'm open to chat if you're interested: michael (a t)
| bigpicture.io
| wilburli wrote:
| thanks! great to know, just emailed you!
| swatcoder wrote:
| > delivering results 50x faster at 50x less cost
|
| What about _quality_ of results? Are you measuring that too? Did
| you do so for the traditional reference practice? Using what sort
| of methodology? How did it your technique compare in quality?
| What kind of errors was it most likely to make? What techniques
| have you devised for spotting those errors? Are they the same
| kind of errors that users would experience when outsourcing? Are
| the errors easier or harder to spot for one than the other? Are
| they faster to remediate with one?
|
| I see a clever _concept_ but given the state of LLM 's and the
| nature of how they work, I don't know that nominal cost and speed
| differences are really enough to sell on. Not for something
| "crucial to big business decisions." I'd want to know that my
| failure/miss rate is no worse than when outsourcing and that my
| _net_ cost and time (including error identification and recovery)
| still end up ahead. I don 't see either of those vital issues
| touched upon here.
| JPalakapilly wrote:
| This is a great point. We completely agree that high-quality
| results is essential for adoption. It's basically table stakes
| for any tool like this to be useful. We've had several versions
| of this tool that weren't quite "good enough" and never saw any
| real use. Our latest version seems to meet the first quality
| threshold for actual work use.
|
| Our method of evaluating quality is not super systematic right
| now. For this competitive landscape task, we have a "test
| suite" of ~10 companies and for each we have a sort of "must-
| include", "should-include", "could-include" set of competitors
| that should be surfaced. We run these through our tool and
| others and look at precision and recall on the competitor sets.
|
| In terms of errors, right now our results are a little noisy,
| since we're biased towards being exhaustive vs selective. There
| are obviously irrelevant companies in the results that no human
| would have ever included. Our users can fairly easily filter
| these out by reading the one sentence overviews of the
| companies but it's still not a great UX. Actively working on
| this.
| andy99 wrote:
| I wonder if it's more about convincing yourself that it
| faithfully follows the same workflow an analyst would follow.
| It's always possible to miss stuff, so the best a person or a
| machine can do is be demonstrably methodical, it sounds
| like... and that is easier to test. Unless there is really
| some magic tacit step that human analyst perform to get
| better answers.
| sroussey wrote:
| Well, human analysts get on the phone and ask people
| questions.
|
| Not that an AI can't do that too!
|
| Though I may hang up...
| madiator wrote:
| I am sure most people never asked these questions to a human
| doing this research.
| spaceman_2020 wrote:
| Most B2B focused AI tools will do 10x better if they pretend
| to be a normal human run company but just have the AI at the
| back.
|
| Their clients want to know that the research report was
| written by a real person and not a bot.
|
| But that doesn't mean it _actually_ has to be written by a
| real person and not a bot.
| internet101010 wrote:
| Yeah when it comes to anything involving numbers there is
| absolutely no room for hallucination. It must be 100% accurate
| 100% of the time. No exceptions.
| alexdoesstuff wrote:
| Have you ever seen the research coming out of some of the
| outsourcing shops that the OP discusses in the post? They are
| hard not living up to that standard. It's important to
| realize that this is input for the analyst at a fund or
| investment bank to do some more digging on the companies and
| in the process potentially discover more. This isn't going
| straight to the CEO to form the basis of an investment
| decision.
| killerstorm wrote:
| As somebody who have seen a lot of market research coming from
| "real humans"(TM) I'm pretty sure it's actually hard to make it
| worse. Maybe there are boutiques which make quality research,
| but average is abysmal.
|
| I've been in a meeting with a senior executive, PhD, who argued
| that Total Addressable Market for a supply chain tracking
| software (basically just a database) is same as the market for
| the goods it tracks.
|
| It is as if a startup which prints restaurant menus would claim
| the value of all food sold in restaurants as their TAM.
|
| And this person was otherwise reasonable...
|
| So, yeah, never underestimate the power of Natural Stupidity.
|
| Perhaps
| mikejulietbravo wrote:
| This was a huge problem when I was working on a similar product
| some years ago. The issue is that false positives or worse,
| false negatives, can be pretty catastrophic when you're
| thinking about valuing companies in particular. What's hard for
| LLMs (currently) is to decide what is most important or to
| infer the things that aren't explicitly stated.
| babyshake wrote:
| Do you plan to offer an API as an alternative to the UI linked?
| JPalakapilly wrote:
| not in the near term but happy to talk about what your use case
| might be if you want to shoot me an email at
| joseph(at)meticulate(dot)ai
| chintler wrote:
| Congrats on the launch!
|
| If I have want to help you in your roadmap- specifically around
| Find Companies, how can I contribute?
| JPalakapilly wrote:
| Thanks! shoot me an email at joseph(at)meticulate(dot)ai
| fuzzythinker wrote:
| It couldn't recover from the task after the computer sleeps.
| barfbagginus wrote:
| I once had a business owner interview me and conclude that he
| wanted to hire me because he felt that I was a good thinker,
| someone who was, "real meticulate".
|
| I told the boss that actually that is spelled meticulous. They
| said that he was pretty sure that it was spelled meticulate. I
| got the job offer, but I refuse the job. The thought of working
| as and inferior for somebody who would sucker down on the
| spelling "meticulate" was just way too much for me. What other
| harebrained bull** would they have me commit to?
|
| Not sure why I bring that up here. It's a personal experience
| that makes me have no interest in this product whatsoever. Which
| is not rational. I guess I'm just a hater and I wanted to share
| this funny story while also maybe causing a loss of revenue to a
| random company I have nothing to do with.
|
| Cheers!
| VoodooJuJu wrote:
| It's ironic because you were the one harebraining over trivial
| bullshit.
|
| We have millions of customers to serve and billions of dollars
| at stake, but you're going to nitpick over this guy's
| colloquial word misuse?
|
| Come on.
| Shindi wrote:
| I'm pretty sure it's actually hair brained
| ProjectArcturis wrote:
| No, it's hare as in rabbit.
| spencerchubb wrote:
| Your attention to spelling further shows your meticulousness
| mikejulietbravo wrote:
| This sounds a lot like what Grata does, although they've built a
| pretty impressive proprietary dataset
| novariation wrote:
| I tried it quickly on my company (some IT consulting
| multinational) to find competitors, and I was wondering if there
| was an easy way to do the following:
|
| - for each company, find their main product lines and their
| respective market share (i.e. my firm does multiple things such
| as architecture, data governance, data engineering, data
| analysis, etc) but we're not necessarily the best on every type
| of service. I'm assuming that's a common pattern (e.g. some
| automotive companies are great at trucks but not the best at
| smaller cars, etc). - get basic financial stats (revenue, market
| share) - (bonus) get time-dependent stats (what was their revenue
| like 3 years ago ? What's the YoY growth ?)
___________________________________________________________________
(page generated 2024-03-15 23:03 UTC)