[HN Gopher] Launch HN: Undermind (YC S24) - AI agent for discove...
___________________________________________________________________
Launch HN: Undermind (YC S24) - AI agent for discovering scientific
papers
Hey HN! We're Josh and Tom from Undermind
(https://www.undermind.ai/). We're building a search engine for
complex scientific research. There's a demo video at
https://www.loom.com/share/10067c49e4424b949a4b8c9fd8f3b12c?..., as
well as example search results on our homepage. We're both
physicists, and one of our biggest frustrations during grad school
was finding research -- There were a lot of times when we had to
sit down to scope out new ideas for a project and quickly become a
deep expert, or we had to find solutions to really complex
technical problems, but the only way to do that was manually dig
through papers on Google Scholar for hours. It was very tedious, to
the point where we would often just skip the careful research and
hope for the best. Sometimes you'd get burned a few months later
because someone already solved the problem you thought was novel
and important, or you'd waste your time inventing/building a
solution for something when one already existed. The problem was
there's just no easy way to figure out what others have done in
research, and load it into your brain. It's one of the biggest
bottlenecks for doing truly good, important research. We wanted to
fix that. LLMs clearly help, but are mostly limited to general
knowledge. Instead, we needed something that would pull in research
papers, and give you exactly what you need to know, even for very
complex ideas and topics. We realized the way to do this is to
mimic the research strategies we already know work, because we do
them ourselves, and so we built an agent-like LLM pipeline to
carefully search in a way that mimics human research strategies.
Our search system works a bit differently from casual search
engines. First, we have you chat back and forth with an LLM to make
sure we actually understand your really complex research goals up
front, like you're talking to a colleague. Then the system
carefully searches for you for ~3 minutes. At a high level, it does
something similar to tree search, following citation rabbit holes
and adapting based on what it discovers to look for more content
over multiple iterations (the same way you would if you decided to
spend a few hours). The 3 minute delay is annoying, but we're
optimizing for quality of results rather than latency right now. At
the end there's a report. We're trying to achieve two things with
this careful, systematic agent-like discovery process: 1. We want
to be very accurate, and only recommend very specific results if
you ask for a specific topic. To do this, we carefully read and
evaluate content from papers with the highest quality LLMs (we're
just reading abstracts and citations for now, because they're more
widely accessible - but also working on adding full texts). 2. We
want to find everything relevant to your search, because in
research it's crucial to know if something exists or not. The key
to being exhaustive is the adaptive algorithms we've developed
(following citations, changing strategy based on what we find,
etc). However, one cool feature of the automated pipeline is we can
track the discovery process as the search proceeds. Early on, we
find many good results, and later on they get more sparse, until
all the good leads are exhausted and we stop finding anything
helpful. We can statistically model that process, and figure out
when we've found everything (it actually has an interesting
exponential saturation behavior, which you can read a bit more
about in our whitepaper
(https://www.undermind.ai/static/Undermind_whitepaper.pdf), which
we wrote for a previous prototype.) You can try searching yourself
here: https://www.undermind.ai/query_app/promotion/. This is a
special HN link where, for today, we've dropped the signup gate for
your first few searches. Usually we require login so you can save
searches. We're excited to share this with you! We'd love to hear
about your experiences searching, what's clear or not, and any
feedback. We'll be here to answer any questions or comments.
Author : jramette
Score : 106 points
Date : 2024-07-25 15:36 UTC (7 hours ago)
| phren0logy wrote:
| I have only tried one search, but so far it's impressive. I have
| been using elicit.com, but they seem to be taking a different
| approach that is less AI-heavy. I would definitely give this a
| shot for a few months.
| tom_hartke wrote:
| We're trying to bias the system toward more autonomous
| execution, rather than a "copilot"-like experience where you
| iterate back and forth with the system. That lets us run more
| useful subroutines in parallel in the backend, as long as you
| specified your complex goal clearly.
| KrisGaudel wrote:
| This is really cool, excited to see where this goes!
| pointlessone wrote:
| OK, I'm both impressed and disappointed.
|
| I did 2 searches.
|
| First I asked about a very specific niche thing. I gave me
| results but none I wanted. It looked like I missed a crucial
| piece of information.
|
| So I did the second search. I started with the final request it
| written for the previous search and added the information I
| though I missed. It gave me virtually the same results with a
| little sprinkle of what I was actually after.
|
| A few observations:
|
| 1. I'm not sure but it seems like it relies too much on citation
| count. Or maybe citations in papers make it think that the paper
| is absolutely a must read. I specifically said I'm not interested
| in what's in that paper and I still got those results.
|
| 2. I don't see much dissertations/theses in the result. I know
| for sure that there a good results for my request in a few
| dissertations. None of them are in the results.
|
| That said, while I didn't get exactly what I want I've found a
| few interesting papers even if they're tangential to the actual
| request.
| tom_hartke wrote:
| A few possibilities: - We only use abstracts for now. Have to
| make sure you ask for something present there. - Did you ask
| for a scientific topic? (Sometimes people ask for papers by a
| specific author, journal, etc. The system isn't engineered to
| efficiently find that).
|
| Regarding citations: we use them, but only for figuring out
| which papers to look at next in the iterative discovery
| process, not for choosing what to rank higher or lower at the
| end (unless you explicitly ask for citations). It's ranking
| based on topic match.
|
| If you're comfortable, posting the report URLs here can let us
| debug.
| glitchc wrote:
| This is a nice search rngine. I found it to be more effective
| than crawling with google scholar. Good work guys!
| redeux wrote:
| I've been using a similar platform that I really like called
| Answer This[1]. I'll have to check out yours as well and see how
| it compares.
|
| 1. https://answerthis.io/
| tom_hartke wrote:
| I ran these two example searches we have on our homepage on
| AnswerThis: (3D ion shuttling)
| https://undermind.ai/query_app/display_one_search/b3767fb7b6...
| (laser cooling to BEC)
| https://undermind.ai/query_app/display_one_search/c5f77f862a...
|
| The results from their website aren't sharable, but their lists
| of references do not seem relevant (ie. they miss the fact that
| shuttling needs to be in 3D, and the list of experiments for
| laser cooling to BEC is missing all of the relevant papers).
|
| I think, like other research tools, they're more focused on the
| summarization/extraction of information, rather than the
| discovery process (though they are similar to us in the way
| they say they do multi-stage retrieval and it takes some time).
| brainwipe wrote:
| Independent researcher without academic address; can't get in.
| Best of luck.
| tom_hartke wrote:
| You should be able to try it here without loggin in:
| https://www.undermind.ai/query_app/promotion/ (set up for HN
| today). If not message support@undermind.ai and I'll set you
| up.
| toisanji wrote:
| can you fix it so anyone can get it, that sounds like a waste
| of time to block people.
| bravura wrote:
| Same. This is me:
| https://scholar.google.com/citations?user=eQ1uJ6UAAAAJ&hl=es
|
| And post.harvard.edu has been sunsetted for alums, so I don't
| have that email either.
| setgree wrote:
| Very cool, and very relevant to my life -- I am currently writing
| a meta-analysis and finishing my literature search.
|
| I gave it a version of my question, it asked me reasonable
| follow-ups, and we refined the search to:
|
| > I want to find randomized controlled trials published by
| December 2023, investigating interventions to reduce consumption
| of meat and animal products with control groups receiving no
| treatment, measuring direct consumption (self-reported outcomes
| are acceptable), with at least 25 subjects in treatment and
| control groups (or at least 10 clusters for cluster-assigned
| studies), and with outcomes measured at least one day after
| treatment begins.
|
| I just got the results back:
| https://www.undermind.ai/query_app/display_one_search/e5d964....
|
| It certainly didn't find everything in my dataset, but:
|
| * the first result is in the dataset.
|
| * The second one is a study I excluded for something buried deep
| in the text.
|
| * The third is in our dataset.
|
| * The fourth is excluded for something the machine should have
| caught (32 subjects in total), but perhaps I needed to clarify 25
| subjects in treatment and control _each_.
|
| * The fifth result is a protocol for the study in result 3, so a
| more sophisticated search would have identified that these were
| related.
|
| * The sixth study was entirely new to me, and though it didn't
| qualify because of the way the control group received some aspect
| of treatment, it's still something that my existing search
| processes missed, so right away I see real value.
|
| So, overall, I am impressed, and I can easily imagine my lab
| paying for this. It would have to advance substantially before it
| was my _only_ search method for a meta-analysis -- it seems to
| have missed a lot of the gray literature, particularly those
| studies published on animal advocacy websites -- but that 's a
| much higher bar than I need for it to be part of my research
| toolkit.
| tom_hartke wrote:
| For a meta-analysis, you might want to try the "extend"
| feature. It sends the agent to gather more papers (we only
| analyze 100 carefully initially), so if your report might say
| "only 55% discovered", could be useful.
|
| (Also, if you want, you can share your report URL here, others
| will be able to take a look.)
| setgree wrote:
| Thanks, I added my URL
| kelloggm wrote:
| I'm a CS academic who _should_ be working on finalizing a new
| submission, so when I saw this on HN I decided to give it a try
| and see if it could find anything in the literature that I'd
| missed. Somewhat to my surprise, it did - the top 10 results
| contained two items that I really ought to have found myself
| (they're from my own community!), but that I'd missed. There were
| also some irrelevant results mixed in (and lots of things I was
| already aware of), but overall I'm very impressed with this and
| will try it out again in the future. Nice work :)
| smcsdp wrote:
| Any idea how i can use your tool for a vs code extension
| gillesjacobs wrote:
| Pretty good, it found some useful references I missed in Google
| Scholar and Arxiv. Looks promising, will use it more.
| Geee wrote:
| I've been using https://exa.ai for this. It doesn't do any
| advanced agent stuff like here, but it's way better than Google,
| especially if you're not quite sure what you're looking for.
| tom_hartke wrote:
| Agreed, exa is great - particularly, it's the best thing I've
| found for fast web retrieval of slightly more complex topics
| than Perplexity, Google, etc can handle.
| minznerjosh wrote:
| Are you planning to offer a search API at some point?
| tom_hartke wrote:
| Potentially. Given the latency and the cost/compute we put into
| each result, it doesn't fit the usual API mechanics.
|
| What use case are you thinking of?
| rjchint wrote:
| How would you compare your product to elicit.ai?
|
| In my opinion elicit has better looking UI and much more features
| and further along
| tom_hartke wrote:
| I think the biggest difference is our focus on search quality,
| and being willing to spend a lot on compute to do it, while
| they focus on systematic extraction of data from existing
| sources and on being fast. It's a bit of an oversimplification
| (they of course have search, and we also have extraction).
|
| Feature-wise, we definitely have a lot of work to do :) What
| crucial pieces do you think we're missing?
| sitkack wrote:
| How is this different than the work that semantic scholar is
| doing around AI?
| tom_hartke wrote:
| Semantic Scholar seems more focused on 1. being the data
| provider/aggregator for the research community, and 2. long
| term, I think they plan to develop software at the reading
| interface that learns as a researcher uses it to browse papers
| (a rich PDF reader, with hyperlinks, TLDRs, citation contexts,
| and a way to track your interactions over time, and remind you
| of what you've seen or not).
|
| Their core feature now is a fast keyword search engine, but
| they also have a few advanced search features through their API
| (https://api.semanticscholar.org/api-docs/) like
| recommendations from positive/negative examples, but neither KW
| search nor these other systems are currently high enough
| quality to be very useful for us.
|
| FYI our core dataset for now is provided by Semantic Scholar,
| so hugely thankful for their data aggregation pipeline and open
| access/API.
| sitkack wrote:
| Do you plan on adding an API? I already have an inhouse
| knowledge discovery, annotation and search system that could
| be augmented by your service. Not super critical at this
| point, but a would be nice.
|
| And yes, Semantic Scholar is a wonderful part of the academic
| commons. Fingers crossed they don't go down the jstor/oclc
| path.
| shashkingregory wrote:
| I've used undermind for literature search and it was very
| precise! Thanks for the product! I wonder how you plan to
| extend the search to full paper content (will Semantic
| Scholar api allow this) - and do you plan to connect more
| datasets (which ones)? (many of them are paid...)
| jramette wrote:
| We'll certainly be able to include open access full texts,
| which is already a substantial fraction of the published
| papers, and a growing fraction too, as the publishing
| industry is rapidly moving toward open access. Paywalled
| full text search would require working with the publishers,
| which is more involved.
| robwwilliams wrote:
| Awesome! I just took you up on your offer and compared roughly
| similar questions using Claude 3.5 Sonnet and Undermind.
|
| Claude 3.5 is reluctant to provide references---although it will
| if coaxed by prompting.
|
| Undermind solves this particular problem. A great complement for
| my research question --- the evidence that brain volume is
| reduced as a function of age in healthy cognitively normal
| humans. In mice we see a steady slow increase that averages out
| to a gain of 5% between the human equivalents of 20 to 65 years
| of age. This increase is almost linear as a function of log of
| age.
|
| Here is the question that was refined with Undermind's help:
|
| >I want to find studies on adult humans (ages 20-100+) that have
| used true longitudinal repeated measures designs to study
| variations in brain volume over several years, focusing on
| individuals who are relatively healthy and cognitively
| functional.
|
| I received 100 ranked and lightly annotated set of 100 citations
| in this format:
|
| >[1] Characterization of Brain Volume Changes in Aging
| Individuals With Normal Cognition Using Serial Magnetic Resonance
| Imaging S. Fujita, ..., and O. Abe JAMA Network Open 2023 - 21
| citations - Show abstract - Cite - PDF 99.6% topic match Provides
| longitudinal data on brain volume changes in aging individuals
| with normal cognition. Analyzes annual MRI data from 653 adults
| over 10 years to observe brain volume trajectories. Excludes
| populations with neurodegenerative diseases; employs true
| longitudinal design with robust MRI techniques.
| tom_hartke wrote:
| It's worth highlighting that first result is _exactly_ what you
| asked for, given all 4 of your criteria:
|
| 1. It's on adults.
|
| 2. It's longitudinal over multiple years.
|
| 3. It studies variations in brain volume.
|
| 4. It focuses on healthy individuals.
|
| You can see the full results for that search text here:
| https://undermind.ai/query_app/display_one_search/e1a3805d35...
| sam1234apter wrote:
| How is this different from Scite, Elicit, Consensus, and Scopus
| AI for Generating Literature Reviews
| tom_hartke wrote:
| Ours is slow, but accurate, even for complex topics. The rest
| are fast, but generally can't handle complex topics. (There's
| more nuanced explanations in other comments)
| timdellinger wrote:
| I'll write the obligatory comment about doing literature searches
| in the 90s, which involved trudging to the physics library, the
| chemistry library, and the engineering library in search of dead
| tree copies of the journal articles you're after. Also: skimming
| each paper quickly after you photocopy it, to see if it
| references any other papers you should grab while you're at the
| library.
| smcsdp wrote:
| what are your biggest drawbacks?
| tom_hartke wrote:
| Latency, compute required, and lack of full texts (paywalled
| publisher content).
| iamacyborg wrote:
| I'm a marketer rather than a scientist but this proved very
| useful in helping me find research that's applicable to my field
| of work (crm marketing). Nothing particularly new was surfaced
| but I suppose I wasn't expecting it to either
| http://www.undermind.ai/query_app/display_one_search/7140cc6...
| winddude wrote:
| Curious as to what it's doing under the hood, the query to return
| the results takes an excruciatingly long time... are you
| searching remote sources vs a local index?
|
| this was the search
| <https://www.undermind.ai/query_app/display_one_search/cba773...>
| if you need a reference too it, ie bugs or performance
| monitoring...
| tom_hartke wrote:
| The few minute time delay is primarily because of the
| sequential LLM processing steps by high quality LLMs, not
| database access times. The system reads and generates
| paragraphs about papers, then compares them, and we have to use
| the highest quality LLMs, so token generation times are
| perceptible. We repeat many times for accuracy. We find it's
| impossible to be accurate without GPT-4 level models and the
| delay.
| mangoparrot wrote:
| would this be able to find the latest articles on a given topic?
|
| let's say i am interested in coffee and i'd like to get new
| research papers on it. would this work?
| tom_hartke wrote:
| In short, yes, though it's geared toward topic search.
|
| From a strategy perspective, we designed it for topic search
| because it makes more sense to find _everything_ on a topic
| first, then filter for the most recent, if recent is what you
| want. That 's because there is a lot of useful information in
| older articles (citation connections, what people discuss, and
| how), and gathering all that helps uncover the most relevant
| results. Conversely if you only ever filtered on articles in
| the last year, you might discover a few things, but you
| wouldn't have as much information to adapt to help the search
| work better.
|
| So, you can ask for articles on coffee (though ideally it
| should be something a bit more specific, or there will be
| thousands of results). Our system will carefully find all
| articles, then you can filter for 2024 articles or look at the
| timeline.
| viraj_shah wrote:
| Here is an open source tool for summarizing Arxiv papers:
| https://summarizepaper.com/
| viraj_shah wrote:
| Also very curious to see how this compares to:
| https://www.semanticscholar.org/
| sirlunchalot wrote:
| Very happy subscriber here, thank you for the tool. I do a lot of
| searching with it, however due to some changes in my life in the
| near future I will not need it as much so I wont be willing to
| spend $20 a month on it. So my question is, would you consider
| adding an option where one could pay per query rather than just
| per monthly subscription? I would love to use it for the
| occasional spark of curiosity when I want to know more about a
| certain topic without having to familiarise myself with the
| academic field surrounding it. Having a way for using undermind
| for situations like that would be truly amazing! Would gladly pay
| 1-2 or maybe even 3 dollars per extended query.
| jramette wrote:
| We've thought quite a bit about usage-based pricing, but found
| that doesn't work psychologically for most people. Generally,
| people seem happier by paying up front for access, then feel
| good about having the system available whenever they need it,
| rather then having to think through cost tradeoffs every time
| they want to do a search or use up credits. Please do reach out
| at support@undermind.ai though, we'd love to talk about a
| solution for you and get your feedback.
| benzguo wrote:
| This is really cool! Both of my parents are cell biologists, and
| I've done some time in labs as well, so a lot of paper exploring
| and reading in the family. "Review" articles are a good index but
| something more on-demand makes a lot of sense, I can definitely
| see this being extremely useful.
___________________________________________________________________
(page generated 2024-07-25 23:01 UTC)