[HN Gopher] Show HN: Deep search of all ML papers
___________________________________________________________________
Show HN: Deep search of all ML papers
Built an automated system to run a deep search of ArXiv and
carefully find all the precise papers that exist on a complex
topic. It's different from simple RAG because it searches,
classifies, and adapts based on relevant papers it uncovers, and
then continues until it finds every paper on a topic (trying to
mimic the human research process). Benchmarked 10x higher accuracy
and total retrieval compared to Google Scholar for a median search
(whitepaper on website). Also knows when it is complete, and misses
virtually nothing (< 3% or so, once it's converged). Website has a
free trial and a bunch of example search reports. Want feedback and
suggestions.
Author : tomhartke
Score : 91 points
Date : 2024-01-24 14:44 UTC (8 hours ago)
(HTM) web link (app.undermind.ai)
(TXT) w3m dump (app.undermind.ai)
| tomhartke wrote:
| Here's an example report on: tokenization-free large language
| model architectures, which have been shown to achieve
| compute/accuracy tradeoffs comparable to or better than
| traditional token-based models
| https://app.undermind.ai/query_app/display_one_search/05f0b8...
| axpy906 wrote:
| What about compare to ArXiv Sanity Preserver?
| axg11 wrote:
| I would change the main CTA to "Try it now" and then use a
| different style for "Read the stats". It currently looks like
| there are two equally important CTAs.
|
| If you can find a way to make the results closer to real-time,
| this will be a really popular product.
| tomhartke wrote:
| Appreciate the advice. Re: timing, it's bottlenecked by the
| sequential nature of the search. To be comprehensive, we
| discover a few papers, and use that info to choose where to
| look more closely next.
| knicholes wrote:
| Immediately bailed once it required I provide my email address.
| huqedato wrote:
| Nice but way too pricey.
|
| https://chat.openai.com/g/g-dGz4aw9iA-research-refiner - the free
| version (just ChatGPT Plus subscription needed)
| canadiantim wrote:
| Awesome had no idea this existed. Very useful, thanks!
| tomhartke wrote:
| The goal is to be systematic and handle complex topics. ChatGPT
| + keyword search can't handle complex topics at all, and isn't
| systematic either.
| danielmarkbruce wrote:
| It's still early days. Many folks think that GPT-4 + simple
| RAG works well. It doesn't. Building a good tool like this is
| hard.
| 3abiton wrote:
| I wonder how many of such services chatgpt undercuts already?
| callalex wrote:
| Free, just needs a subscription?
| wackget wrote:
| There's something disagreeable about charging a subscription to
| search freely-available scientific papers.
|
| Yeah I get you're technically paying for the "advanced" search
| but it still leaves a bad taste in the mouth because this
| service's entire existence depends on open source knowledge.
|
| P.S. hiding pricing behind registration isn't cool
| bbsz wrote:
| I think that full text search queries over long text data is
| already kind of expensive server side. Users are paying
| specifically for this, not better UI or simple direct match
| search available in free to use projects. I would say it's very
| reasonable to charge for costs incurred here.
| kingkongjaffa wrote:
| It's a shame the research publishing industry is a bunch of
| walled gardens.
|
| Since this only supports arXiv, and not paper repositories from
| other industries.
| jakderrida wrote:
| Maybe the model will get smart enough to go to SciHub and
| Libgen? IP holders and distributors come after me with
| evidence, I'll just pull out my belt and tell them I gotta go
| teach some naughty GPUs another lesson.
| krohling wrote:
| "Our AI agent finds precisely what you ask for, 10-50x better
| than Google Scholar"
|
| I was curious how this was measured since benchmarking accuracy
| for LLMs is tough. Found this in the paper: "This classification
| accuracy was benchmarked by manually analyzing over 400 papers
| across a range of representative searches, and comparing the
| human evaluation to the language model's judgment"
|
| I'm skeptical that their dataset of 400 papers with 3
| classification labels (highly relevant, closely related, or
| ignorable) is large enough to represent the diversity of queries
| they're going to get from users. To be clear, I don't think this
| undermine's (haha) the value of what they've built, still very
| cool.
| basb77 wrote:
| How does this compare to a system like Elicit? Seems to be very
| similar at first glance.
| tomhartke wrote:
| FYI system is a bit delayed because of traffic levels. May take a
| bit longer to generate results at the moment (usually takes ~10
| min).
| frogamel wrote:
| IMO if you're going to profit off open research, you should at
| least make your own work available for other researchers. The
| white paper has 10 pages of performance benchmarks but 5
| sentences on methodology.
| espadrine wrote:
| It is certainly interesting, and I would love to try it for my
| hobbyist use-cases. I don't do much research at work, but a fair
| bit on the weekends.
|
| Are you filtering users however? I cannot sign up in a personal
| capacity with a GMail email. The page raises this error: "Please
| use a valid institutional or company email address."
| htrp wrote:
| what is pricing?
| WhitneyLand wrote:
| I don't want to be negative but you asked for feedback, so I'll
| give you a few impressions including the superficial and quite
| subjective fwiw:
|
| 1. The hyperbolic claims are going to be off-putting to some.
| You've "solved" ML search? 50x better than Google scholar on a
| metric no one's been benchmarking against? Consider your audience
| and what they would find credible.
|
| 2. The UX needs work. To give one aesthetic example, in the
| results there are large, brightly colored, red and green circles
| that are used inconsistently, and they clash the palette. This
| stuff can affect how sticky your service is.
|
| 3. Don't restrict signup by email domain. This is nuts. Never add
| friction to gaining customer relationships. If you're capacity
| constrained limit the trial. If you're trying to segment the
| market there are better ways.
|
| 4. The name "Undermind", is not working to my ear. It's worth
| changing. At least find a product person whose opinion you
| respect and ask their take.
|
| 5. I think a lot of people here would be willing to give you
| useful technical feedback on the architecture and approach if
| more information were shared about how the service works, but I
| didn't notice that was available.
___________________________________________________________________
(page generated 2024-01-24 23:01 UTC)