[HN Gopher] Show HN: Research Hacker News, ArXiv & Google with H...
___________________________________________________________________
Show HN: Research Hacker News, ArXiv & Google with Hierarchical
Bayesian Models
Hi Hacker News! I'm a Bayesian statistician that has been working
on applying hierarchical mixture models (originally developed for
genomics) to structure text data, and in the process, used these
models to build (what started as a personal) tool for conducting
literature reviews and deep research. My literature review process
starts with a broad search to find a few key papers/groups, and
from there expands along their citation networks. I needed to
conduct a few rounds of literature reviews during the course of my
research and decided to build a tool to facilitate this process.
The tool started as an experimental wrapper over low-level
statistical software in C, quickly became a testing/iteration
ground for our api, and is now my personal go-to for lit reviews.
The tool organizes corpuses of text content, visualizes the high
level themes, and enables me to pull up relevant excerpts. Unlike
LLMs, this model transparently organizes the data and can train
from scratch quickly on small datasets to learn custom hierarchical
taxonomies. My favorite part of the tool is the citation network
integration: any research paper it pulls up has a button "Citation
Network Deep Dive" that pulls every paper that cites or is cited by
the original paper, and organizes it for further exploration. I
initially built this tool for academic research, but ended up
extending it to support Hacker News to mine technical conversation,
the top 200 Google results, and earnings transcripts. We have a
gallery of ready to explore results on the homepage. If you are
kicking off a custom deep dive, it takes about 1-5 minutes for
academic search, 3-7 minutes for Hacker News, and 5-10 minutes for
Google. To demonstrate the process, I put together a video
walkthrough of a short literature review I conducted on AI
hallucinations: https://www.youtube.com/watch?v=OUmDPAcK6Ns I host
this tool on my company's website, free for personal use. I'd love
to know if the HN community finds it useful (or to hear what
breaks)!
Author : kianN
Score : 58 points
Date : 2025-10-28 15:49 UTC (1 days ago)
(HTM) web link (sturdystatistics.com)
(TXT) w3m dump (sturdystatistics.com)
| kianN wrote:
| Some statistical notes for those interested:
|
| Under the hood, this model resembles LDA, but replaces its
| Dirichlet priors with Pitman-Yor Processes (PYPs), which better
| capture the power-law behavior of word distributions. It also
| supports arbitrary hierarchical priors, allowing metadata-aware
| modeling.
|
| For example, in an earnings-transcript corpus, a typical LDA
| might have a flat structure: Prior - Document
|
| Our model instead uses a hierarchical graph: Uniform Prior -
| Global Topics - Ticker - Quarter - Paragraph
|
| This hierarchical structure, combined with the PYP statistics,
| consistently yields more coherent and fine-grained topic
| structures than standard LDA does. There's also a "fast mode"
| that collapses some hierarchy levels for quicker runs; it's a
| handy option if you're curious to see the impact hierarchy has on
| the model results (or in a rush).
| jcynix wrote:
| Nice and interesting. I'm still investigating so might refine
| that later ;-) Can the search result be saved somehow for later
| use?
|
| BTW:, the circular graphics of the result are really cool! How
| did you do this?
| kianN wrote:
| The URL is unique to your search and saves it's state!
|
| In the technical notes I sort of laid out our model graph on
| the document branch. We also have a topic branch that is also
| structured hierarchically: Uniform Prior - High Level Topic
| Word - Granular Topics - Document Lever Variation in Topics. We
| just directly visualize that hierarchical representation in the
| sunburst.
|
| The low level model graph is all written in C and exports
| granular annotations of the model graph. We use the model
| output to annotate the original text data. We do some work to
| store these hierarchical results in a SQL queryable format in
| DuckDB.
|
| What's cool about this process is it's all annotation based.
| You can query data at the topic level, analyze topics and sql,
| and at any point pull up the exact excerpts to which the high
| level data refers.
|
| Curious what you've been using it to search for?
| jcynix wrote:
| > Curious what you've been using it to search for?
|
| For starters I've done some trivial things, like "emacs
| elisp" on HackerNews and now "git tutorial" on
| AcademicSearch. The later is still running and organizing
| results. But the results don't have relevance for "git" as it
| seems.
|
| I'll do some searches in French and German later to see how
| it works with foreign languages (not searching on HackerNews,
| obviously ;-)
| kianN wrote:
| So this may have been something worth mentioning above, but
| the hacker news search is exact match.
| hirako2000 wrote:
| It is covered in the doc. Even the plotting code is shown.
|
| The doc also explains the UX issue of a simple sunburst graph,
| thus using a tiered sun burst graph.
| mkmccjr wrote:
| Just tried this out, and my mind is blown:
| https://platform.sturdystatistics.com/deepdive?fast=0&q=camp...
|
| I did a google search for "camping with dogs" and it organized
| the results into a set of about ~30 results which span everything
| I'd want to know on the topic: from safety and policies to
| products and travel logistics.
|
| Does this work on any type of data?
| kianN wrote:
| Awesome so glad the result were helpful! What's cool is because
| it's built on hierarchical Bayesian sampling, it is extremely
| robust to any input -- it just kinda works.
| robrenaud wrote:
| The relevance here is pretty weak.
|
| https://sturdystatistics.com/deepdive?fast=0&q=reinforcement...
|
| I think only 1/10 of the articles is really on topic.
| kianN wrote:
| I see that the model has not yet finished training: I think you
| are referring to the "Raw Search Results Section".
|
| Our tool works a little different than LLM style tools. We are
| doing a bulk search -- for academic search, ~1000 papers -- and
| then training a hierarchical Bayesian model to organize the
| results. Once the model trains, it provides a visual
| representation of the high level themes that you can then use
| to explore the results.
|
| The trade off is we are willing to lower the relevance filter
| to enable a broad set of exploration.
| kianN wrote:
| Quick update: I ran into a rate limit issue for one of my data
| sources. Apologies to anyone who has hit errors in the past 15
| minutes. I think the issue should be resolved.
| aster0id wrote:
| This could become the missing piece for RAG with LLMs for company
| data. Every query that requires a lookup can use this model and
| then an agentic LLM can crawl through the hierarchy of results to
| extract the relevant information for the user's query. I suspect
| that'll work much better than the current methods of chunking and
| storing data with metadata like title and author in a vector
| database and then performing a hybrid search
| kianN wrote:
| That's actually an application we've had a lot of success in.
| This framework allows you to really easily traverse the graph
| at a thematic level (with sql filtering if needed), then for
| any high level theme, you can pull up granular excerpts. This
| site itself is actually just a thin wrapper over our API
| (https://docs.sturdystatistics.com/).
___________________________________________________________________
(page generated 2025-10-29 23:01 UTC)