[HN Gopher] STORM: Get a Wikipedia-like report on your topic
___________________________________________________________________
STORM: Get a Wikipedia-like report on your topic
Author : fragmede
Score : 31 points
Date : 2024-09-16 08:12 UTC (1 days ago)
(HTM) web link (storm.genie.stanford.edu)
(TXT) w3m dump (storm.genie.stanford.edu)
| chankstein38 wrote:
| Does anyone have more info on this? They thank Azure at the top
| so I'm assuming it's a flavor of GPT? How do they prevent
| hallucinations? I am always cautious about asking an LLM for
| facts because half of the time it feels like it just adds
| whatever it wants. So I'm curious if they addressed that here or
| if this is just poorly thought-out...
| EMIRELADERO wrote:
| Here's the paper: https://arxiv.org/abs/2402.14207
| akiselev wrote:
| And the code: https://github.com/stanford-oval/storm
| morsch wrote:
| Thanks. There's an example page (markdown) at the very end.
| You can pretty easily spot some weaknesses in the generated
| text, it's uncanny valley territory. The most interesting
| thing is that the article contains numbered references, but
| unfortunately those footnotes are missing from the example.
| infecto wrote:
| If you ask an LLM what color is the sky it might say purple but
| if you give it a paragraph describing the atmosphere and then
| ask the same question it will almost always answer correctly. I
| don't think hallucinations are as big of a problem as people
| make them out to be.
| pistoriusp wrote:
| Yet remains unsolvable.
| infecto wrote:
| Huh?
| chx wrote:
| There are no hallucinations. It's just the normal bullshit
| people hang a more palatable name on. _There is nothing
| else_.
|
| https://hachyderm.io/@inthehands/112006855076082650
|
| > You might be surprised to learn that I actually think LLMs
| have the potential to be not only fun but genuinely useful.
| "Show me some bullshit that would be typical in this context"
| can be a genuinely helpful question to have answered, in code
| and in natural language -- for brainstorming, for seeing
| common conventions in an unfamiliar context, for having
| something crappy to react to.
|
| > Alas, that does not remotely resemble how people are
| pitching this technology.
| misnome wrote:
| So, it only works if you already know enough about the
| problem to not need to ask the LLM, check.
| keiferski wrote:
| No, if the _data_ you're querying contains the information
| you need, then it is mostly fine to ask for that data in a
| format amendable to your needs.
| o11c wrote:
| The problem with LLMs is _not_ a data problem. LLMs are
| stupid even on data they just generated.
|
| One recent catastrophic failure I found: Ask an LLM to
| generate 10 pieces of data. Then in a second input, ask
| it to select (say) only numbers 1, 3, and 5 from the
| list. The LLM will probably return results _numbered_ 1,
| 3, and 5, but chances are at least one of them will
| actually copy the data from a different number.
| infecto wrote:
| Are you just writing negative posts without even seeing the
| product? The system queries the internet, aggregates that
| information and writes information based on your query.
| DylanDmitri wrote:
| Seems a promising approach. Feedback at the bottom is (?) missing
| a submit button. Article was fine, but veered into overly verbose
| with redundant sections. A simplification pass, even on the
| outline, could help.
| kingkongjaffa wrote:
| It auto-saves I believe.
| thebuguy wrote:
| for the sources it used the crappy AI generated websites that pop
| up in the first page of google
| globular-toast wrote:
| What's the point of the "elaborate on purpose" box? It makes you
| fill it in but doesn't seem to affect the article, at least not
| that I can tell.
| mrpf1ster wrote:
| Probably just metadata about the request for the researchers at
| Stanford
| jedberg wrote:
| Did anyone figure out how to share the article after you generate
| it?
| philipkglass wrote:
| I think that you have to download the PDF and upload it to your
| own site.
|
| They have a "Discover" page with previously generated articles,
| but I think that they have some sort of manual review process
| to enable public access and it's not updated frequently. The
| newest articles there were from July. I tried copying the link
| for a previously generated article of mine and opening it from
| a private browser window but I just get sent to the main site.
| ssalka wrote:
| I'm guessing this just isn't implemented yet. It feels like a
| very alpha-stage project, when I sign in with another account
| and use the URL from my previous session, it tries generating
| the article again, but seems to be hanging. Also, my 2nd
| account is unable to view the Discover page: a 403 error in dev
| tools says "User has not accepted consent form"
|
| I would think sharing by URL _should_ work, but has some bugs
| with it currently.
| jedberg wrote:
| Same experience. Tried sharing by URL and had the same issues
| you did.
| dvt wrote:
| I want to build this locally, I think it would be an absolute
| killer product. Could also consider doing an internet "deep dive"
| where the agent would browse for maybe 1-2 hours before sorting &
| collating the data. Add multi-modality for even more
| intelligence-gathering.
| kingkongjaffa wrote:
| Very cool! I asked it to create an article on the topic of my
| thesis and it was very good, but it lacked nuance and second
| order thinking i.e. here's the thing, what are the consequences
| of it and potential mitigations. It was able to pull existing
| thinking on a topic but not really synthesise a novel insight.
|
| Synthesis of Topic Outlines through Retrieval and Multi-
| perspective Question Asking.
|
| From the paper it seems like this is only marginally better than
| the benchmark approach they used to compare against:
|
| >Outline-driven RAG (oRAG), which is identical to RAG in outline
| creation, but
|
| >further searches additional information with section titles to
| generate the article section by section
|
| It seems like the key ingredients are:
|
| - generating questions
|
| - addressing the topic from multiple perspectives
|
| - querying similar wikipedia articles (A high quality RAG source
| for facts)
|
| - breaking the problem down by first writing an outline.
|
| Which we can all do at home and swap out the wikipedia articles
| with our own data sets.
| kingkongjaffa wrote:
| I was able to mimic this in GPT with out the RAG component with
| this custom instruction prompt, it does indeed write decent
| content, better than other writing prompts I have seen.
|
| PROMPT: create 3 diverse personas who would know about the user
| prompt generate 5 questions that each persona would ask or
| clarify use the questions to create a document outline, write
| the document with $your_role as the intended audience.
| dredmorbius wrote:
| [delayed]
___________________________________________________________________
(page generated 2024-09-17 23:00 UTC)