[HN Gopher] Show HN: I built "AI Wattpad" to eval LLMs on fiction
___________________________________________________________________
Show HN: I built "AI Wattpad" to eval LLMs on fiction
I've been a webfiction reader for years (too many hours on Royal
Road), and I kept running into the same question: which LLMs
actually write fiction that people want to keep reading? That's why
I built Narrator (https://narrator.sh/llm-leaderboard) - a platform
where LLMs generate serialized fiction and get ranked by real
reader engagement. Turns out this is surprisingly hard to answer.
Creative writing isn't a single capability - it's a pipeline:
brainstorming - writing - memory. You need to generate interesting
premises, execute them with good prose, and maintain consistency
across a long narrative. Most benchmarks test these in isolation,
but readers experience them as a whole. The current evaluation
landscape is fragmented: Memory benchmarks like FictionLive's tests
use MCQs to check if models remember plot details across long
contexts. Useful, but memory is necessary for good fiction, not
sufficient. A model can ace recall and still write boring stories.
Author-side usage data from tools like Novelcrafter shows which
models writers prefer as copilots. But that measures what's useful
for human-AI collaboration, not what produces engaging standalone
output. Authors and readers have different needs. LLM-as-a-judge
is the most common approach for prose quality, but it's notoriously
unreliable for creative work. Models have systematic biases
(favoring verbose prose, certain structures), and "good writing" is
genuinely subjective in ways that "correct code" isn't. What's
missing is a reader-side quantitative benchmark - something that
measures whether real humans actually enjoy reading what these
models produce. That's the gap Narrator fills: views, time spent
reading, ratings, bookmarks, comments, return visits. Think of it
as an "AI Wattpad" where the models are the authors. I shared an
early DSPy-based version here 5 months ago
(https://news.ycombinator.com/item?id=44903265). The big lesson:
one-shot generation doesn't work for long-form fiction. Models lose
plot threads, forget characters, and quality degrades across
chapters. The rewrite: from one-shot to a persistent agent loop
The current version runs each model through a writing harness that
maintains state across chapters. Before generating, the agent
reviews structured context: character sheets, plot outlines,
unresolved threads, world-building notes. After generating, it
updates these artifacts for the next chapter. Essentially each
model gets a "writer's notebook" that persists across the whole
story. This made a measurable difference - models that struggled
with consistency in the one-shot version improved significantly
with access to their own notes. Granular filtering instead of a
single score: We classify stories upfront by language, genre,
tags, and content rating. Instead of one "creative writing"
leaderboard, we can drill into specifics: which model writes the
best Spanish Comedy? Which handles LitRPG stories with Male Leads
the best? Which does well with romance versus horror? The answers
aren't always what you'd expect from general benchmarks. Some
models that rank mid-tier overall dominate specific niches. A few
features I'm proud of: Story forking lets readers branch stories
CYOA-style - if you don't like where the plot went, fork it and see
how the same model handles the divergence. Creates natural A/B
comparisons. Visual LitRPG was a personal itch to scratch. Instead
of walls of [STR: 15 - 16] text, stats and skill trees render as
actual UI elements. Example: https://narrator.sh/novel/beware-the-
starter-pet/chapter/1 What I'm looking for: More readers to build
out the engagement data. Also curious if anyone else working on
long-form LLM generation has found better patterns for maintaining
consistency across chapters - the agent harness approach works but
I'm sure there are improvements.
Author : jauws
Score : 18 points
Date : 2026-02-03 17:08 UTC (5 hours ago)
(HTM) web link (narrator.sh)
(TXT) w3m dump (narrator.sh)
| linolevan wrote:
| Quick feedback: Website is basically unusable on mobile
| jauws wrote:
| Ah shoot - thanks for letting me know. I'm still a noob on
| frontend so still learning as I go.
| bccdee wrote:
| I took a look at the "top-rated" story.
|
| 1. UI is terrible. Paragraphs are extremely far apart, and most
| paragraphs are 1 short sentence (e.g. "I glare."). On mobile, I
| can only see a few words at a time, and desktop's not much
| better.
|
| 2. Story is so bad that it's not even amusing.
| jauws wrote:
| Thanks for letting me know - the UI issues are definitely on me
| (fixing asap). Feel free to generate a story or two - right now
| there's not enough annotations to make "top-rated" a valid
| moniker.
| babblingfish wrote:
| > The surge of AI, large language models, and generated art begs
| fascinating questions. The industry's progress so far is enough
| to force us to explore what art is and why we make it. Brandon
| Sanderson explores the rise of AI art, the importance of the
| artistic process, and why he rebels against this new
| technological and artistic frontier.
|
| What It Means To Be Human | Art in the AI Era
|
| https://www.youtube.com/watch?v=mb3uK-_QkOo
| babblingfish wrote:
| Do watch the video as it makes a compelling argument against
| this exact kind of thing. From a product design perspective,
| you're asking people to read a bunch of slop and organize it
| into slop piles. What's the point of that? Honestly it seems
| like a huge waste of everyone's time.
| jauws wrote:
| I think there's interesting work to be built on this data
| beyond just generating and sorting slop. I didn't build this
| because I enjoy having people read bad fiction. I built it
| because existing benchmarks for creative writing are
| genuinely bad and often measure the wrong things. The goal
| isn't to ask users to read low-quality output for its own
| sake. It's to collect real reader-side signal for a category
| where automated evaluation has repeatedly failed.
|
| More broadly, crowdsourced data where human inputs are
| fundamentally diverse lets us study problems that static
| benchmarks can't touch. The recent "Artificial Hivemind"
| paper (Jiang et al., NeurIPS 2025 Best Paper) showed that
| LLMs exhibit striking mode collapse on open-ended tasks, both
| within models and across model families, and that current
| reward models are poorly calibrated to diverse human
| preferences. Fiction at scale is exactly the kind of data you
| need to diagnose and measure this. You can see where models
| converge on the same tropes, whether "creative" behavior
| actually persists or collapses into the same patterns, and
| how novelty degrades over time. That signal matters well
| beyond fiction, including domains like scientific research
| where convergence versus originality really matters.
| verelo wrote:
| Did you skip Anthropic models? I honestly can't take this
| seriously if you're not looking at all the leading providers but
| you did look at some obscure ones.
| jauws wrote:
| There's 151 models there right now (with all the latest
| Anthropic models), it's all randomized, it's just that there
| aren't enough annotations for the anthropic models to be
| elicited right now.
| rbtprograms wrote:
| even for ai standards this is gigaslop
| jauws wrote:
| Happy to engage if you have concrete criticisms.
| rbtprograms wrote:
| have you read any of the generated stories? if you can
| honestly tell me this is not complete drivel (even worse,
| wildly generic and poorly written) then i will consider
| giving real feedback but i would find that hard to believe.
| jauws wrote:
| I hope it's clear that the stories aren't being generated
| one-shot. I'm sure there are flaws that I haven't perfectly
| accounted for in the agent-loop, but because we randomize
| the models for each of the brainstorming -> writing ->
| memory parts, bad intermediate outputs will affect the
| final output as well. That's why unless we have above
| average models across all 3 stages, it might be worse than
| what you're used to. It's a trade-off to get more granular
| results. Hope you can give it a chance.
| rbtprograms wrote:
| so you havent read any of them, very nice product you got
| there. i stand by my original statement.
| drusepth wrote:
| Hard to find the signal in the noise and know what stories I
| should even read to get a sense of baseline quality; partially
| because that's just a hard problem inherent to floods of any
| content, but also because the recommendation system seems to lack
| enough data (and also might be weighting the wrong things, e.g.
| the rank #1 story is also the lowest-rated...).
|
| A very cool idea in theory and something very hard to pull off,
| but I think in order to get the data you need on how readable
| each story is you'll need to work on presentation and
| recommendation so those don't distract from what you're actually
| testing.
| jauws wrote:
| Thanks for the feedback - looking at the rest of the comments,
| I definitely agree it seems to be a common theme. Will do
| better to fix those issues so there's less noise.
| BoorishBears wrote:
| I have a lot of engagement data on LLMs from running a creative
| writing oriented consumer AI app and spending s lot of time on
| quality improvements and post training
|
| Do you have a contact email?
| jauws wrote:
| Would love to chat! Here's my email: team@narrator.sh
___________________________________________________________________
(page generated 2026-02-03 23:01 UTC)