[HN Gopher] Show HN: Cancer diagnosis makes for an interesting R...
___________________________________________________________________
Show HN: Cancer diagnosis makes for an interesting RL environment
for LLMs
Hey HN, this is David from Aluna (YC S24). We work with diagnostic
labs to build datasets and evals for oncology tasks. I wanted to
share a simple RL environment I built that gave frontier LLMs a set
of tools that lets it zoom and pan across a digitized pathology
slide to find the relevant regions to make a diagnosis. Here are
some videos of the LLM performing diagnosis on a few slides:
(https://www.youtube.com/watch?v=k7ixTWswT5c): traces of an LLM
choosing different regions to view before making a diagnosis on a
case of small-cell carcinoma of the lung
(https://youtube.com/watch?v=0cMbqLnKkGU): traces of an LLM
choosing different regions to view before making a diagnosis on a
case of benign fibroadenoma of the breast Why I built this:
Pathology slides are the backbone of modern cancer diagnosis.
Tissue from a biopsy is sliced, stained, and mounted on glass for a
pathologist to examine abnormalities. Today, many of these slides
are digitized into whole-slide images (WSIs)in TIF or SVS format
and are several gigabytes in size. While there exists several
pathology-focused AI models, I was curious to test whether frontier
LLMs can perform well on pathology-based tasks. The main challenge
is that WSIs are too large to fit into an LLM's context window. The
standard workaround, splitting them into thousands of smaller
tiles, is inefficient for large frontier LLMs. Inspired by how
pathologists zoom and pan under a microscope, I built a set of
tools that let LLMs control magnification and coordinates, viewing
small regions at a time and deciding where to look next. This
ended up resulting in some interesting behaviors, and actually
seemed to yield pretty good results with prompt engineering: - GPT
5: explored up to ~30 regions before deciding (concurred with an
expert pathologist on 4 out of 6 cancer subtyping tasks and 3 out
of 5 IHC scoring tasks) - Claude 4.5: Typically used 10-15 views
but similar accuracy as GPT-5 (concurred with the pathologist on 3
out of 6 cancer subtyping tasks and 4 out of 5 IHC scoring tasks)
- Smaller models (GPT 4o, Claude 3.5 Haiku): examined ~8 frames and
were less accurate overall (1 out of 6 cancer subtytping tasks and
1 out of 5 IHC scoring tasks) Obviously, this was a small sample
set, so we are working on creating a larger benchmark suite with
more cases and types of tasks, but I thought this was cool that it
even worked so I wanted to share with HN!
Author : dchu17
Score : 32 points
Date : 2025-11-12 17:01 UTC (5 hours ago)
| n2d4 wrote:
| How would a human classify the cancers? I assume the LLM training
| data does not include a whole bunch of cancer samples, so
| assumably there are some rules that it follows?
|
| > While there exists several pathology-focused AI models
|
| Would also be curious how the LLM compares to this and other
| approaches. What's the performance of the models trained
| specifically on this task, and random guessing, compared to the
| expert pathologist? Correct me if I'm wrong but this seems like
| the sort of task where being right 90% of the time is not good
| enough, so even if the LLM beats other approaches, it still needs
| to close the gap to human performance.
| dchu17 wrote:
| > What's the performance of the models trained specifically on
| this task, and random guessing, compared to the expert
| pathologist?
|
| I should probably first clarify here, the disease
| classification tasks are about subtyping the type of cancer
| (i.e classifying a case as invasive ductal carcinoma of the
| breast) rather than just binary malignant/benign classification
| so random guessing is much more difficult and makes this model
| performance more impressive.
|
| > Would also be curious how the LLM compares to this and other
| approaches.
|
| There aren't a lot of public general purpose pathology
| benchmarks. There are some like (https://github.com/sinai-
| computational-pathology/SSL_tile_be...) but focus on just
| binary benign/malignant classification tasks and binary
| biomarker detection tasks.
|
| I am currently working on self-hosting the available open-
| source models.
|
| > this seems like the sort of task where being right 90% of the
| time is not good enough, so even if the LLM beats other
| approaches, it still needs to close the gap to human
| performance
|
| Yep, your intuition is right here, and actually the expectation
| is probably closer to mid-high 90%, especially for FDA approval
| (and most AI tools position as co-pilots at the moment). There
| is obviously a long way to go, but what I find about
| interesting about this approach is that it allows LLMs to
| generalize across (1) a variety of tissue types and (2)
| pathology tasks such as IHC H-score scoring.
| ytrt54e wrote:
| You should read out to Eric Topol...
| 0xbeebs wrote:
| very cool, have you tried some of the newer segmenting models to
| see if they make a difference? I've seen some in the past two
| weeks that look really effective...I wonder if it could help out
| the RL environment
| dchu17 wrote:
| Nope I haven't, I can take a look and see if I can fit it in
| Utkarsh_Mood wrote:
| Do you think finetuning these LLMS would bring about comparable
| results to specific models trained for this?
| dchu17 wrote:
| I think so. It feels like there is more to be squeezed from
| just better prompts but was going to play around with fine-
| tuning Qwen3
| Utkarsh_Mood wrote:
| fair enough. I wonder if fine-tuning over different
| modalities like IMC, H&E etc would help it generalize better
| across all
| areoform wrote:
| Did you fine tune GPT 5, Sonnet 4.5, or any of the other models?
| Or, were the models able to do this "out of the box?"
| Utkarsh_Mood wrote:
| none of the models you mentioned are open source...
| dchu17 wrote:
| Nope, I just did some prompt engineering on ootb models. I
| thought about doing some fine-tuning on like Qwen but think
| that there is still more performance to be squeezed out with
| just prompts here.
| austinwang115 wrote:
| Wow this is pretty interesting. Excited to see the benchmark!
| xrd wrote:
| Fascinating stuff.
|
| For some reason, this reminds me the way video encoders compress
| video:
|
| https://en.wikipedia.org/wiki/Video_compression_picture_type...
|
| It makes me wonder if you could use a similar technique (iframes,
| bframes or pframes) to get the diff of a "normal" WSI and then
| train on pattern recognition of those.
|
| These different frames are used to reduce network transmission
| costs, but it feels similar to the context window if you squint
| at it as a throughput problem rather than a context window size
| problem.
|
| It feels like there would be a lot of tools and codecs you could
| leverage here.
| lawrencechen wrote:
| I wonder if navigation plays a significant role in performance.
| If you just randomly select 15 frames (presumably with
| interesting pixels), will the model perform similarly well?
___________________________________________________________________
(page generated 2025-11-12 23:01 UTC)