[HN Gopher] My trick for getting consistent classification from ...
___________________________________________________________________
My trick for getting consistent classification from LLMs
Author : frenchmajesty
Score : 37 points
Date : 2025-10-13 18:01 UTC (7 days ago)
(HTM) web link (verdik.substack.com)
(TXT) w3m dump (verdik.substack.com)
| jawns wrote:
| If you already have your categories defined, you might even be
| able to skip a step and just compare embeddings.
|
| I wrote a categorization script that sorts customer-service calls
| into one of 10 categories. Wrote descriptions of each category,
| then translated into embedding.
|
| Then created embeddings for the call notes and matched to closest
| category using cosine_similarity.
| kurttheviking wrote:
| Out of curiosity, what embedding model did you use for this?
| nerdponx wrote:
| How did you construct the embedding? Sum of individual token
| vectors, or something more sophisticated?
| olliepro wrote:
| sentence embedding models are great for this type of thing.
| svachalek wrote:
| That was my first thought, why even generate tags? Curious to
| see if anyone's proved it's worse empirically though.
| soldeace wrote:
| In a recent project I was asked to create a user story
| classifier to identify whether stories were "new development"
| or "maintenance of existing features". I tried both
| approaches, embeddings + cosine distance vs. directly asking
| a language model to classify the user story. The embeddings
| approach was, despite being fueled by the most powerful SOTA
| embedding model available, surprisingly worse than simply
| asking GPT 4.1 to give me the correct label.
| frenchmajesty wrote:
| OP here. It depends what you use it for. You do want the tags
| if you intend to generate data. Let's say you prompt an LLM
| to go tweet on your behalf for a week, having the ability to:
|
| - Fetch a list of my unique tags to get a sense of my topics
| of interests
|
| - Have the AI dig into those specific niches to see what
| people have been discussing lately
|
| - Craft a few random tweets that are topic-relevant and
| present them to me to curate
|
| Is very powerful workflow that is hard to deliver on without
| the class labels.
| axpy906 wrote:
| Arthur's classifier will only be as accurate as their retrieval.
| The approach depends on the candidates to be the correct ones for
| classification to work.
| frenchmajesty wrote:
| OP here. This is true. If you make your min_score .99 you can
| have very high confidence in copy-pasting the label, but then
| this is not very useful. The big question is then how far can
| you get from 0.99 while still having satisfying results?
| sethkim wrote:
| Under-discussed superpower of LLMs is open-set labeling, which I
| sort of consider to be inverse classification. Instead of using a
| static set of pre-determined labels, you're using the LLM to find
| the semantic clusters within a corpus of unstructured data. It
| feels like "data mining" in the truest sense.
| frenchmajesty wrote:
| OP here. This is exactly right! You perfectly encapsulated the
| idea I stumbled up so beautifully.
| dinobones wrote:
| Dunno if this passes the bootstrapping test.
|
| This is sensitive to the initial candidate set of labels that the
| LLM generates.
|
| Meaning if you ran this a few times over the same corpus, you'll
| probably get different performance depending upon the order of
| the way you input the data and the classification tag the LLM
| ultimately decided upon.
|
| Here's an idea that is order invariant: embed first, take samples
| from clusters, and ask the LLM to label the 5 or so samples
| you've taken. The clusters are serving as soft candidate labels
| and the LLM turns them into actual interpretable explicit labels.
___________________________________________________________________
(page generated 2025-10-20 23:00 UTC)