[HN Gopher] Show HN: BBC "In Our Time", categorised by Dewey Dec...
___________________________________________________________________
Show HN: BBC "In Our Time", categorised by Dewey Decimal, heavy
lifting by GPT
I'm a big fan of the BBC podcast In Our Time -- and (like most
people) I've been playing with the OpenAI APIs. In Our Time has
almost 1,000 episodes on everything from Cleopatra to the evolution
of teeth to plasma physics, all still available, so it's my
starting point to learn about most topics. But it's not well
organised. So here are the episodes sorted by library code. It's
fun to explore. Web scraping is usually pretty tedious, but I
found that I could send the minimised HTML to GPT-3 and get
(almost) perfect JSON back: the prompt includes the Typescript
definition. At the same time I asked for a Dewey classification...
and it worked. So I replaced a few days of fiddly work with 3 cents
per inference and an overnight data run. My takeaway is that I'll
be using LLMs as function call way more in the future. This isn't
"generative" AI, more "programmatic" AI perhaps? So I'm interested
in what temperature=0 LLM usage looks like (you want it to be
pretty deterministic), at scale, and what a language that treats
that as a first-class concept might look like.
Author : genmon
Score : 277 points
Date : 2023-03-08 19:28 UTC (3 hours ago)
(HTM) web link (genmon.github.io)
(TXT) w3m dump (genmon.github.io)
| genmon wrote:
| Some unlinked features...
|
| If you put the Dewey division in the URL, the directory auto-
| opens. e.g. here are episodes about prehistoric life (my current
| jumping-off point)
|
| https://genmon.github.io/braggoscope/directory#560
|
| There's a visual map of episodes. After principal component
| analysis of the episode embedding vectors, these are the most
| significant two components as the x,y
|
| https://genmon.github.io/braggoscope/map.html
|
| (it's not super useful tbh -- e.g. the Manhattan Project and the
| Cambrian Explosion have the same x,y... presumably because they
| are both about explosions?)
|
| Many episodes have a reading list, and these are all linked to
| Google Books (so you can purchase/check out from a library), e.g.
| this episode page
|
| https://genmon.github.io/braggoscope/2022/10/20/the-fishtetr...
|
| There are ~4,600 books, and I have ~88% coverage on getting a
| Google Books page from the original data. Any ideas about what to
| do with this big list of academic-recommended books v welcome!
| CrypticShift wrote:
| Excellent! I'd love to see a script that sends a list of
| "descriptions" (1-100 words) to ChatGPT and directly gives you
| back a ready-made (embedding vectors closeness) map in a
| (textual) graph/chart format (like your above map or your plot
| https://interconnected.org/more/2023/02/in_our_time-PCA-
| plot...)
| genmon wrote:
| It turns out that "closeness" is usually hard to
| visualise/explore when you're dealing with a
| 1,000-dimensional space... and PCA has the failures mentioned
| above.
|
| It's weird -- it's locally useful to navigate, and at a high
| level kinda useful, but only if you squint and don't look at
| the problems. So I feel like a fisheye visualisation would be
| appropriate? That's something that I'm exploring in other
| projects.
| gwern wrote:
| I wouldn't necessarily reach for PCA. No reason to think
| that the first two principal components necessarily encode
| anything particularly interesting. If you want to lay out
| each point in 2D in a way which keeps similar points
| nearby, something like t-SNE is worth a try - visualizing
| embeddings is what it was invented for.
| genmon wrote:
| Excellent, new to me and I'll give it a go, thanks!
|
| I gravitate to PCA for terrible reasons (undergrad so
| it's what I think of first) and like you say, it's
| beguiling yet disappointing, the components rarely have
| any human meaning.
| pigscantfly wrote:
| I'd suggest trying t-SNE [1] instead; you'll be losing
| almost all of the variance by projecting onto the first two
| eigenvectors produced by PCA.
|
| [1] http://karpathy.github.io/2014/07/02/visualizing-top-
| tweeps-...
| p0pcult wrote:
| Love the visual map. What does color mean? Any way to do a 3rd
| PC, and put the visualization in a cube one can toy around
| with?
| genmon wrote:
| Colour is the 3rd component -- I wanted to see the difference
| between overlapping episodes.
|
| As for the 3D plot... here you go!
|
| https://interconnected.org/more/2023/03/in_our_time-
| PCA-3D-p...
|
| Basic PCA + Plotly is actually in OpenAI's official Python
| library (in `embedding_utils`) -- this plot is just the
| output from that.
| p0pcult wrote:
| :D this made my day.
| sacrosancty wrote:
| [dead]
| flir wrote:
| This is great. I'll certainly be thinking of "classification"
| uses for ChatGPT in the future.
|
| Thinking out loud: Add the experts, not just the reading lists.
| They're a jumping-off point into academic-paper-space. What
| have they published? In what journals? Who have they
| collaborated with?
| dvt wrote:
| The BBC offers pretty exhaustive RSS feeds
| (https://podcasts.files.bbci.co.uk/b006qykl.rss) so I'm not
| exactly sure what ChatGPT even did here (Maybe the Dewey
| classification? Which is of dubious usefulness).
| genmon wrote:
| There's more on the About page but in summary: extracts the
| synopsis, guests (name, affiliation), and reading list (title,
| author, publisher, year) as structured data. Not massively hard
| with a bit of web scraping, but tedious and results in brittle
| code -- this took 20 minutes to write the prompt plus 3 cents
| per episode.
|
| https://genmon.github.io/braggoscope/about
|
| (GPT-3 not ChatGPT for the model.)
| sorokod wrote:
| For comparison two links for the (currently) last episode,
| first from BBC, the second from genmon
|
| https://www.bbc.co.uk/programmes/m001jkzg
|
| https://genmon.github.io/braggoscope/2023/03/02/megaliths.ht...
| theodorewiles wrote:
| I emailed you but did something similar and posted to Show HN a
| while back -
|
| https://weekend-collection.s3.amazonaws.com/Catalog+-+Feb+17...
| genmon wrote:
| Just to make sure that others see this, I found your clustering
| technique super interesting and very effective, and something I
| want to try myself. Your technical writeup is here:
|
| https://weekendcollection.substack.com/p/technical-details
|
| (Recursive coarse clustering as opposed to one-shot fine
| clustering.)
| donut wrote:
| This is so cool! Thanks for sharing.
|
| Taxonomies are inherently limited -- I love this portion of the
| talk "Everything is Miscellaneous" about Melvil Dewey:
| https://www.youtube.com/watch?v=x3wOhXsjPYM&t=1206s
|
| Surely some things fit equally well in more than one category.
|
| Have you considered asking for best two or three categories, and
| placing the episodes in multiple locations? Or would that be too
| noisy?
| genmon wrote:
| I found that Dewey was borderline acceptable -- more categories
| seemed to degrade it, though admittedly I didn't spend much
| time prompt tuning. I also tried tags and these weren't
| reliable (not at a consistent "scale" from episode to episode).
|
| I suspect I'll need a more mechanical approach, long term.
| vidro3 wrote:
| I always assumed the podcast was current with the show but it
| seems like it's about 4 weeks behind.
|
| e.g. I have Tycho Brahe on March 2 in my podcast app, rss shows
| it released Feb 2.
| genmon wrote:
| The BBC is pushing its Sounds app by lagging the podcast.
| Disappointing but understandable I guess given they want the
| opportunity to suggest other programmes.
| astroalex wrote:
| I love this project!!
|
| Ever since my partner and I discovered In Our Time a few years
| back, it's been our go-to podcast to listen to together. Part of
| the allure is that the archive is so vast, but that makes it hard
| to browse.
|
| My partner made her own archive of In Our Time here, if you're
| interested: https://shelby.cool/melvyn/
|
| She used Wikipedia to find and categorize each episode. I also
| really like that she indexed episodes by guest, too. Certain
| guests are REALLY good and have been on many episodes.
|
| Super excited to see someone else make an archive; we'll
| definitely be exploring yours!
| genmon wrote:
| No way! This is incredible. That h1 "Hello," <3
|
| Is the tagging manual? It's really good.
| bscphil wrote:
| > She used Wikipedia to find and categorize each episode.
|
| This is a really clever use of an existing dataset. I clicked
| through before reading this and was stunned by how thorough the
| tag set was. Even more obscure things like "Alumni of Magdalen
| College, Oxford" have multiple episodes. I'm going to keep this
| in mind on future projects for sure.
| zeristor wrote:
| That's really impressive.
|
| I've had a mild idea for ages to have a sort of annotated In
| Our Time. Listening to the podcast on a webpage as the text
| rolls by links could appear to explain or give background to
| the item or person being discussed.
|
| SMIL is the multi-media mark up language. Generally if one
| thinks of something there's someone on the Internet who has
| already had that idea.
|
| Additional: I think the BBC is very careful about
| transciptions. They've sold a book of the transcripts of
| several episodes, but it would be a great way to go through a
| subject.
| EamonnMR wrote:
| In Our Time is one of the best podcasts out there. Are there any
| similar ones?
| maccaw wrote:
| Love it - also a huge fan of In our Time.
|
| V2 would be:
|
| - Transcribe through Whisper - Semantic search
| specproc wrote:
| I love In Our Time, a real BBC gem.
|
| I've been meaning to pull all the audio for a while and this has
| inspired me.
|
| A fun thing to do would be to pass through Whisper, a great
| corpus to play with.
| genmon wrote:
| I've been considering this, but my assumption is that it would
| be tripped up by the specialist words.
|
| I wonder... is there a way to "prime" Whisper (e.g. with the
| embedding of the episode synopsis) so that it "listens out" for
| words related to a particular topic? I haven't looking but this
| would be neat!
| taberiand wrote:
| I haven't tried, but the Open AI docs mention priming on the
| whisper model being available prompt
| string Optional An optional text to
| guide the model's style or continue a previous audio segment.
| The prompt should match the audio language.
| genmon wrote:
| I hadn't noticed that, very much appreciated.
| simonw wrote:
| Oh that's really neat! I hadn't seen that:
| https://platform.openai.com/docs/guides/speech-to-
| text/promp...
| coder543 wrote:
| In my experience, Whisper does a great job even with
| specialized terminology. It won't catch everything, but I
| think it will exceed your expectations. One of the hardest
| things about Whisper is choosing which model to use; they
| offer a variety of sizes, and sometimes the smaller ones do
| better than the larger ones. It's worth trying a few
| different models and deciding what is best for each
| particular application.
|
| I will also say that I've personally been unimpressed with
| the new "large-v2" model, even though it supposedly scores
| better. The original "large-v1" model seems to work better
| than the "large-v2" model in the audio clips I've been
| testing Whisper against, but results will vary. In general, I
| find I'm really happy with what "small.en" and "medium.en"
| will emit, and they're much faster than the large models.
| (The ".en" models are specialized to English, and usually
| perform better for strictly English input, whereas the
| non-".en" models are trained on multiple languages.)
| simonw wrote:
| I'd still like the ability to prime Whisper. I used it to
| transcribe a podcast episode I appeared on recently and one
| of the fixes I had to make was that ChatGPT came out as
| "chat GPT" every time it was mentioned:
| https://simonwillison.net/2023/Mar/7/kqed-forum/#kqed-forum
|
| Update: turns out this exists already:
| https://platform.openai.com/docs/guides/speech-to-
| text/promp...
| coder543 wrote:
| I recorded myself saying a few sentences from that
| transcript, then fed it through different Whisper models.
| "small.en" and "large-v1" both generated "chat GPT",
| "large-v2" generated "chat-gpt", but somehow "medium.en"
| correctly generated "ChatGPT".
|
| This was the same audio sample fed through each of those
| four models, with no "prompting" as you're discussing.
|
| If I add "--initial_prompt ChatGPT", then all four models
| are able to get the spelling correct.
|
| Regardless, I don't think "chat GPT" versus "ChatGPT" is
| a huge deal. There will always be some level of
| uncertainty and ambiguity in the transcript, and even
| books written by humans always have a few typos get past
| multiple stages of copy editing. Perfection is virtually
| unachievable, but you can always scroll through the
| transcript and make some edits after the fact, if
| desired. Maybe some future model will magically eliminate
| all typos.
| simonw wrote:
| Yeah it wasn't a big problem for me - I had to do a bunch
| of other tidy-ups on the transcript anyway to add things
| like the name of the person who was speaking.
|
| I cleaned that bit up with a bulk replace of "chat GPT"
| with "ChatGPT" in VS Code.
| DC-3 wrote:
| Nice project OP. I also love In Our Time.
|
| Some favourite episodes off the top of my head:
|
| * Wilfred Owen - https://www.bbc.co.uk/programmes/m001df48
|
| * The Evolution of Crocodiles -
| https://www.bbc.co.uk/programmes/m000zmhf
|
| * The May Forth Movement -
| https://www.bbc.co.uk/programmes/m001282c
|
| * The Valladolid Debate -
| https://www.bbc.co.uk/programmes/m000fgmw
|
| * Gerard Manley Hopkins -
| https://www.bbc.co.uk/programmes/m0003clk
|
| * Henrik Ibsen - https://www.bbc.co.uk/programmes/b0b42q58
|
| * Wuthering Heights - https://www.bbc.co.uk/programmes/b095ptt5
|
| And finally, in which three mathematicians heroically attempt to
| explain asymptotic analysis to (septuagenarian novelist and
| cultural broadcaster) Melvyn:
|
| * P v NP - https://www.bbc.co.uk/programmes/b06mtms8
| dendrite9 wrote:
| I really enjoyed the recent Superconductivity episode.
| Especially hearing Melvyn say 'Good god' half way through.
| genmon wrote:
| Great list! Some personal faves in return, in no particular
| order:
|
| - The evolution of teeth
| https://www.bbc.co.uk/programmes/m0003zbg
|
| - The fish-tetrapod transition
| https://www.bbc.co.uk/programmes/m001d56q
|
| - The late Devonian extinction
| https://www.bbc.co.uk/programmes/m000sz7x
|
| - The American West https://www.bbc.co.uk/programmes/p00548gg
|
| - Metamorphosis (Ovid)
| https://www.bbc.co.uk/programmes/p00546p6
|
| - Politeness https://www.bbc.co.uk/programmes/p004y29m
|
| - The Bronze Age collapse
| https://www.bbc.co.uk/programmes/b07fl5bh
|
| - Doggerland https://en.wikipedia.org/wiki/Doggerland
| yung_steezy wrote:
| Also going to chime in to recommend the Gin Craze episode
| https://www.bbc.co.uk/programmes/b084zk6z
| boringg wrote:
| Also a big fan of the show -- cool project!
| mattlondon wrote:
| Finally, my interest in LLMs is piqued!
|
| Seems like everyone has been getting excited around the search or
| code-generation use cases ... or simply trying to make it say
| naughty things (boring, not interested, wake up in a few more
| years), but this is eye opening.
|
| The idea of this as a "universal coupler" is fascinating, and I
| think I agree with the author that we are probably standing at an
| early-90s-web moment with LLMs as a function call (the technology
| is kinda-there and mostly-works, and people are trying out a lot
| of ideas ... some work, some don't).
|
| My mind is racing. Thanks for the epiphany moment.
| aardvark179 wrote:
| I see several very dubious classifications. Shouldn't the great
| stink be under civil engineering rather than agriculture, and why
| is Plato's Atlantis under computer science?
| genmon wrote:
| Now that's an interesting regression! I don't remember seeing
| it there before.
|
| (The worst I've noticed before has been Lawrence of Arabia
| under History of the Ancient World. Very much 20th century
| really.)
|
| Several other classifications are arguable -- which I think
| shows one of the limitations of this technique: it's not
| possible to iterate + improve.
|
| So instead I've been wondering about using the embeddings of
| each episode synopsis, and comparing to the embeddings of Dewey
| subdivisions. I should be able to tune the results better that
| way.
|
| There's also a technique from Google called CAVs (Concept
| Activation Vectors) that I'm intrigued about trying -- would
| love to hear if anybody has experience using this
|
| https://arxiv.org/abs/1711.11279
| once_inc wrote:
| There's also an episode about aliens under 000.
| jgrahamc wrote:
| What a beautiful thing.
| haunter wrote:
| Something like this for Essential Mix episodes with tracklists
| and genre would be mindblowing
| jccalhoun wrote:
| It is an interesting approach and I appreciate it. However, an
| alphabetical list would be more useful for me when I am
| interested in topics and might not know where they are
| classified.
| valgaze wrote:
| This is a really interesting use-case
|
| Applying "transformations" or classifying data in this way
| without having to setup a lot of detail-work seems like a real
| labor-saver/multiplier
| lerchmo wrote:
| My company has gone years wanting our product catalog to have
| structured data around our products but not going through the
| tedium of extracting it all. about an hour of prompt tweaking
| and it can pull, normalize, summarize and output valid json
| from all of our products. Basically pulling it out of a big
| unstructured html blob.
| genmon wrote:
| I've started thinking about LLMs as a "universal coupling" _,
| if that makes sense? It 's wild to be able to conceive of APIs
| to plain text, and natural language queries on structured APIs,
| but that's what we've got.
|
| My mind was really opened by Nat Friedman's work in GPT for
| browser automation: https://github.com/nat/natbot
|
| And of course using langchain/ReACT.
|
| So different from ChatGPT and (imo) way more intriguing.
|
| _ mentioned in this blog post:
| https://interconnected.org/home/2023/02/07/braggoscope
| jack_riminton wrote:
| In Our Time has been a life-long companion, one of things that
| makes me proud of the BBC
| genmon wrote:
| Have you read the New Yorker articles about it?
|
| On _In Our Time_ -> https://www.newyorker.com/culture/podcast-
| dept/escape-the-ne...
|
| Profile of Melvyn Bragg ->
| https://www.newyorker.com/culture/the-new-yorker-interview/t...
|
| Both well worth your time.
| jack_riminton wrote:
| That NewYorker summarises it nicely. I can't stand the over-
| produced stuff on NPR for example. In Our Time has no
| politics, no hook to get you listening for the next episode.
| It's just a lovely small window into academia. Above all it
| makes things simple without dumbing it down
| cheschire wrote:
| No longer does Tony the Pony need to parse XHTML with regex. Who
| knew GPTs would be the actual solution?
| haolez wrote:
| Wait. Is temperature=0 "pretty deterministic", or is it
| deterministic?
| genmon wrote:
| In the strict definition, it's deterministic: you get the same
| response for the same prompt, each time (given the exact same
| model).
|
| But the prompt is parameterised. The bulk of the prompt is
| requesting a list of guests and speakers to be extracted, and
| the episode synopsis is appended as the "parameter". And I've
| noticed that the variation of the parameter changes what the
| overall prompt returns... so it might start being less reliable
| at responding with valid JSON, for example.
|
| So it's instance-deterministic but, across a range of
| parameters, class-fuzzy, if that makes sense?
| acchow wrote:
| On a related note, has anyone noticed the growing use of
| "pretty" as a hedge against ever being wrong? Also, "not", as
| in "not the fastest" or "not the hottest".
|
| It could be 46C/115F outside and you say "wow, today is
| unbearably hot", to which someone retorts "nah, it's fine. It's
| not the hottest day". That's pretty good hedging. You can make
| infinite technically correct statements this way without ever
| saying anything meaningful
| messe wrote:
| > Also, "not", as in "not the fastest" or "not the hottest".
|
| Mid twenties Hiberno-English speaker here and that's always
| been a fairly common form of hedging, and I've not noticed an
| increase.
| tedsanders wrote:
| Temperature=0 is not perfectly deterministic for the OpenAI
| API.
|
| It's mentioned briefly in the OpenAI text completion guide:
| https://platform.openai.com/docs/guides/completion/introduct...
|
| If you have two possible tokens with probability 40% and 30%,
| you'll always get the 40% token at T=0. But if you have two
| possible tokens at 40% and 39.99%, you may get the 39.99% token
| on occasion, even if at T=0. (Numbers illustrative.)
| maccaw wrote:
| Well technically everything is deterministic.
| LeoPanthera wrote:
| It's my understanding that 0 is completely deterministic,
| except for when the model is updated, which does happen
| occasionally.
| gwillen wrote:
| Their docs say (somewhat recently updated, I think) that even
| 0 is not perfectly deterministic in all cases (though it's
| very close). Some people had previously observed this and
| speculated that it was some kind of floating point roundoff
| issue, when two outputs have almost identical scores.
| valgaze wrote:
| @haolez I just tested w/ davinci-003 w/ temperature to 0
|
| Prompt: https://imgur.com/YtQ4fbf --
|
| Reveal the question marks in an interesting way:
|
| The dog goes ????????????? --
|
| With temperature == 0, it consistently ("pretty
| deterministically"?) generated "woof!"
|
| ex. The dog goes woof!
| PaulHoule wrote:
| What's the prompt?
| valgaze wrote:
| From what I understand:
|
| - Typescript definition (turn raw html markup >> well-formed
| JSON)
|
| - Dewey Decimal classification/score
|
| So probably combination of those two
| genmon wrote:
| One of the prompts is Extract the description
| and a list of guests from the supplied episode notes from a
| podcast. Also provide a Dewey Decimal Classification
| code and label for the description Return valid JSON
| conforming to the following Typescript type definition:
| { "description": string, "guests": {"name":
| string, "affiliation": string | null}[]
| "dewey_decimal": {"code": string, "label": string}, }
| Episode synopsis (Markdown): {notes}
| Valid JSON:
|
| (And the completion tends to be JSON, but not always.)
| lerchmo wrote:
| I like the typescript definition, rather than example json
| that I normally use.
| genmon wrote:
| Credit where it's due: I was working with structured data
| as JSON for the completion, and the Typescript definition
| hugely increased reliability. I took that from helpful
| advice (on Twitter) from Noah Brier who afiak came up with
| the approach:
|
| https://brxnd.substack.com/p/the-prompt-to-rule-all-
| prompts-...
| simonw wrote:
| I wanted to see which speakers had been on the most episodes.
|
| I went to https://genmon.github.io/braggoscope/guests and opened
| the Firefox DevTools console and ran this:
| guests = Array.from(document.querySelectorAll('ul a.text-
| blue-500.underline')).map(el => ({ name: el.innerText,
| count: parseInt(el.nextElementSibling.textContent.slice(1, -1),
| 10) }))
|
| Then this: guests.sort((a, b) => a.count <
| b.count) console.log(JSON.stringify(guests, null, 2))
|
| The top few were: [ {
| "name": "Simon Schaffer", "count": 24 },
| { "name": "Angie Hobbs", "count": 23
| }, { "name": "Martin Palmer",
| "count": 22 }, { "name": "Steve
| Jones", "count": 21 }, {
| "name": "Paul Cartledge", "count": 20 }
| genmon wrote:
| I think Angie Hobbs will take it! Looks like she is referred to
| with a few different names in the episode notes and I'll need
| to merge them manually (will be 25 appearances)
|
| (btw thanks for datasette. I use SQLite as an intermediary db
| and datasette was invaluable for exploring and refining
| queries.)
| simonw wrote:
| Any chance you might share the intermediary DB? Even just
| including the binary database file on GitHub somewhere would
| be neat, since I could play around it by doing
| https://lite.datasette.io/?url=URL-to-your-db-file-on-GitHub
| rwmj wrote:
| In my memory, Frank Close has been on just about every one, but
| I think that's because I mostly listen to the ones about
| physics and astronomy.
| https://en.wikipedia.org/wiki/Frank_Close
| I_complete_me wrote:
| Lots of great contributors to the show over the years. My
| favourite is Steve Jones for his huge intellect, humility,
| knowledge and listenabilty - yeah it's the right word for
| someone like him. I've listened to a lot of episodes (some many
| times over). I'll just mention The Migration of Birds as a
| sound listen.
|
| However, Melvyn is an "interesting" catalyst but still great to
| fall asleep to. Long may it continue.
|
| I believe there is sister program with a younger female
| presenter. Name escapes me...
___________________________________________________________________
(page generated 2023-03-08 23:00 UTC)