[HN Gopher] FOSDEM 2023 Transcribed by Whisper
___________________________________________________________________
FOSDEM 2023 Transcribed by Whisper
Author : jonatron
Score : 90 points
Date : 2023-02-12 09:24 UTC (13 hours ago)
(HTM) web link (jonatron.github.io)
(TXT) w3m dump (jonatron.github.io)
| remram wrote:
| I was wondering how that compared to Vosk, a lightweight pip-
| installable speech-to-text library. So I ran Vosk on some of the
| videos to compare. The model I used is 40MB and transcribed the
| 32min keynote in 2m12s on 1 CPU core, using 400MB of memory.
|
| The result is not that far. There are some errors of course,
| notably on acronyms and technical terms ("IBM", "GNU", "GitHub"),
| but a lot of the mismatches are in areas where both models got
| things wrong.
|
| Neither of the models got "FOSDEM" right, though Whisper is
| consistent at using the word "fosstem", whatever that is?
| [deleted]
| nbenitezl wrote:
| This is gold for us not native english speakers.
| specproc wrote:
| It's gold for non-native speakers of any supported language.
|
| I'm trying to learn some Spanish, and a fun and easy way to
| study is to watch films in Spanish with Spanish-language subs.
|
| Finding Jurassic Park on piratebay was easy, finding the right
| subs was impossible. Enter Whisper.
|
| My favourite little hack this year.
| fareesh wrote:
| The accuracy of whisper seems to be the best available for mere
| mortals, but the resource requirements are fairly high. It
| doesn't seem like it is possible to use it for real-time
| transcription.
|
| What solutions are Twitter spaces and the like using for real-
| time live captions?
| adgjlsfhk1 wrote:
| there's a c++ fork of whisper that can run the smaller models
| on a laptop CPU in real time
| nindalf wrote:
| This is pretty good! I spot checked one video
| (https://jonatron.github.io/fosdem2023whisper/files/rust_core...)
| and it's done a decent job. It mostly makes mistakes around a few
| proper nouns.
|
| - coreutils => corridors/coroutils/cori-
| teals/curricules/corretails
|
| - Rust => rest/Resi
|
| - glibc => GLC
|
| - Gentoo => Gen2
|
| - Fuchsia => Fushia
|
| - SELinux => AC Linux
|
| Other proper nouns it transcribed well - LLVM, Mozilla, Debian,
| Ubuntu, clang. These are much better known I suppose.
|
| It finds the speaker's French accent a bit difficult, because I
| see a few mistakes like "find => fine", "imposter =>
| imposterous", "command => comment".
|
| Overall pretty good. The only thing I'd like to improve
| readability is a pass with an LLM to break it into paragraphs but
| that would be way more compute and no longer local (like
| Whisper).
| margorczynski wrote:
| Maybe fine tuning on "tech lingo" would improve that. Give that
| to your average person to transcribe and the results most
| probably would be worse than that.
| CraftingLinks wrote:
| Missed opportunity to feed the transcripts to LLM and prompt it
| to clean it up, (remove stop words, stuff like that)? -> instant
| blog post to go with the talk.
| delusional wrote:
| I tried to get chat.openai.com to generate something
| interesting for the transcript linked to in another comment and
| it failed rather spectacularly. The transcript contains error
| for Rust (which becomes rest) and coreutils (which becomes
| corridors) making the output of the LLM useless. The LLM also
| seems to completely misinterpret the author when he speaks
| about working for Mozilla, omitting the part where he
| explicitly mentions that he's only saying this so that you wont
| associate this work with Mozilla. It also repeatedly gets
| confused about what the author is talking about, likely due to
| a small context size.
|
| It would be a very low quality blog-spam level blog post.
| arlort wrote:
| I'm curious if you could use the transcript + something like
| the title/subtitle of the talk with chatgpt to fix the
| spelling mistakes
| delusional wrote:
| Interestingly, If i give it the filename title "Rust
| Coreutils" it spots "the corridor -> coreutils", but if i
| provide the proper title of the talk "Reimplementing the
| Coreutils in a modern language (Rust) Doing old things with
| modern tools" it doesn't find any mistakes. Once again
| lending creadence to the observation that it doesn't
| actually "understand" since the proper title is quite
| clearly not markedly different (in relation to the
| transcription mistakes) from the filename title.
| CraftingLinks wrote:
| I had something more basic in mind: Just clean up syntax
| without much, if any, semantic re-interpretations.
| sluongng wrote:
| This is wonderful. I was able to search for my favorite topics by
| cloning the repo and grep through all the TXT files. Amazing!
| tuukkah wrote:
| Wow! Is there an easy way to combine these subtitles with the
| video in the browser?
| usr1106 wrote:
| Not a format I would recognize, but I am not an expert in the
| area. Two lines of awk would convert it to srt I guess.
| tuukkah wrote:
| It's even closer to WebVTT, just had to add the line "WEBVTT"
| to the top and replace all square brackets with line breaks:
| curl https://jonatron.github.io/fosdem2023whisper/files/celeb
| rating_25_years_of_open_source.webm.txt | ( echo "WEBVTT";
| sed -e 's/^\[\|\] \?/\n/g' )
| >celebrating_25_years_of_open_source.webm.vtt
|
| Plays in the browser with HTML like this:
| <video controls src="https://video.fo
| sdem.org/2023/Janson/celebrating_25_years_of_open_source.webm
| " > <track default
| srclang="en"
| src="celebrating_25_years_of_open_source.webm.vtt"
| > </video>
| cratermoon wrote:
| Is my version of sed dumb, or is 's/^\\[\|\\] \?/\n/g'
| effectively a no-op? Or did HN garble something?
| tuukkah wrote:
| I used GNU sed on Ubuntu. The only difference I notice is
| that you have one space where I had two, but that
| difference shouldn't matter actually. Perhaps your sed
| has a different opinion on which of the backslashes
| should be there.
|
| You can access the resulting WebVTT files here: https://g
| ithub.com/tuukka/fosdem2023whisper/tree/feat/webvtt...
| karencarits wrote:
| A sidenote, but adding subtitles to pure audio in HTML is
| surprisingly difficult!
| tuukkah wrote:
| Does it not work to put a track element inside an audio
| element?
| karencarits wrote:
| You can try to fake it as a video, but in my experience,
| it is difficult to properly format the element so that
| the subtitles or captions actually shows
|
| https://developer.mozilla.org/en-
| US/docs/Web/HTML/Element/au...
|
| > The <audio> element doesn't directly support WebVTT.
| You will have to find a library or framework that
| provides the capability for you, or write the code to
| display captions yourself. One option is to play your
| audio using a <video> element, which does support WebVTT.
| jonatron wrote:
| It looks pretty easy, but I'm not going to do it, if someone
| else wants to.
| funstuff007 wrote:
| The New Yorker had a recent piece on Whisper as well.
|
| https://www.newyorker.com/tech/annals-of-technology/whispers...
|
| For them, the key difference (other than it's great performance)
| was that it was a standalone executable that did not need to
| access remote resources or "share" sensitive information with
| such remote resources.
| trynewideas wrote:
| Every use of "fuck":
|
| > So, you can see my Firefox is in Italian, but you can see that
| it automatically detected that the page is in French and it is
| suggesting me to translate it to Italian. I will change it to
| English. Oh, fuck.
|
| > The reason for that is because there is so many ways to extend
| or to define an architecture, and there is some really fucked up
| things that can be done in so many architectures
|
| > We don't measure RSS because it measures anything useful we
| measure it because it's really fucking easy to measure.
|
| > There were a few customers that didn't respond, but at some
| point you just have to decide to. Don't give a fuck.
|
| > Computational storage, what the fuck's that?
|
| > like on MacOs you can have a recovery key, on Windows you can
| have a recovery key, and this is what we should do here as well,
| right, like use TPMs absolutely, and then hide entropy recovery
| keys that things are fucked up, things are fucked up
|
| > we get money, we invest in open source, and then we have the
| problem that what we do in open source could be reused by non-
| European companies. And thinking, oh, my God. Oh, that's so
| fucking wrong. Why do you do that?
|
| > So, you know, if Monty is not screaming at me saying you are
| fucking moron, Peter, that is not how it is, then probably I am
| not doing my job properly.
|
| > I'm going to wail and gnash my teeth and hope it doesn't happen
| in practice because at the moment, you know, something like EDK2,
| to be clear, by the way, I don't give a fuck about non-free
| firmwares
|
| > Is there anybody here who doesn't know what a kubelet is? Okay.
| The kubelet, oh, you fuck, anyway.
|
| > So for now we're targeting your own CA so you can just say fuck
| it, I will enroll my own keys to the firmware.
|
| > So one of the idea is to start to come to a more predictable
| fashion, which is one major API break and API break every year
| around December, January, so we're in February, and we fuck this
| year.
|
| > It's really not sure whether we are going to be able to do that
| because you're transpiling assembly. Like, what the fuck are you
| talking about?
|
| > your problem was where you called, in this case, pop on an
| empty stack, and not the completely pointless unwrap in there,
| because you want to know where you're fucked up and not here,
| obviously.
|
| > So if it is, then say, say like, this is fucking wrong, Peter,
| you know, so I can fix my slides when I talk next time, I have
| the wrong stuff, right?
| PeterZaitsev wrote:
| OOps. Busted :) At least one of those is mine
| ldubost wrote:
| Very Nice. I've use the "all transcript" to make a Word Cloud:
| https://framapiaf.org/@ldubost/109851711189159771
| carbocation wrote:
| Can you describe your setup / how you piped the audio from
| YouTube to Whisper in order to generate these transcripts?
| jonatron wrote:
| Used a small python script to download the webm files from
| https://fosdem.org/2023/schedule/events/
|
| Used a small bash script to run `whisper $f --output_format
| json --language en | tee $f.txt` over every file.
|
| Ran a small python script to make the HTML. All totally
| trivial, it just required some GPU-hours.
| grammers wrote:
| Uh, nice, thanks for sharing!
| zX41ZdbW wrote:
| It did not get confused by my heavy accent, and the result is
| surprisingly good:
|
| https://jonatron.github.io/fosdem2023whisper/files/fast_data...
|
| Original: https://www.youtube.com/watch?v=JlcI2Vfz_uk
|
| But some mistakes are hilarious!
|
| > 500 megabytes per second, Britain.
|
| Britain. Guess what it actually was :)
| jonatron wrote:
| To be fair, you did say Britain, when I think you mean B
| Written.
| ronef wrote:
| This is great! Does it include all talks? I just noticed that it
| might be missing
| https://fosdem.org/2023/schedule/event/nix_and_nixos_runix/.
| college_physics wrote:
| Wow!
|
| In terms of resource usage, for this and many other "emerging"
| opportunities, is there some thinking about pooling idle
| CPU's/GPUs for creating open and widely available models and/or
| datasets based on models?
|
| SETI type crowdsourcing of compute was always a bit niche but
| maybe now we have some use cases of very broad interest
| wslh wrote:
| I don't know how companies like Grammarly will survive this AI
| summer.
| jonatron wrote:
| Subtitles, LLM, DeepSpeech comparisons are good ideas, so say if
| you're going to do them. I'm not going to do much more.
| tuukkah wrote:
| You can git pull from here for subtitles in the WebVTT format:
| https://github.com/tuukka/fosdem2023whisper/tree/feat/webvtt
|
| (I'm not able to make a pull request because GitHub is giving
| me the pink unicorn error page when I try to fork your repo.)
| AstixAndBelix wrote:
| Doesn't Mozilla have their own speech recognition library? I
| think it would have been more interesting to see the results,
| since we're talking about FOSDEM.
|
| Also, since OpenAI was already used, it could have been cool to
| provide a summary of the transcript!
| adgjlsfhk1 wrote:
| whisper is mit licensed.
| homarp wrote:
| and run "offline" and locally, so the content transcribed is
| not sent anywhere.
|
| There is also a CPU only version discussed a few times:
| https://news.ycombinator.com/item?id=33877893
| bonzini wrote:
| I checked my own talk and it's not perfect but it's more than
| understandable, despite my having an accent, speaking fast and
| not having rehearsed very much. Together with the slides it would
| be clear what was going on.
| simonw wrote:
| I wonder how hard it would be to automatically add slides by
| scanning the talk video for points at which the on-screen
| slides change and mixing a frame from the video into the
| transcript at the relevant point.
___________________________________________________________________
(page generated 2023-02-12 23:02 UTC)