[HN Gopher] Open-source tool translates and dubs videos into oth...
___________________________________________________________________
Open-source tool translates and dubs videos into other languages
using AI
Author : oldcai
Score : 99 points
Date : 2024-08-13 12:15 UTC (10 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| vouaobrasil wrote:
| Unlike other methods of automation, AI is replacing human beings
| too fast. And before you say, "new jobs will be created" -- look
| at history. After the computer, new jobs have been created, but
| what kind of jobs? Every year, we are becoming more entwined in
| wage slavery as the wealth accumulates at the top and jobs become
| more meaningless.
|
| So, no, new jobs will not be created, except the kind of jobs
| that crush the human spirit into oblivion so that the rich tech
| oligarchs can play God.
| nickthegreek wrote:
| >AI is replacing human beings too fast.
|
| citation needed.
| RIMR wrote:
| This is a subjective statement. If you disagree with it,
| counter it. Asking for a citation doesn't make sense here.
| chompychop wrote:
| How is it subjective? Can't terms like "replacing" and
| "fast" be quantified by metrics related to rates of
| unemployment and adoption of AI systems for tasks
| previously manned by humans? I'm not saying the data is
| readily available, but I do see a route to objectively
| measuring this.
| bsaul wrote:
| << It's too sweet >> falls into the same measurable
| phenomenon, and yet remains highly subjective
| homarp wrote:
| can't I with that creates video that have the world as
| potential users?
|
| and the video can be the products, or tutoriala for another
| products. This allows me to do more, not less.
| Kiro wrote:
| Strange submission to post this comment on. It's not like
| translating and dubbing videos is the highest form of labor.
| wongarsu wrote:
| It's not like it's mindless drudgery either. Losing as little
| as possible in translation while fitting within the
| constraints of subtitles (people can only read so fast) can
| be very challenging. And decent dubbing tries to
| approximately match the mouth movements, which puts further
| constraints on the translation and often requires creative
| text changes.
|
| There is a point about industry size (not that many countries
| dub all their movies), but it is one of the intellectually
| more challenging professions.
| Kiro wrote:
| Translation work is often associated with horrible working
| conditions and poor pay. It definitely fits OP's
| description of "jobs that crush the human spirit into
| oblivion." This might not be the case if you're a renowned
| translator of famous literature, but that is not
| representative of the majority of translators.
| insane_dreamer wrote:
| It actually requires a fair bit of skill to do it correctly
| and accurately
| surfingdino wrote:
| There is finesse to it. You are translating from one culture
| to another.
| notkaiho wrote:
| Have you tried? It's a genuine "skilled labour" job, both the
| translation and the voice acting.
| 42lux wrote:
| In which country? Germany and France put some effort in and
| it's still not great most of the time but most languages
| dub only in 1 or 2 voices for all characters. Just look at
| the ridiculous long credits of Netflix shows.
| evandrofisico wrote:
| Brazil has a long history of dubbing, not only
| translating but also localizing tv shows and movies. In
| some cases the shows are actually better dubbed, because
| the voice actors are better at emoting than the original
| cast.
| pizza234 wrote:
| Italian voice actors are extremely good (although I
| personally don't like dubbing at all).
| blacksmith_tb wrote:
| Anime dubs from Japanese into English are often quite
| good, even idiomatic (I am sure there's some license
| taken, say for something like <<Kill La Kill>>, but the
| overall result is much better for it).
| pizza234 wrote:
| Voice acting (dubbing) _is_ acting, which is a "high form of
| labor". In some languages/countries, most of the dubbing is
| performed by a few extremely good voice actors.
| ZoomerCretin wrote:
| Watch Squidgame's English dubs and tell me that couldn't be
| improved.
|
| If for no other reason, this tool is a net-positive because
| it may raise the bar on the quality of dubs expected from
| dubbing actors.
| joe_the_user wrote:
| LLMs aren't going replace all actors or all voice actors
| just as they won't replace all illustrators or writers.
|
| LLMs provide a certain level of mid/low quality content in
| nearly all mediums. And given that there many people
| producing such mid/low quality content today, LLMs will
| have an impact. LLMs affecting sales writers? Sure (not the
| best sales writers but the point is sales is mediocre but
| acceptable is a norm).
|
| And say LLMs specifically. There's good evidence the
| technology has roughly peaked. That doesn't mean it's
| impact has peaked but it's indication that "all jobs at
| risk" might be an exaggeration.
| codedokode wrote:
| It is a form of art. I hate watching dubbed videos, because
| the voice actors who translate it almost always are worse
| than original actors.
| insane_dreamer wrote:
| The problem is that we are hurtling towards the unknown without
| a plan, driven by the "need" to make higher returns for
| shareholders and to capture the new market. It may be that some
| new types of employment magically appear that soak up the jobs
| lost, but you can be sure there is no one working on solving
| that problem since the goal is to eliminate labor not create
| it.
| numpad0 wrote:
| I don't think generative AI is replacing humans at all. It's
| like how SQL replaced software engineers, with added bonus of
| copyright doubts gatekeeping common folks from exploiting it.
| It's obviously killing open Internet fast and encouraging power
| concentration too. It's worst of couples of worlds.
| IndySun wrote:
| >...new jobs will not be created, except the kind of jobs that
| crush the human spirit into oblivion...
|
| AI certainly means everyone will be able to create 'art' and as
| a result we'll have more art than we know what to do with,
| music and images are already confetti, soon so will full length
| 'films/movies'. That leaves anyone who can actually sing,
| paint, play, dance, in prime position to take up those mantles.
| amonith wrote:
| What do you mean by the last sentence? Isn't it the opposite?
| People who could do art stuff won't be able to sustain
| themselves using those skills anymore because those will be
| devalued a lot.
| pizza234 wrote:
| This is a fallacy called "Lump of labour fallacy" (see
| https://en.wikipedia.org/wiki/Lump_of_labour_fallacy).
| ZoomerCretin wrote:
| Calling something a fallacy does not mean it truly is one. I
| have not seen strong evidence that there is not a "lump of
| labor" in extremely high-rent countries like the US.
|
| For a sufficiently small period of time, there absolutely is
| a "lump of labor". If I were to go into a county with robots
| and rent them out to employers for 1/4 of the wage of their
| current employees, and they all fired their employees and
| accepted my robots, do you really believe that all of ex-
| employees would be able to find work again within 6 months?
| Or even 70% of them? What about their new wages? Do you think
| these new jobs would pay as well? I have a hard time
| believing that to be the case.
| cubbic wrote:
| Oh I made something similar but for Netflix!
|
| https://github.com/cubbK/dubbing_ai_netflix_client
|
| I want to learn swedish and because there are so few dubbed
| movies in Swedish I take the subtitles(Netflix is good at having
| subtitles in different languages) and text-to-speech it :)
| sam_perez wrote:
| How would you rate the quality of the dubs?
|
| Do you think it's usable for learning? Seems like you could end
| up with some quirky learnings.
| sunnybeetroot wrote:
| This is fantastic! Any direction on if I wanted to change the
| language to something else?
| ranger_danger wrote:
| An English README would have been nice
| pcarion wrote:
| there is one:
|
| https://github.com/jianchang512/pyvideotrans/blob/main/docs/...
| microflash wrote:
| Probably you missed it:
| https://github.com/jianchang512/pyvideotrans/blob/main/docs/...
| netsharc wrote:
| Huh, if it's good enough, movies/TV shows dubbed with AI-clone of
| the original voice would be great (if we can ignore the ethics of
| using the actor's voice and the loss of work for the dubbing
| companies and actors).
|
| For example here's how weird Friends is in German:
| https://www.youtube.com/watch?v=nCoNSZV--z0 . Or Italian:
| https://www.youtube.com/watch?v=wO5qTzvyQ1s
|
| Can AI detect the emotional tone of sentences yet, and recreate
| it in the target language?
| programjames wrote:
| Yes, pop in a normalizing flow that removes tone and recreates
| it with a small audio sample as context.
|
| https://arxiv.org/abs/2312.01479
| devindotcom wrote:
| they're workin on it
| yieldcrv wrote:
| Yes, have you used OpenAI's voice model? It uses and reacts to
| tones
|
| My favorite conversation has been getting it to tell me about
| marshmallow vs marshmellow spelling and pronunciation, it
| became very strict but patient with me
|
| It can reply in other languages too, but I cant detect dialect
| as well to say
| randomgiy3142 wrote:
| Because translations are copyrighted so it is complex to get
| legal rights for them.
| CyberDildonics wrote:
| Is this using an open source text to speech model or is it going
| out to some other internet service?
| tourmalinetaco wrote:
| It can use Whisper which is open source, although optionally it
| can also use GoogleSpeech.
| CyberDildonics wrote:
| I took a look and that seems to be speech to text / speech
| recognition but not text to speech.
| lossolo wrote:
| Based on english doc it seems it's not dubbing but voice over.
| LeoPanthera wrote:
| For years I've wanted this for live TV. Even just subtitles would
| be amazing. I've always wanted to be able to watch news TV from
| other countries.
| alphabetatheta wrote:
| Some of my friends have been using: https://app.vivalabs.ai/ as a
| more managed/paid service recently
| cyberax wrote:
| I would pay a lot for a tool that removes the freaking laugh
| track from videos.
|
| I just physically can't watch them. I wanted to watch the
| Blackadder series, but I couldn't even get through one episode.
| mrtksn wrote:
| Back in high school, when I got my first PC a plumber came over
| to fix some stuff and when he saw the computer he got excited and
| asked some questions and one of the questions was "how do you
| translate the VCD with this, I have a movie to watch but hate
| subtitles".
|
| I was like "silly dude doesn't know how computers work" but maybe
| I was the silly one who can't dare to imagine how something like
| that can work.
| codedokode wrote:
| As I understand, it first extracts text from original video into
| subtitles, translates them using external LLM, and then converts
| text to speech. All of this is done using thrid-party solutions,
| and the project seems to be just a GUI app that allows to
| integrate them.
|
| You obviously cannot use this to translate songs or movies
| because this method loses important information like voice,
| intonation, etc.
|
| So it is still better to use subtitles.
___________________________________________________________________
(page generated 2024-08-13 23:00 UTC)