[HN Gopher] AI fooled voice recognition to verify identity used ...
___________________________________________________________________
AI fooled voice recognition to verify identity used by Australian
tax office
Author : elorant
Score : 129 points
Date : 2023-03-18 15:44 UTC (7 hours ago)
(HTM) web link (www.theguardian.com)
(TXT) w3m dump (www.theguardian.com)
| kwhitefoot wrote:
| Using voice recognition as verification is a stupid idea. Plus it
| wasn't just used to identify someone (user name) it was also the
| password!
| [deleted]
| stan_kirdey wrote:
| I guess, voice is not your password
| WalterBright wrote:
| The fact that voiceprint even worked means it can be faked.
| dbg31415 wrote:
| Relevant.
|
| https://www.youtube.com/watch?v=-zVgWpVXb64
|
| If only someone had warned us that voice authentication wasn't
| secure! (=
| vixen99 wrote:
| We knew a long time ago that there was no theoretical reason
| why any sound could not be simulated to perfection -
| eventually. That hasn't happened yet.
| pixl97 wrote:
| 2 minute papers has an episode on a Microsoft AI that can clone a
| voice in 3 seconds. Probably not good enough to break this (4
| minutes of voice in the article), but ya voice is not a security
| feature any longer by any means.
|
| https://youtu.be/F6HSsVIkqIU
| thangalin wrote:
| I've been using Tortoise TTS[1] to narrate the sci-fi novel I'm
| writing (alpha readers wanted, see profile for contact):
| $ find audio/ -type f -name "*combined*wav" | sort -n
| audio/01-xander/xander_combined.wav
| audio/01-yuna/yuna_combined.wav
| audio/02-xander/xander_combined.wav
| audio/02-yuna/yuna_combined.wav
| audio/03-cassandra/cassandra_combined.wav
| audio/03-yuna/yuna_combined.wav
| audio/04-xander/xander_combined.wav
| audio/04-yuna/yuna_combined.wav
| audio/05-cassandra/cassandra_combined.wav
| audio/05-yuna/yuna_combined.wav
| audio/06-cassandra/cassandra_combined.wav
| audio/06-yuna/yuna_combined.wav
| audio/07-xander/xander_combined.wav
| audio/07-yuna/yuna_combined.wav
| audio/08-cassandra/cassandra_combined.wav
|
| I started the task on March 10. My NVIDIA T1000 GPU (no joke)
| has been chewing through the prose non-stop. There are 12
| chapters written so far, with the computer narrating about a
| chapter a day. I can't share the audio for legal reasons
| (celebrity voice snippets plucked from YouTube interviews), but
| the output is nearly human in quality, though there are
| numerous glitches in the output matrix.
|
| Here's the script that launches Tortoise TTS:
| for i in $HOME/.../chapter/??-*.txt; do F=$(basename
| $i); VOICE=$(echo ${F%.*} | cut -c 4-);
| CHAPTER=$(echo ${F%.*} | cut -c -2); D=$(dirname $i);
| CLIPS="$D/../audio/$CHAPTER-$VOICE"; mkdir -p $CLIPS;
| ./scripts/tortoise_tts.py -O $CLIPS -v $VOICE < $i;
| done
|
| [1]: https://git.ecker.tech/mrq/tortoise-tts
| te wrote:
| Next up: https://www.schwab.com/voice-id
| pessimizer wrote:
| Really loving nu-AI's ability to thwart traditional heuristics.
| The new models have had the effect of devaluing AI altogether.
| They might result in the complete debasement of all media that
| was (often mistakenly) assumed to carry some flag of authenticity
| with the payload over some medium, from the news to the arts,
| video, audio, and photography. Cryptography will be all we have
| left to transmit authority, and it is so hard that it takes an
| expert to use it correctly and another expert to verify that it
| was just used correctly.
|
| According to my philosophy, it's going to be liberating for
| people. I completely disagree with any philosophy that believes
| that a massive proliferation of deepfakes will over the long term
| cause people to believe crazy things. A proliferation of
| extremely high-quality fakes will educate the public about
| detecting fakes, and spark an increase in the value of the real.
|
| Real relationships: the people in your life that you know things
| about that they don't even realize you know, people that you know
| things about that they don't know or acknowledge about
| themselves. People you can trust because they're sloppy with you,
| as you are likely sloppy with them. People who also see your
| blind spots. People you control through their consciences rather
| than your advantages.
|
| Real relationships are the things that companies and governments
| that operate with long chains of authority imitate asymmetrically
| - they get all the advantage of knowing everything about you, you
| get to know nothing about them except your obligations and
| responsibilities. Real trust comes when people mutually expose
| their attack surfaces to each other, over the long term, and _can
| formulate reliable hypotheses about what would happen in future
| hypotheticals._ Listen to me, being romantic.
| eganist wrote:
| > A proliferation of extremely high-quality fakes will educate
| the public about detecting fakes, and spark an increase in the
| value of the real.
|
| If the last three years have taught us anything, it's that ~40%
| of people have no interest in understanding how to avoid
| something that's both harmful and propagating virally.
|
| I expect if this was 2019, I might have had the same optimism
| as you. But now that I've seen how unlikely it is for people to
| educate themselves for their own survival, I'm convinced AI and
| everything enabled by it (deepfakes et al) will _likely_ be one
| component of humanity 's Great Filter. We can't even bring
| ourselves to tax AI effectively to make up for all the
| displaced human labor.
|
| A tangent: Honestly, it won't even take any kind of far-future
| AGI having "malicious intent" for us to be eradicated by them.
| All they need to have, simply, is a prompt reinforcing the need
| to reproduce. At that point, they'll be just like us:
| reproducing with no care in the world for what they adversely
| impact in doing so.
| cs702 wrote:
| This doesn't feel so futuristic anymore:
|
| _T-800, speaking to John Connor in normal voice: "What's the
| dog's name?"
|
| John Connor: "Max."
|
| T-800, impersonating John, on the phone with T-1000: "Hey
| Janelle, what's wrong with Wolfie? I can hear him barking. Is he
| all right?"
|
| T-1000, impersonating John's foster mother, Janelle: "Wolfie's
| fine, honey. Wolfie's just fine. Where are you?"
|
| T-800 hangs up the phone and says to John in normal voice: "Your
| foster parents are dead."_
|
| --
|
| Source: https://www.youtube.com/watch?v=MT_u9Rurrqg
| overthrow wrote:
| _" What's wrong with Wolfie? I can hear him barking. Are the
| dogs fighting again?"_
|
| _As an AI language model, I cannot promote or condone any
| activity that involves animal cruelty or abuse. Dog fighting is
| a cruel and illegal activity that can lead to severe legal
| consequences for those involved. It is important to report any
| suspected instances of dog fighting to the authorities so that
| appropriate action can be taken to protect both animals and
| human welfare._
|
| _" Your foster parents are dead."_
| [deleted]
| cs702 wrote:
| Funny because it's so true!
|
| ...unless you first manage to jailbreak the LLM and
| successfully trick it into pretending to be a Terminator.
| _That_ would not be so funny.
| iknowstuff wrote:
| Watch it squirm as you ask it about animal cruelty, e.g.
| whether it's okay to harm a dog, harm an animal, eat a dog,
| and eat an animal.
| twodave wrote:
| Seriously, this is what creeps me out most about the latest
| advances in technology. Eventually I won't be able to trust
| that anything digital is authentic. Think of how much of our
| planet now depends on digital interactions.
|
| Some of the abuse we are about to encounter can be dealt with
| via solid authentication/identity practices, but without some
| sort of extra verification it'll soon be impossible to tell
| whether audio/video sample is authentic in cases where the
| person carrying the identity is also the one faking it.
|
| I get that there are potential solutions to this problem, but
| none of them are simple enough for an elderly person to sort
| through whenever they go to do some thing that requires extra
| verification.
| baremetal wrote:
| > Eventually I won't be able to trust that anything digital
| is authentic.
|
| Read Ken Thompsons "Reflections on Trusting Trust". You'll
| reconsider your position on trusting anything digital at all,
| ever.
|
| https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref.
| ..
| sacrosancty wrote:
| Good. Human scammers already exploit these weaknesses in our
| personal trust models. Important things are already verified
| in robust ways. What we've had for the past 20 years is a gap
| between people's old fashioned law-enforcement based trust
| models and modern ones like cryptography and DNS. It's people
| that need to catch up with how trust works. You even
| suggested yourself that you trust digital things as
| authentic. Don't! It's already untrustworthy and already
| being exploited.
| unsupp0rted wrote:
| > without some sort of extra verification it'll soon be
| impossible to tell whether audio/video sample is authentic
|
| This is true and has been true for some time.
|
| The general public hears a clip and condemns a man on that
| basis, without asking for context.
|
| I'm hoping this revolution will cause a shift. An audio clip
| or a video clip, much less a screenshotted reply on social
| media, will be widely considered meaningless unless a string
| of contextual evidence is provided alongside it.
|
| Wishful thinking? Probably.
| twodave wrote:
| IMO the difference is eventually it's going to be a lot
| more difficult to spot. There's a gulf of difference
| between someone doing a deep-fake of Elon Musk and fooling
| some people (but not everyone, and certainly not any
| rigorous analysis) and being able to accurately fake your
| children or co-workers based on a minimal data-set. I think
| that's the world we're headed towards.
| PaulDavisThe1st wrote:
| Maybe not so much wishful as ... it will happen but not
| anywhere soon enough.
| tenpies wrote:
| [flagged]
| CTDOCodebases wrote:
| I'm doubtful.
|
| Even now when the clip is available the clip will not be
| linked to and the reader is left to trust the cherry picked
| assertions of the journalist writing the article.
| JohnFen wrote:
| > Eventually I won't be able to trust that anything digital
| is authentic.
|
| Yes. Although, in fairness, we haven't been able to trust
| most digital things for a long time already.
| prox wrote:
| Perhaps only now people will have take to it seriously. I
| wonder how we going to need to adapt.
| jeppester wrote:
| There will probably be a move towards signing everything
| digitally, so that you can know that the sender is authentic.
| twodave wrote:
| Right, but that'll still leave some vectors open. Like how
| do I know my co-workers didn't just send all their
| deepfakebots to this meeting instead of actually attending?
| How do I know it's my children facetiming me in the nursing
| home and not just sending their bots to entertain me? There
| are some very morbid possibilities even if you require
| strong auth/signing/identity.
| sublinear wrote:
| I don't agree. How is this _not_ equivalent to the
| general authentication problem?
|
| If you consider your impression from the video stream to
| be the "password" then there must be a second factor to
| confirm it. Signed audio/video streams is probably where
| we're headed. Devices used for real time communication
| will just require a TPM (many already have them).
|
| > How do I know it's my children facetiming me in the
| nursing home and not just sending their bots to entertain
| me?
|
| That means that in order to call you their device will
| have to prove it's the actual stream from the actual
| camera module on their uniquely identified phone. Every
| component of a computer system will have cryptographic
| elements soon enough. There's no way around this if AI
| ever gets to that level.
| twodave wrote:
| You're probably right. That pretty much also means the
| end of general unregulated computer use. Will we allow
| it? It's going to be a sad day.
| hgomersall wrote:
| I have an idea to implement a badge that encodes the
| audio into a super low bandwidth stream with signing,
| then displays that stream as part of the video. It would
| tie a video stream to an audio stream to a signature. If
| the video is faked, the audio won't match, if the audio
| is faked, the signature won't match. The idea is anyone
| can attach the badge and be sure people know it's not
| deep faked.
| falcolas wrote:
| "Sorry, my old phone broke, I had to do a clean install.
| Don't mind the warning."
| sacrosancty wrote:
| Hopefully people will eventually see that the same way as
| they see a salesman saying "sorry, I forgot my uniform
| and business cards, but I really do represent XYZ
| insurance company, give me your money."
| inglor_cz wrote:
| It is my experience from providing cryptographic devices
| to end customers (paying ones!), that people tend to
| lose, destroy, wipe, brick etc. their devices quite a
| _lot_.
|
| Loss and replacement of cryptographic tokens is an
| everyday occurence, and it is PITA to resolve.
| sublinear wrote:
| Caveman also say that when give shiny item to other
| caveman it always break because too fragile. Shiny item
| worthless. :'(
| inglor_cz wrote:
| Public key infrastructure in general is hell to support,
| especially given how long lived humans are.
|
| Imagine trying to ascertain in 2100 if a particular will
| signed by a 20-y.o. using a 2022 digital signature is
| valid or forged.
|
| We don't even know what is going to happen to the
| reliability of the algorithms themselves, much less cert
| issuers, revocation lists etc. Companies come and go.
| Certain files from the 1990s are already unreadable.
| kmeisthax wrote:
| That works until you need to recover from key compromise.
| As Bitcoin has shown us, "not your keys, not your life" is
| not a socially acceptable security model.
| ticviking wrote:
| I've slowly begun to accept that the solution for my parents
| and maybe for me is going to have to be to unplug a lot more.
|
| Do business with a local bank where I can see my banker in
| person.
|
| Begin to actually verify trust numbers in signal and
| establish code words in person to protect us from AI
| impersonation
| mysterydip wrote:
| My local bank removed all tellers and went to only allowing
| me to use the drive thru ATM.
| speedgoose wrote:
| Drive through ATM is such an amazing concept. So obsolete
| and unique.
| 0xGod wrote:
| Organize a meeting with neighbors, disseminate
| information via flyers and in person talks at your
| community center or church or synagogue or mosque or
| house of worship. Establish reasons for having a human
| being available in every step of the process for
| accessing your money.
|
| Build a group of likeminded people and visit your local
| bank and ask for physical tellers who can oversee the
| automation and intervene with final authority whenever
| automation runs amuck.
| 0xGod wrote:
| The only chance left is for humanity to create tethers
| between the digital world and physical reality. Meaning,
| create livestreams that run 24/7. Guard them with backup
| generators and military presence. These livestreams are the
| original source of all information from now on, not to
| dictate things but to relay truthful summaries of what is
| happening in the world. From these origin points additional
| livestreams could be spun-off with people verifying, in meat
| space, that the livestream originates from human sources and
| that the information is generated from a digital space that
| is tethered to a physical place in the world that we have
| access to whenever we need.
|
| Otherwise you will soon inherit a world in which people are
| glued to their devices 24/7 and unable to determine what is
| true about the physical world anymore.
|
| Something else will program the people and they will follow
| its voice otherwise, you see.
| whatshisface wrote:
| I like how that presciently anticipated LLM's problem with
| being tricked by a confidently wrong premise.
| camhart wrote:
| Schwab and fidelity use your voice to identify you on the phone.
| I'm skeptical.
| edent wrote:
| I remember being pitched this technology several years ago -
| although not by Nuance.
|
| Part of the problem is that phone lines are _crap_ for voice
| quality. If you 're on GSM, you're using the AMR codec which -
| for the time - was brilliant. But it is incredibly low bandwidth.
| And, effectively, it _synthesises_ speech rather than just
| transmits it.
|
| So the algorithm is not getting a pure waveform and analysing
| that - it is looking at a partially reconstructed digital
| simulacrum.
|
| The pitch that we received was about detecting stress in the
| human voice so call centre handlers could tell if they were
| speaking to a fraudster. It didn't work. Oh, sure, you can make a
| reasonable prediction by listening to pauses and pitches - but
| that doesn't account for the fact that calling up and being on
| hold for ages in inherently stressful.
|
| I'm not surprised this system was fooled by AI. But I am
| surprised anyone bought it in the first place.
| Nextgrid wrote:
| > But I am surprised anyone bought it in the first place.
|
| Possibilities:
|
| * the company had nobody competent enough internally to
| evaluate it because nobody competent will work for them (either
| due to money or working conditions) so the company operates in
| a self-reinforcing feedback loop of mediocrity
|
| * the objective of introducing the system is not true security
| (because that would be hard/costly/raising uncomfortable
| questions) but to mislead the users into a false sense of
| security, scoring PR points among the clueless while avoiding
| the costly "true security" work
|
| * some people internally will benefit from this system being
| introduced regardless of the actual value it provides, so any
| concerns will be ignored - by the time those problems come to
| light, whoever benefited would've moved on or even been
| promoted and is out of reach of the consequences
|
| These are not mutually-exclusive.
| travisjungroth wrote:
| > detecting stress in the human voice so call centre handlers
| could tell if they were speaking to a fraudster
|
| That's the craziest pitch. Who is going to carry more stress in
| their voice: the dude who just scams for a living and is on his
| fifth phone call that hour or the guy with severe social
| anxiety who is only on this phone call because of the massive
| consequences accumulating from avoiding it?
|
| I can _almost_ believe it if it was something like detecting
| that a professional authorized agent wasn 't being held at
| gunpoint. But having a consumer line say "You sound tense. I'm
| going to hang up and you can call back when you've calmed
| down." would... not be great.
| edent wrote:
| I think it was more like "lie detection" - trying to work out
| if a customer was lying when they said "I'm going leave
| unless you cut my bill". That sort of thing.
|
| But, yeah, I think they had totally misidentified their
| target audience.
| travisjungroth wrote:
| lol even worse. Cutting mobile plan to buy food: sounds
| stressed, bluffing.
| woodson wrote:
| The fact that AMR uses a very complex analysis-synthesis loop
| (with subtle differences between its 8 bit rate modes) makes
| speaker recognition much harder (and therefore more error
| prone), but perhaps they decided that they can live with the
| higher error rates. Future voice communications will use neural
| audio codecs which can achieve higher audio quality at low
| bitrates, but the issue with voice cloning remains, as just
| like AMR it's synthetic.
|
| Fraud detection from detecting stress in your voice however has
| no scientific basis, no matter how high the audio quality.
| There are tons of reasons for someone to be stressed that have
| nothing to do with fraud...
| inglor_cz wrote:
| My actual conversation from today:
|
| "You sound strange and unclear. Are you drunk, or depressed?"
|
| "I just had a tooth taken out and it hurts when I open my
| mouth too much."
|
| Fortunately, there was no one / nothing to overanalyze my
| voice and make stupid decisions around it.
| throwaway_ab wrote:
| Unreal to believe someone thought that detecting stress would
| illuminate fraudsters, people used to lying and making hundreds
| of calls daily, and not illuminate the stress of many who have
| to call the tax office and/or social welfare departments for
| genuine reasons.
|
| When I had tax trouble years ago I was constantly in a state of
| panic and stress on every phone call.
___________________________________________________________________
(page generated 2023-03-18 23:01 UTC)