[HN Gopher] The race to teach sign language to computers
___________________________________________________________________
The race to teach sign language to computers
Author : sohkamyung
Score : 45 points
Date : 2021-03-07 12:41 UTC (2 days ago)
(HTM) web link (www.economist.com)
(TXT) w3m dump (www.economist.com)
| vitorbaptistaa wrote:
| Portuguese-only, but in Brazil we have VLibras [1], which is a
| free open-source software to automatically translate between
| written text and a 3D model using sign language.
|
| [1] https://www.vlibras.gov.br/
| elliekelly wrote:
| That's really cool. Was this funded by the government? I ask
| because it's on a .gov site.
| ars wrote:
| Can someone explain why this is necessary (important) when typing
| into a computer seems like a reasonable alternative?
|
| I'm tying to make an analog to Google home like devices, but I
| can't imagine someone putting cameras all over a room to capture
| sign language from any angle.
|
| Can someone help me understand what's the use case for computers
| understanding sign language that is not served by typing?
| toomuchredbull wrote:
| What if you had deaf friends and could wear glasses that would
| translate sign language for you? What if you were deaf and had
| a robotic assistant? What if you wanted to do translations for
| a hearing audience?
| curtisf wrote:
| Sign language is not a way to speak English using your hands --
| it is a distinct language with distinct grammar and distinct
| vocabulary.
|
| If you are born Deaf, you will not have the opportunity to
| natively acquire a spoken language. Our brains are not designed
| to acquire spoken language through writing -- and our spoken
| languages are designed to be acquired through listening, not
| reading.
|
| It's easy to think that a Deaf signer living in America is a
| native speaker of English, but they very well may not be,
| having never heard the sounds and therefore been unable to
| internalize all of the many, many necessary grammatical and
| morphological rules of the language.
|
| Expecting all Deaf people to simply use written English when
| dealing with a computer is like expecting all Americans to use
| Spanish to interact with their computers -- technically
| possible (Spanish is a perfectly good language, and relatively
| accessible starting even from grade school here), but ignores
| the human reality that people want (need) to speak their own
| language.
|
| There are not yet standard ways to write sign languages. Partly
| this is because of difficulties in translating the medium to
| text -- unlike speech, sign language production is not
| _entirely_ linear, in addition to the fact that your hands can
| be articulated comprehensibly in much more sophisticated ways
| than your mouth can.
| ars wrote:
| Does that mean people with use Sign language as their first
| language do not read books? How do they learn anything?
|
| It really seems to me you are implying they can't read
| regular English - doesn't that make regular life quite a
| challenge? I would have assumed learning written English
| would be high priority.
|
| Your answer just makes me even more confused. If they can't
| read English, what is the computer translating the sign
| language into? What are they doing with the results?
|
| And this just brings up even more questions - how do they
| speak remotely to people? I was thinking that a video phone
| was an amazing advance, except you'd have to put the phone
| several feet away from you, making it pretty hard to see the
| screen.
|
| Every single scenario I'm thinking of just makes the need for
| written language even more critical - otherwise such a person
| is almost entirely isolated, except when near other signing
| people.
| cowboysauce wrote:
| Most people who know ASL can also communicate in English,
| but language skills are a spectrum and some people may know
| enough to get by in daily life, but may not know enough
| English to understand more complicated topics. It's really
| no different than a native Spanish speaker living in the
| United States. Some may only know a basic level of English
| while others may be equally comfortable in both languages.
|
| This isn't even unique to the Deaf community. Something
| like 15 to 20 percent of the general population are
| considered functionally illiterate.
| bluefirebrand wrote:
| > Expecting all Deaf people to simply use written English
| when dealing with a computer is like expecting all Americans
| to use Spanish to interact with their computers
|
| Kind of a false equivalence here. For many Americans there is
| no real reason to learn Spanish. However, I have to imagine
| that for literally all deaf people learning to read and write
| the dominant local language is required in order to
| participate in the larger society. Signs will be written in
| that language, restaurant menus, newspapers, business emails,
| etc. You will be surrounded by writing of your local language
| every day.
|
| There doesn't really exist a world where a person can get by
| without being able communicate with others in their vicinity.
| Deaf people are more often surrounded by non-deaf people who
| they will have to communicate with somehow. Sharing a common
| spoken language is kind of out of the question, sharing a
| common written language is the next best thing.
|
| So, sure. Create a written sign language, make unicode
| support for it and create new locale settings for each
| dialect of sign language. This will not eliminate or even
| really reduce deaf people's need to learn to write their
| locally dominant language.
| dlandis wrote:
| One thing to keep in mind is that ASL, for example, is the
| first language of many people and it is a totally different
| from the English language.
| kingant wrote:
| Check out this paper: https://arxiv.org/abs/1908.10546 (not
| mine)...but links to some recent work using deep learning.
| karlkatzke wrote:
| Complete aside: How cool of a name is "Dr. Zsolt Robotka"? Pretty
| sure Dr. Robotka will also be the name of the next Bond villain.
| neonate wrote:
| https://archive.is/ex59g
| _Microft wrote:
| If you wonder why you might not be able to access this, keep in
| mind that archive.is and Cloudflare DNS do not play nicely with
| each other [0]. It might also bite you if you are using Firefox
| because of the DNS over HTTPS service from Cloudflare that
| Firefox uses [1].
|
| Loading the article directly into Firefox's reader mode by
| prepending _about:reader?url=_ to the URL works fine though.
|
| [0] https://twitter.com/archiveis/status/1018691421182791680
|
| [1] https://support.mozilla.org/en-US/kb/firefox-dns-over-https
| runnr_az wrote:
| Signal me when they've taught em semaphore.
| jfengel wrote:
| That should be way, way simpler. It sounds like a neat computer
| vision class project.
| runnr_az wrote:
| That's what I'm sayin'... duolingo for flag communication.
| TheAdamAndChe wrote:
| I can't read it, but what makes interpreting sign language so
| difficult? You can have a model that inputs two pictures of the
| person 5/60 seconds apart. If the change in movement is below a
| certain threshold(indicating a word is being displayed) then crop
| it around the hands and run it though a resnet trained to
| categorize images based off of words.
|
| Is there something I'm missing? If it's such a big deal then I
| may work on it myself.
| schwartzworld wrote:
| American Sign Language grammar is subtle and complex.
| sohkamyung wrote:
| This paragraph from the article highlights some of the
| problems.
|
| > Research led by a team at Microsoft, a big computing firm,
| and published in 2019, estimated that a typical publicly
| available corpus of a spoken language consists of around a
| billion words from as many as 1,000 different speakers. An
| equivalent data-set in a sign language might have fewer than
| 100,000 signs from just ten people. Besides large numbers, a
| good corpus also needs variety. This means conversations
| between native signers of diverse backgrounds, dialects and
| levels of fluency. Because deaf people more often have physical
| disabilities than do those with unaffected hearing,
| representing those with restricted fluency of movement is
| important.
| PeterisP wrote:
| Fragmentation also is a big issue. Not only in the sense that
| there are separate very different sign languages (American Sign
| Language has little in common with British Sign Language, they
| are not mutually comprehensible), but even for each separate
| sign language you also get different 'dialects' in different
| areas/major cities, which are mutually comprehensible but have
| lexical variation depending on the speaker's location/community
| that needs to be properly represented in your datasets in order
| to be acknowledged and understood. In some sense it seems the
| equivalent of what the fragmentation was for spoken languages
| before the printing press and age of nationalism facilitated
| language standardization.
| hmry wrote:
| Sign languages don't only use hand shape, but also hand motion,
| body pose, facial expression, mouthing words, etc. Cropping
| still images around the hands won't do you much good at all.
| medstrom wrote:
| Not to mention polysynthetic signs - I don't want to be the
| one to program that.
|
| An example sentence of mostly polysynthetic signs:
|
| "i was out driving, just enjoying the scenery, the nature and
| all the trees passing by me," (3 signs) "then suddenly" (just
| raise the eyebrows and look scared at some point in front of
| you, with your hands still on the imaginary steering wheel),
| "my car crashed into a wall (or other immobile object) (and
| the hood got all deformed)" - 1 sign.
|
| So that takes about 5 seconds to sign, but up to a minute to
| represent all the embedded information in English. But in
| practice, you'd skip a lot of the embedded information when
| translating to English, depending on what was actually
| important. So how do you make the machine know what was
| important? It depends on who you're translating for!
| codyogden wrote:
| All sign languages use much more than handshape and position.
| And even outside of facial expression grammar (ie non-manual
| communication) there is a whole host of other complexities that
| make parsing sign language difficult.
|
| Take classifiers: there are eight morphological types that
| allow users to express complex ideas without the use of any
| type of vocabulary-based utterances. You can certainly sign the
| equivalent of, "Did you see the car parked close to the tree,"
| (CAR PARK NEAR TREE SEE?) using vocabulary. Sure. It gets the
| point across, but it's not efficient, and doesn't really
| utilize sign language in a way that feels normal for users.
|
| But that same question can be posed in a more condensed format
| if you use a classifier, (#CAR 5:CL-"tree"/3:CL-"car parked
| really close" SEE?). In the second utterance, one use the
| lexicalized (fingerspelled "car"), then holds up a five hand
| with bent elbow in their non-dominant hand, use the three hand
| representing the car placing it reeeeeeally close (with
| grammatical, NMC emphasis) to the tree, and finally asking the
| receiver if they saw it.
|
| In that example alone, I'm using two types of classifiers:
| element and locative. Imagine a complex story using all eight
| classifier types with just a little vocabulary mixed in? I
| doubt AI/ML could interpret the context, characters, actions,
| events, descriptions, etc. accurately. Anyway, I hope this
| example shows there is a lot more complexity in computerized
| interpretation of signed languages well beyond
| gesture/vocabulary recognition.
|
| Note: My glossing may not be perfect. It's been a while, and
| I'm a bit rusty.
| whitten wrote:
| Is this an ASL thing or some other system?
|
| What would be a good source to get the full list of the eight
| morphological types, and how they are used?
| codyogden wrote:
| I learned this in a Linguistics of ASL course as a part of
| my degree in ASL. My understanding is that classifiers
| exist in every natural signed language and isn't exclusive
| to ASL.
|
| https://www.allenisd.org/cms/lib/TX01001197/Centricity/Doma
| i...
|
| The linked PowerPoint does a good job of explaining a brief
| overview of the different types of classifiers. (I actually
| referenced it to correct my own answer because I knew
| something felt off. lol)
| codyogden wrote:
| Correcting myself: The classifiers used in my example are
| "semantic and locative," not "element."
| medstrom wrote:
| Essentially, vocal languages are one-dimensional, thus easy to
| solve. Filter out disturbances, background noise, and train the
| machine to adapt to dialects, and you're halfway home.
|
| Sign language is at least three-dimensional, it can embed more
| information. The ASL signs for "forget" and "because" differ
| only in how you mouth them - easy enough, but sometimes you
| even skip mouthing them because there is sufficient context.
| What's worse is the location of the hands can vary depending on
| how lazy the signer is feeling, so "forget" is supposed to be
| signed from the forehead but it can also be signed from some
| empty point in front of your shoulder, off a "phantom person"'s
| imaginary forehead. You rely on the listener to understand that
| you intend for there to be a phantom person there.
|
| Parsing sign language means you have to turn a 2D recording
| into a 3D model - even humans can have difficulty with this,
| parsing signs over Zoom - and you need to solve parsing facial
| expressions, reading lips and more. _And_ you need the machine
| to be smart enough to detect the assignment of phantom persons
| to various locations around the signer. That 's my impression
| as a deaf person anyway.
| kleiba wrote:
| _Essentially, vocal languages are one-dimensional, thus easy
| to solve._
|
| cough.
___________________________________________________________________
(page generated 2021-03-09 23:03 UTC)