[HN Gopher] Patent for attention-based sequence transduction neu...
___________________________________________________________________
Patent for attention-based sequence transduction neural networks
(2019)
Author : ukuina
Score : 94 points
Date : 2023-05-09 17:24 UTC (5 hours ago)
(HTM) web link (patents.google.com)
(TXT) w3m dump (patents.google.com)
| ilrwbwrkhv wrote:
| Google might go down as one of the greatest business failures in
| history.
| kernal wrote:
| The vitriol you must have towards Google must consume you.
| [deleted]
| AnimalMuppet wrote:
| A lot of us wish we had businesses that were half the failure
| that Google is.
| qeternity wrote:
| I'd settle for 0.1%
| ilrwbwrkhv wrote:
| Not me. I would choose my business a hundred times over
| Google.
| sebzim4500 wrote:
| Do you own one of the three companies in the world worth
| more than Google is? If not, then you would probably be
| making a stupid choice.
| dkfljvslkj2 wrote:
| [flagged]
| iamgopal wrote:
| If one would have opportunity, who should be the CEO of the
| Google ?
| WanderPanda wrote:
| Elon, releasing 150k people from their golden handcuffs
| [deleted]
| nerdo wrote:
| Marissa Meyer
| speedgoose wrote:
| I suggest myself. I wouldn't mind a few hours of the CEO
| salary and the line on my CV.
| phkahler wrote:
| Oh, can we get in line for this? I might even try to make
| an improvement in my hours.
| neximo64 wrote:
| For a company that has
|
| - cornered the market on AI researchers
|
| - has the most researchers
|
| - the best researchers
|
| - has a head start on the GPU issue with TPUs
|
| - has patents on the models everyone else uses
|
| - already has distribution
|
| - developed the models everyone else uses
|
| - has models years ahead of everyone else (e.g video ones)
|
| They sure are likely to be a filure, they're just late to the
| party. Keep in mind Apple is always late to the party.
|
| While I would agree with you the CEO of Google doesn't know
| what he's doing, Google has the best of everything to succeed
| in the race. It is incredible they are doing this all openly
| and sharing their research with everyone while doing it.
|
| The complaint with OpenAI is they are going too fast (the 6
| months letter..) with Google they were going at a responsible
| speed - which is where in a game theory fight with someone else
| going faster, would leave them in the position they are today
| in terms of perception, but not the loser.
| karmasimida wrote:
| I don't find this too be very controversial, Google sponsored the
| invention of Transformer, they deserve to enjoy the benefits for
| a period of time.
| amelius wrote:
| But first they got everybody addicted to it ... including
| OpenAI.
| dudeinjapan wrote:
| Attention is all you need. And a gigantic army of lawyers. But
| mostly attention.
| jahewson wrote:
| Attorneys are all you need.
| happytiger wrote:
| Can't wait to see the patent system break under AI. It's time for
| patents to go away or at least lower the timeframes of protection
| so we don't stifle the innovation waves that are inevitably
| coming.
|
| Whole areas will be off limits and dominated by a few companies
| -- the patent system is a system for the olden times.
|
| The patent system unfairly enforced incumbent advantage and needs
| to be reformed.
| kelseyfrog wrote:
| Even just making patents un-assignable would be an improvement.
| cynicalsecurity wrote:
| Besides, both China and Russia give zero shit about Western
| patents.
| thewataccount wrote:
| I'm very curious to see if patents/copyright/IP just become
| "washable" with AI making them near useless.
|
| Given the recent determinations, it almost sounds like since it
| won't be human made, it might not be protectable? Not to
| mention the exceptions generally made for modifications to the
| IP to improve it.
| qeternity wrote:
| People are going to work around this. Midjourney generates an
| image, but then I change the opacity 0.01% and so it becomes
| copyrightable (or whatever the smallest amount of human
| modification is required).
| thewataccount wrote:
| > but then I change the opacity 0.01% and so it becomes
| copyrightable
|
| AFAIK Courts generally have determined it must be
| "transformative" for fair use, and for patents - but like a
| substantive change. You can't just add a pixel in a corner
| and call it good.
|
| https://www.justia.com/intellectual-
| property/patents/types-o...
|
| Slightly different (was for getting a patent for ai work) -
| https://cdn.arstechnica.net/wp-content/uploads/2023/02/AI-
| CO...
| orhmeh09 wrote:
| Do you mean like, "Here is an algorithm for attention-based
| sequence transduction neural networks. Come up with an
| algorithm that is as similar as possible and produces the
| same outputs, but is unencumbered by patents"?
| thewataccount wrote:
| Essentially yes. You can already do it very effectively for
| code.
| kazinator wrote:
| Code is a specific expression covered by copyright, not
| an incredibly broad claim that covers everything from
| toasters to spacecraft.
|
| If the code's algorithm is patented, you cannot get
| around the patent by using different function and
| variable names and perturbing the code organization.
| orhmeh09 wrote:
| Neat. Is there any way to thwart this? Edit: I suppose no
| more than you could prevent a human from trying to do the
| same.
| morkalork wrote:
| Sure, stop issuing patents for software!
| efficientsticks wrote:
| "Then, automatically file a patent for it"
| sp332 wrote:
| This only works for copyright. Patents don't protect parallel
| inventions. And trademarks cover anything that could confuse
| a potential customer, even unintentionally.
| anticensor wrote:
| Trademarks are field specific, unlike copyrights and
| patents.
| sp332 wrote:
| Right. If your use of the trademarked name or logo is not
| infringing, you can just copy and paste it. No laundering
| necessary.
| thewataccount wrote:
| Yeah definitely doesn't cover trademarks as they are for
| something else like you said.
|
| To my understanding patents do allow for either
| "improvements" or anything that achieves the same result as
| long as it's not the same solution as the patent?
|
| They would certainly be a lot trickier then copywrite/IP
| but I think LLM's would still be able to generate possible
| solutions? One thing I'm thinking of for example is
| medication analogs - there's common substitutes you can
| make that achieve the same or better results that you can
| make.
|
| To my understanding redbull actually did this (without ai)
| to modafinil with this patent -
| https://patents.google.com/patent/US20210380545A1/en
|
| EDIT: Modafinil might have expired but it looks like
| redbull filed their patent before the expiration.
| sp332 wrote:
| When arithmetic coding was patented, lots of projects
| avoided using it. There was no workaround that got the
| same results without infringing.
| pclmulqdq wrote:
| I'm not so sure it's going to break. Google may have a hard
| time enforcing this one thanks to _Alice_ , but we will see if
| they try. Given that there has been no lawsuit against OpenAI,
| I'm guessing that they aren't planning to.
| tejus_vignesh wrote:
| Can someone explain how this will affect any open-source or
| transformers-based model?
| karmasimida wrote:
| It means if Google sued (not in all cases), they can block your
| service from using Transformer model.
| sebzim4500 wrote:
| Looking over the patent, I think they would really struggle
| to enforce this against someone using a decoder only
| transformer. Which is basically everyone at this point.
| ttul wrote:
| Yes, but patent US11556786B2 would perhaps give them that
| ability.
| mirekrusin wrote:
| The party is over.
| ttul wrote:
| As the lawyer behind the one-click patent once personally
| explained to me, it's the claims that matter most. And the first
| claim matters more than all of the other sub-claims.
|
| This patent specifically covers ONLY transformers in which there
| is an encoder and a decoder.
|
| Claim 1 of the patent contains the following:
|
| "...the sequence transduction neural network comprising: an
| encoder neural network configured to receive the input sequence
| and generate a respective encoded representation of each of the
| network inputs in the input sequence... and a decoder neural
| network configured to receive the encoded representations and
| generate the output sequence."
|
| Claims 29 and 30 (the only other independent claims) also specify
| an encoder and a decoder. So long as your transformer network
| does not make use of an encoder in combination with a decoder,
| this patent does not apply to you.
| ttul wrote:
| Of course, in patent US11556786B2, Google finally mopped up the
| decoder-only idea.
| https://patents.google.com/patent/US11556786B2/en
|
| "1. A method of generating an output sequence comprising a
| plurality of output tokens from an input sequence comprising a
| plurality of input tokens, the method comprising, at each of a
| plurality of generation time steps: generating a combined
| sequence for the generation time step that includes the input
| sequence followed by the output tokens that have already been
| generated as of the generation time step; processing the
| combined sequence using a self-attention decoder neural
| network, wherein the self-attention decoder neural network
| comprises a plurality of neural network layers that include a
| plurality of masked self-attention neural network layers, and
| wherein the self-attention decoder neural network is configured
| to process the combined sequence through the plurality of
| neural network layers to generate a time step output that
| defines a score distribution over a set of possible output
| tokens; and selecting, using the time step output, an output
| token from the set of possible output tokens as the next output
| token in the output sequence."
| amelius wrote:
| If you prune your transformer based NN after training, does that
| still violate the patent?
| netdur wrote:
| here your moat google!
| ianbutler wrote:
| This does not mean much for non encoder-decoder versions. GPT and
| the like are decoder only and wouldn't fall under this patent.
|
| You can look at their supplied diagrams and general summary to
| confirm.
| ttul wrote:
| Yes, but patent US11556786B2 claims the decoder-only
| transformer.
| ianbutler wrote:
| Drats!
| j-pb wrote:
| ffs, that patent never should have made it in the european patent
| office.
|
| It's obviously math.
|
| This whole "computer implemented invention" workaround is a
| complete sham.
|
| To think that the EU wastes billions annually on this broken
| institution, while completely failing to properly fund startups
| is simply infuriating.
| eikaramba wrote:
| EU? what are you talking about? The patent is registered in
| USA. the only country missing from the list as far as i see is
| actually the EU. Or did you mean USA instead of EU?
| j-pb wrote:
| The patent is applied (and granted) with multiple patent
| offices. The European (EU) patent office is abbreviated as
| EP.
|
| See: https://patents.google.com/patent/EP3542316B1/en
| arjvik wrote:
| What are transduction neural networks, and how are they different
| from existing attention-based transformer models?
| mcyc wrote:
| The mean transduction in the sense of sequence-to-sequence
| models (transducing one sequence to another).
|
| Attention based models don't necessarily need to be sequence to
| sequence. They can be classifiers, decoder only, etc. Attention
| is just one tool in the ML architecture toolkit.
| ttul wrote:
| Patent lingo is necessarily more generic than the lingo used
| within the machine learning space. For example, within patents,
| they will talk about using computers to do things, which is
| obvious outside of the patent world. But when writing a patent,
| you have to actually mention that you're going to use a
| computer to implement a neural network.
| [deleted]
| [deleted]
| mosfets wrote:
| Is Google actually suing or plan to sue OpenAI or Microsoft for
| patent infringement?
| Jackson__ wrote:
| >2018-06-28 - Application filed by Google LLC
|
| 2019-10-22 - Application granted
|
| Should probably put a (2019) in the title. Furthermore
| considering how fast the ML space moves, the fact that google
| hasn't used this to create a model significantly better than
| competitors seems to show that the patented architecture did not
| perform better than others.
| theGnuMe wrote:
| I think this patent is invalid based on the disclosure date.
| The attention is all you need paper was on Arxiv on 12 Jun 2017
| and you have to patent within one year. In this case June 28th
| 2018 is too late..
| kurthr wrote:
| Apparently, they thought of that, Line 3 of the patent
| application US10452978B2: Provisional
| Application No. 62/510,256, filed on May 23, 2017
|
| There are however other patents and applications:
| Neural machine translation with latent tree attention
| US20180300317A1 James BRADBURY
|
| I'm not an expert so I can't read the spec and claims as to
| relevance with authority. However, it still wouldn't count
| directly as prior art as it was published in Oct 2018.
| Patents are now first to file and not first to invent (to
| match the rest of the world).
|
| edit: They do also site non-patent prior are regarding
| attention. Whether this covers self attention I'm not clear
| and of course their claims have been reviewed by the examiner
| in light of the art so they're presumed valid until re-exam.
| Luong et al. "Effective approaches to attention based neural
| machine translation," arXiv 1508.04025v2, Sep. 20, 2015, 11
| pages.Luong et al. "Effective approaches to attention based
| neural machine translation," arXiv 1508.04025v2, Sep. 20,
| 2015, 11 pages.
| cma wrote:
| > However, it still wouldn't count directly as prior art as
| it was published in Oct 2018. Patents are now first to file
| and not first to invent (to match the rest of the world).
|
| There is an investor's grace period, as long as your own
| public prior art disclosure is the earliest, you get 1 year
| to file even under first to file. First to file refers to
| people filing for undisclosed inventions, disclosure acts
| as prior art against anyone else filing, along with the
| grace period for delaying your filing.
|
| Often "wow the company detailing internals at this
| conference is so generous!" is actually them getting an
| extra year on the of patent expiration date by taking
| advantage of the grace period.
| avsteele wrote:
| I doubt it. They probably filed a provisional earlier.
| theGnuMe wrote:
| Ah yes, looks like they did.
| gerash wrote:
| The patent was granted in 2019. The patented architecture is
| the Transformer architecture that everyone and their mom is now
| using for LLMs. Patents in your portfolio help defend against
| trolls and other hostile entities.
| tough wrote:
| Let's hope Google doesn't become the Troll here
| lettergram wrote:
| Is it really trolling if you're utilizing the tech and
| someone is infringing?
| mabbo wrote:
| Consider Amazon One-click purchase. That patent was one
| of very few (out of tens of thousands) that they defended
| heavily. Every other company on the planet had to have at
| least two steps to purchase an item or Amazon would come
| knocking with lawyers. They did this until the day that
| patent finally expired.
|
| One has to ask: was that to the benefit of society?
|
| Google may make good use of this tech. But it would be
| better for all of us if everyone did and didn't have to
| pay them a fee for the privilege.
| amelius wrote:
| > Google may make good use of this tech. But it would be
| better for all of us if everyone did and didn't have to
| pay them a fee for the privilege.
|
| This is reverse logic because then Google would not have
| published the tech and would have kept it a trade secret.
| lettergram wrote:
| 100% it's probably worth it in the end. You quite
| literally may not have had Amazon without it.
|
| In this case, google might not have funded the
| development of transformers which could dramatically
| reduce costs of everything for humanity.
|
| Same goes for drug development.
|
| That said, I think there's a question around software
| patents and how long they should last. Perhaps it should
| just be to recoup costs, plus some multiple. I'm not
| sure.
| bsjaux628 wrote:
| Source?
|
| I went to a talk done by Amazon's longest serving IP
| lawyer and he explicitly called out that Amazon has only
| gone to court (or asserted its patent, can't remember the
| phrase) over a kindle related patent.
| anamexis wrote:
| They definitely sued Barnes & Noble for it -
| https://www.pinsentmasons.com/out-law/news/amazoncom-and-
| bar...
| cma wrote:
| Have they released it open-source with a patent grant
| somewhere? As long as you derive from that...
| CaptainNegative wrote:
| Silly question, how did they get their patent granted in 16
| months when I have some patents pending for more than 3x as
| long?
| dragonwriter wrote:
| Or, it shows that Google can't deliver products in this space
| to (literally) save its life, no matter how good its technology
| is (which they've also shown lots of other ways.)
| onlyrealcuzzo wrote:
| Yes, the company with 8 billion user products can't do
| anything.
| brookst wrote:
| Do you think Google has 8 billion products in the AI space?
|
| Because it seems pretty clear that the criticism is that
| Google is an advertising company that uses search as
| leadgen, and is struggling to ship AI products.
| cma wrote:
| Google translate is widely used. Search also uses
| transformers for generating snippit summary boxes.
| lordvivek wrote:
| [flagged]
| hungryforcodes wrote:
| What is the point here.
| refulgentis wrote:
| Meaningless spam? Why?
| lordvivek wrote:
| its an actual explanation of the patent. but as if you were
| an alchemist. I thought it was funny.
| boppo1 wrote:
| Honestly this kind of stuff is already boring. It's an
| extremely simplified explanation, and overall a pretty low
| effort post. Don't get me wrong, ChatGPT is amazing, and I
| had it write me some recipes in Plato's voice. But now the
| novelty has worn off and this is essentially throwaway
| content. It's not contributing anything to the discussion.
| hungryforcodes wrote:
| Why?
| refulgentis wrote:
| It's cool, I got -2'd and youre still up, I misread the
| room. Just hope it's not the future here, no reason why
| other people shouldn't post ones as a wizard, orc, crypto
| ceo...
| andrewchambers wrote:
| Its getting down voted because if people want that sort
| of thing they can generate it themselves.
| refulgentis wrote:
| There's a sort of circular logic there: the alchemist one
| is fine because people wanted it, the other ones aren't
| fine because people don't want it. I prefer an HN that
| ends up skipping "explain [ARTICLE] as $X" altogether
| unless $X is on-topic
| bitL wrote:
| It's actually a pretty good ELI5 explanation of attention -
| "ingenious mechanism which illuminates certain symbols with
| a bright light"...
___________________________________________________________________
(page generated 2023-05-09 23:01 UTC)