[HN Gopher] Sharing new research, models, and datasets from Meta...
___________________________________________________________________
Sharing new research, models, and datasets from Meta FAIR
Author : ilaksh
Score : 307 points
Date : 2024-12-13 21:07 UTC (1 days ago)
(HTM) web link (ai.meta.com)
(TXT) w3m dump (ai.meta.com)
| cube2222 wrote:
| There's honestly so much interesting stuff here, esp. the llm-
| related things - large concept models (operating on and
| predicting concepts, not tokens), dynamic byte latent
| transformers (byte-level alternative to standard tokenization),
| sparse memory layers (successfully scaling key-value memory
| layers without an increase in computational requirements).
|
| Here they are presented as separate things, each of which
| apparently improves quality / efficiency. I wonder what the
| quality / efficiency increase is of all those methods put
| together? Maybe that's what Llama 4 will be?
|
| This looks like a lot of innovation is happening at Meta in those
| areas, really cool!
| janeway wrote:
| Side track, but does anyone have suggestions about how to
| better present such content. I am struggling with similar
| docs/demos.
|
| As a documentation page, each section is laid out uniformly
| with section heading, content, link to code and link to paper.
|
| However the page itself is a blog post which will be difficult
| to find again next year.
|
| Are there other examples of companies having well presented
| technical summaries which remain findable from the hime page?
| airstrike wrote:
| I'd put a table of contents-like page up front with some
| exciting short description of each section and use
| hyperlinks, allowing the user to navigate to the section and
| back
| ms8 wrote:
| I hope that Llama 4 or 5 will have a different architecture.
| All released llamas are +/- same inference with a better
| training pipeline. The downside is that llamacpp will probably
| not be able to run new models and maybe it will be too much big
| rewrite, so we will need new c,cpp,go,rust programs.
| airstrike wrote:
| This is so cool! Playing around with the first demo is a lot of
| fun. First one to get the model to moonwalk wins. My best attempt
| was probably something like `(body_speed_forward < -0.3) *
| (head_height > 1.0) * (stay_still > 0.2) * (body_speed_vertical <
| 0.1) * (stay_upright > 0.9)`
|
| https://i.imgur.com/O5hGMo5.gif
|
| Then the "Meta Explore Theory of Mind" is even more interesting.
| There was a thread about a month ago in which some of us were
| discussing some of the concepts here like "beliefs" and updating
| a model of the world accordingly.
| https://news.ycombinator.com/item?id=42035985
| pkkkzip wrote:
| meta has certainly redeemed itself and helping AI become moat-
| free
| ponector wrote:
| They still ruin society with the Facebook, no matter how much
| good they do with LLM.
| bubaumba wrote:
| Like it or not Meta is a major player in AI world with its
| free models and tools.
|
| As for social impact of the rest it's debatable. I personally
| don't have active social accounts, and not sure this is good.
| croes wrote:
| Free by accident.
| dailykoder wrote:
| They are not free
| mupuff1234 wrote:
| Like it or not the social impact isn't really debatable,
| there's a decent amount of evidence, enough for the surgeon
| general Dr to issue a warning:
|
| https://www.hhs.gov/about/news/2023/05/23/surgeon-general-
| is...
| echelon wrote:
| Even though Meta doesn't sell I/PaaS, Meta's fitness goes up
| when AI is in the hands of more players than just Google and
| OpenAI. Commoditize AI and you create a diverse set of
| businesses that will reach customers through Meta's platforms.
| mupuff1234 wrote:
| It's not redeeming if you still continue with the original sin.
| Flomolok wrote:
| It's not a hype when it's delivers and I'm also not seeing a
| ceiling yet
|
| Yet again interesting progress.
|
| Also I like the idea of using the pose model to generate not a
| NPC but a avatar living in my phone or glas cube as a hologram.
| That would be quite scifi futuristic
| puttycat wrote:
| Can someone explain how watermarking AI videos _voluntarily_
| helps make AI safer?
| benatkin wrote:
| It lets those providing AI video generation services watermark
| all of their videos. So it isn't intended to by voluntary. You
| would be left with those services that don't comply with
| whatever the current Big Tech rules are, like people who used
| Grok/X.ai to generate images in support of Trump despite
| Grok/X.ai being inferior. https://arstechnica.com/information-
| technology/2024/08/musks...
| refulgentis wrote:
| Think this the wrong / older article - when I click the link,
| this is twitter's hosted Flux model making pictures of Kamala
| and Trump flying into the world trade center and Trump on a
| surfboard with busty cat girls. The X.ai one launched this
| week
| sangnoir wrote:
| X hosted a white-label Flux model for a while, and freely
| admitted so .
| refulgentis wrote:
| Correct, that's how I know :)
| sangnoir wrote:
| Got it! I had misunderstood your earlier comment.
| bee_rider wrote:
| How much does it take to train a model at this point? I'd tend
| to expect that it'll be in range of any major state or most
| oligarchs in the next couple years (if it isn't already). So,
| making it is probably best of everybody understands the
| watermarking to be voluntary. Images and videos aren't worth
| the bits they are printed in at this point, as evidence of
| anything in particular.
| modeless wrote:
| I really hope Dynamic Byte Latent Transformers work out. Death to
| tokenizers!
|
| Interesting that it's a a hierarchical structure but only two
| levels of hierarchy. Stacking more levels seems like an obvious
| direction for further research.
| entilzha wrote:
| Author here :), I do think it's a good direction to look into!
| That said, aside from it being a bit too much to do at once,
| you'd also have to be careful about how you distributed your
| FLOP budget across the hierarchy. With two levels, you can make
| one level (bytes/local encoder) FLOP efficient and the other
| (patches/global encoder) FLOP intensive. You'd also need to
| find a way to group patches into larger units. But ya, there
| are many directions to go from here!
| Permik wrote:
| In a way I'm kinda sad that if tokenizers will go the way of
| the dinosaurs as asking someone to give me a Unicode
| character from the private use area was one of the last ways
| you could actually distinguish a co-operative human from an
| LLM online They simply don't have those characters tokenized,
| so they can't output them. (But this is technically moot if
| the LLM has a python interpreter handy)
| bbor wrote:
| Crazy stuff. Everyone's covering how exciting all these are
| (especially LCM and the non-tokenizing-tokenizer), but I have to
| ask in case anyone's been paying attention: why are they using
| the term "advanced machine intelligence"?
|
| My initial thought is that they want to please/distract the
| doomers, but I'm prolly just self-centered!
| devmor wrote:
| I would guess it's in response to the recent market studies
| showing that the general public views anything labeled "AI" as
| a likely scam and untrustworthy.
| rajman187 wrote:
| It originates in Yann LeCunn's paper from 2022 [1], the term
| AMI being district from AGI. However, the A has changed over
| the past few years from autonomous to advanced and even
| augmented, depending on context
|
| [1] https://openreview.net/pdf?id=BZ5a1r-kVsf
| esafak wrote:
| I think Lecun doesn't like the term AGI.
| intalentive wrote:
| Every time I have to clean text I wonder why I haven't just
| trained a byte level denoising autoencoder to handle it for me.
| anon373839 wrote:
| That's a fun idea. I've always wondered about experimenting
| with u-nets and hourglass nets for text data since they're so
| efficient at capturing global and local context (in vision,
| anyway). But I've never tried it.
| SpaceManNabs wrote:
| This is like learning 10 different new architectures lol
| Roccan wrote:
| Meta's "Video Seal": Because nothing says "trustworthy" like a
| digital chastity belt. Imperceptible, they claim, yet robust
| enough to survive the gauntlet of internet mangling - sounds like
| the perfect tool to invisibly track content, not just watermark
| it.
| redleader55 wrote:
| I want to have a way to detect if content is AI generated. You
| might want to run that model on your own creations to ensure
| you get the credit for them and that no one can steal them.
| UltraSane wrote:
| Like all tools it can be used for good and evil. It could be
| installed directly in cameras to sign videos. And people with
| the power to turn it off could make AI fake videos that much
| more believable.
| idle_zealot wrote:
| I would make the argument that these AI safety initiatives
| yield messaging that muddles and confuses the public on the
| simple fact that they should not, under any circumstances,
| use a video or image as proof or assume its veracity. When I
| tell someone this it is common for them to come back with
| something like "aren't they working on things to detect if a
| video is fake?" I think this idea, that video content can
| still be trusted and that {COMPANY} is being responsible is
| the real goal of the money pumped into these watermarking
| techniques. These techniques will not actually help people,
| images and video will continue to be used for disinformation.
| The only thing that can stymie that is a broad cultural shift
| to default to distrust of photographs and video footage, to
| treat it all like you might a painting or animated cartoon
| depicting an event; maybe an accurate portrayal, but just as
| easily totally fabricated. The responsible thing for
| companies to do would be to spread messaging indicative of
| this fact, but they would rather engage in safety theater and
| score some points while keeping users dumb and easily fooled.
| UltraSane wrote:
| "they should not, under any circumstances, use a video or
| image as proof or assume its veracity"
|
| This is just silly. Courts never assume the validity of
| evidence. It is actually assumed to be invalid unless it
| can be proved to have not been tampered with. Photos have
| been able to be edited for over 100 years but they are
| still used as evidence. The person who took the photo will
| sign an affidavit and or testify in court that it is real.
| And AI videos are going to be easily detectable for a long
| time.
| Rastonbury wrote:
| I think it's reasonable to assume that any large social media
| company is already tracking video similarity in
| reuploads/edits. The remix and reused audio features are
| already baked in. Reverse image search screen caps of
| tiktok/reel pretty often return the source/original
| logicchains wrote:
| It seems such tracking can be gotten around by something as
| simple as sticking a Subway Surfers clip underneath the
| video, given how common that is.
| ks2048 wrote:
| When I wonder about the business behind Meta doing this, I see
| they have $70B in cash, so giving a bunch of AI experts hundreds
| of millions is pocket change.
| aoanevdus wrote:
| "Commoditize you complement" may be a good way of framing it.
| Consider that if OpenAI succeeds dramatically and is the only
| game in town, they could extract huge rents for anyone using
| their service. So it's in other companies interests (or anyone
| who wants to use AI) that the AI ecosystem have lots of
| competition to keep prices low.
| wrsh07 wrote:
| Imagine that something fundamental shifts in the world of AI
| research. It could be anything: AI suddenly makes programmers
| much more productive, AI becomes very good at identifying
| vulnerabilities, AI chat becomes a new major source of
| entertainment, AI images become an item popularly shared on
| Instagram (etc)
|
| Suppose any one of these things happened and suddenly Facebook
| wished that it had access to state of the art models so that it
| could customize them for its uses (internal developers or
| tools, embedding in their app).
|
| Imagine how they would feel if the only way they could access
| these models were by signing 7-9 figure deals with a model
| dealer like OpenAI. Even worse, imagine if one of their main
| competitors in advertising started providing robust AI tools to
| help advertisers adapt their creatives to various form factors.
| Facebook is now way behind and possibly has to shell out
| millions to a company like OpenAI all while also losing ad
| market share worth billions per quarter (ads on Google start
| performing much better, so Google gets more ad spend)
|
| If this worst case scenario came to pass, Facebook would look
| foolish. If even one of these things were likely their
| investments make sense. The rest (open source, make meta a cool
| place to work) are a strategy credit.
| cma wrote:
| You can't have enough top researchers without letting them
| publish.
| almostgotcaught wrote:
| everyone that has responded so far has it wrong (naively so).
|
| FB sells ad space on several apps. those apps needs people on
| them in order for the ad space to be worth anything. people, in
| turn, need content to attract them to the apps. so it's simple:
| enable people/companies/whomever to generate tons of content
| for cheap and consequently share it on the apps. that's it.
| SideQuark wrote:
| Except giving out the tools makes easier for competitors like
| TikTok to do the same, drawing revenue away from meta.
|
| So that's not it. Naively so.
| tzs wrote:
| Couldn't the same argument be made for all kinds of things
| companies have made open? Some examples:
|
| * Tesla gave away its EV patents.
|
| * Pixar and DreamWorks have both open-sourced some of their
| tools, including tools used to make some of their best
| works. For example DreamWorks' MoonRay renderer has been
| used on everything they have done since "How to Train Your
| Dragon: The Hidden World", including "Puss in Boots: The
| Last Wish" and "The Wild Robot", and will be used on their
| upcoming films.
|
| * Facebook open-sourced React.
|
| * Google open-sourced Chromium.
| bee_rider wrote:
| In the case of Tesla, if you want to sell cars, you
| benefit from open up your charging tech, right?
| mttddd wrote:
| the tools but not necessarily the data, presumably they
| have internally trained versions
| almostgotcaught wrote:
| this is like saying that AMD making chips that intel/nvidia
| employees can buy and use to do their jobs is a bad
| strategy for AMD. lol. ok not every single strategic choice
| needs to both grow the top line and be anti-competitive.
| some can _just_ grow the top line.
| mttddd wrote:
| content but also better ad targetting by better understanding
| all of the content that users post
| sangnoir wrote:
| Those AI experts are a played a critical role in Meta getting
| that $70B in the first place
| nurumaik wrote:
| https://dontfuckwithscroll.com/
| mtkd wrote:
| I was fortunate to get to a talk by Ross Taylor ex-Meta recently
| at the AI Engineer London meetup
|
| He's recorded the full talk here now:
| https://www.youtube.com/watch?v=S5l5OvJ01ws
|
| I had missed how much Meta have been doing on reasoning, ToM etc.
___________________________________________________________________
(page generated 2024-12-14 23:01 UTC)