[HN Gopher] SceneScript, a novel approach for 3D scene reconstru...
___________________________________________________________________
SceneScript, a novel approach for 3D scene reconstruction
Author : Klaster_1
Score : 111 points
Date : 2024-03-23 05:08 UTC (17 hours ago)
(HTM) web link (ai.meta.com)
(TXT) w3m dump (ai.meta.com)
| ganzuul wrote:
| I would love to have this as a Blender add-on.
| dTal wrote:
| I'm sure the tech is neat. But I get a real sinking feeling from
| how hard tech giants like "Meta" are trying openly to create
| dystopia. The "Metaverse" is not aspirational!
|
| > That understanding would let AR glasses tailor content to you
| and your individual context, like seamlessly blending a digital
| overlay with your physical space
|
| Horrifying, if it ever takes off in the way that "Meta" wants it
| to. Take the modern smartphone disease and amplify it a few
| orders of magnitude.
|
| > While LLMs rely on vast amounts of training data that typically
| comes from a range of publicly available text sources on the web,
| no such repository of information yet exists for physical spaces
| at the scale needed for training an end-to-end model.
|
| Considering the tremendous scale in question, the "yet" is also
| terrifying. The only way to collect such a dataset would be if
| some large fraction of people were wearing always-on, network
| connected cameras. Meta clearly sees this future and wants it;
| the kind of future where everyone wears a Meta headset, all the
| time, and has their experience of reality mediated through it -
| optimized to maximize shareholder value of course, under the
| watchful eye of The Algorithm. A panopticon AI shoggoth.
|
| Meta is an AI paperclip maximizer, and it is getting
| uncomfortably close to the "RELEASE THE HYPNODRONES" stage.
| globalnode wrote:
| > _A panopticon AI shoggoth_
|
| That made my day. I do agree, it is creepy but its what theyre
| pushing.
| idle_zealot wrote:
| Re: that "yet". I noticed that after a recent update on my
| Quest the OS gave me a pop-up about a new feature, that when
| enabled would allow the device to remember my play boundaries
| and restore them automatically rather than requiring they be
| re-entered every time the system gets confused about where I
| am. I was a little confused why it didn't just turn that on by
| default, it seemed like a straight upgrade to the current
| handling and shouldn't require anything but a bit of local
| storage. Scrolling down in the pop-up I was informed that
| enabling the feature would authorize Facebook to scan my
| surroundings and upload a complete map of them to use for
| whatever purposes they desired. I guess they legally need
| consent for that, and tying a feature improvement to data
| collection is still totally allowed somehow. So I imagine that
| they are building that real-world dataset presently.
| ricardobeat wrote:
| > The only way to collect such a dataset would be
|
| You must have missed the part where they created a synthetic
| dataset of 100000 environments, trained the network on that,
| and released it to the public?
|
| Not saying I'm not worried about their plans too, but that is
| clearly addressed in the text.
| bobajeff wrote:
| They still have to solve for:
|
| * How uncomfortable the headsets are to wear for long periods
| if time
|
| * How useless they are (what are they solution for?)
|
| * How ridiculous they look
|
| * How expensive they are
|
| * How expensive it is to produce content for
|
| After they solve these issues then I will worry about the
| dystopia we are in. VR headsets may look cool and futuristic in
| the movies but in reality they are just screens you wear on
| your head and that's it.
| gibmeat wrote:
| All of these (except maybe the "useless" part) are almost
| always solved by time (Moore's law)
| bobajeff wrote:
| I'd think that all of them would be solved by time. I'm
| just saying they are nothing to be concerned with until
| they are (and we are still far way from that point in
| time). They are just fancy demos to make a company look
| attractive to investors.
| dTal wrote:
| Much has been written about the dystopia we are already in,
| but I am specifically concerned about the dystopia they are
| earnestly building. Meta is aware of these problems and is
| trying very hard to solve them.
|
| And they won't be useless! That's the worst part. They will
| provide compelling value, like smartphones, and as such will
| be impossible to boycott. At first the value will be simple,
| obvious utility, like route overlays. Later it will be value
| _within_ the system, much like how the main utility of
| smartphones now is to run the mandatory apps to participate
| in aspects of society that previously did not require a
| smartphone. And of course they will be addictive.
|
| Here is a short sci-fi film depicting what I fear:
| https://www.youtube.com/watch?v=YJg02ivYzSs
| FeepingCreature wrote:
| > The Metaverse is not aspirational!
|
| Wait, Snowcrash was horror for you? I thought the whole point
| of post-cyberpunk was to _not_ assume all tech would be
| dystopian.
| jprete wrote:
| Snowcrash is a dystopian libertarian world cleverly hidden by
| the viewpoint characters treating it as completely normal.
| ImHereToVote wrote:
| No code. Only for internal research.
| ricardobeat wrote:
| Obviously, this is likely to be part of a future Quest software
| update. They did release the training dataset though.
| FlyingBobcat wrote:
| Want more data? Create an mobile app to cartograph new rental
| appartments. I know in other countries you often rent with
| furniture in it. We in germany for example take everything with
| us and have to check if/how the furniture fits.
|
| With a system like the one in the video you walk through the old
| place, and walk through the new place, and output for example 2
| Sweet Home 3d plan's.
|
| No need to have 3d glasses for that.
| WhatIsDukkha wrote:
| This was a very important missing piece for mixed/augmented and
| virtual reality.
|
| The semantic understanding of the objects around you (without
| upload to a server).
|
| A detailed set of actual meshes of the objects in your living and
| working spaces.
|
| So much can be built on this key building block, to actually help
| people with their daily lives, low vision people are only one
| group that will see massive benefits.
|
| I also think this technique could and should be extended
| generally into computer vision applications, understanding
| semantic meaning, volumes and masses via the often highly
| structured environments around us.
|
| Starting from a principled position of meaning, volumes and
| meshes and then training to assign gaussian splats to those
| meshes seems like something clever people could solve a lot of
| problems with.
| nox101 wrote:
| A huge problem is privacy
|
| > without upload to a server
|
| Doesn't matter. If the app has to render them then it can still
| send them to the server. If the app does not render directly it
| can still infer them by doing collision testing against them.
| If it can't even check for collisions then there isn't much
| point in having the data in the first place.
|
| It gets worse. The more you want AR/MR to understand the scene
| the more fine grained data you need. The camera sees things
| like: the underwear you left on the floor, the medicines/pills
| on your table, the posters on your wall, the bank account
| browser tab still on your computer screen, the adult toys you
| forgot to put away, your undressed body, your dressed body,
| your political baseball cap or t-shirt, your naked
| roommate/spouse/kids who just walked into the room.
|
| I don't see how this can be solved in a trust-less way. In
| other words, you'll have to trust each and every individual
| AR/MR app you use, that they aren't spying on you. The
| platform/device/OS itself can not prevent this and still allow
| the majority of the AR/MR uses that people imagine.
| littlestymaar wrote:
| Companies don't spy on you because they don't like you, they
| spy on you because they make money out of it. Outlaw making
| money out of personal data (by outlawing targeted ads, data
| purchase for insurance, and data brokers) and the incentives
| to spy on you disappear.
|
| You can't be probusiness and pro-privacy at the same time
| though, you need to limit businesses in order to protect the
| people.
| thoughtlede wrote:
| Yes, the camera sees everything. But we could avoid teaching
| the model certain things. For instance, an undressed body and
| a dressed body could both be taught as a body. Likewise,
| medicine pills as well as regular mints could just be taught
| as mints.
|
| While this approach doesn't address privacy completely, it
| avoids certain elements of our life that are considered
| really private.
|
| I think half of the point of using the simulated dataset to
| train the model was to safeguard the above mentioned kind of
| privacy (with the other half being lack of real-world
| dataset).
| jonplackett wrote:
| I think there must be a way of doing this so you _just_ have
| to trust the OS. The camera feed itself could be essentially
| private and only pass on locations and types of objects. In
| this way you could whitelist / blocklist items or
| categories.
|
| But you would defo have to trust the OS, and in this case
| Meta...
| vergessenmir wrote:
| Isn't this more of A semantic understanding of how language
| maps onto geometry rather than understanding of actual space
| and geometry?
|
| I sometimes feel like this may not work without language
| helping it along
| WhatIsDukkha wrote:
| With a multimodal llm watching a stream of these objects,
| should be able answer the question -
|
| "where are my keys?"
|
| It should be able to respond -
|
| "Here on the dining room table" with an astar path drawn to
| them in the headset.
|
| It's a stream of tokens that you can almost just reverse
| search on to find the last instance of "keys" in it (thats a
| simplification but...).
| doctorhandshake wrote:
| My issue with this approach is that the synthetic data, and the
| language used to describe it, necessarily encode a viewpoint - a
| bias - on the part of the creators. I am firmly behind the need
| to use language as an intermediary signifier, not just for
| utility purposes, but because it unlocks the ability to use
| language as an anchor for arbitrary AR data. My problem with this
| particular approach is that it requires us all to agree on what
| things are called, and requires the training dataset to cover
| everything users might come across.
|
| My feeling is the better approach is to train a generalized
| segmentation model, derive multi-parameter representations of
| those segmented 'things', derive embeddings from those things for
| compression purposes, then allow users to 'tag' or 'label' or
| 'describe' those things as a service to other users. Then we
| could choose whose language about the world we 'subscribe' to,
| and anyone could attach any digital payload to any real
| thing/person/place without anyone else's help or permission.
| mycall wrote:
| > My problem with this particular approach is that it requires
| us all to agree on what things are called
|
| Walk around your home and notice [almost] everything already
| has a name for it as it comes from some store (unique handmade
| items are an outlier). Let's just label things we all agree
| they are called (or give them multiple labels).
| doctorhandshake wrote:
| I think you overestimate the degree to which your choice of
| words overlaps with those of others, especially when you try
| to be more specific than a generic noun by applying modifiers
| or less-used nouns. What you call a daybed I might call a
| couch or a sofa or a lounge chair or a chaise or a hassock or
| a flat couch or a downstairs bed or a family bed ... and all
| this presumes I speak the same flavor of English you do, or
| that whoever made the data set is familiar with the type of
| furniture I live around, or how it's used. The examples are
| literally limitless, before you even get into the truly
| subjective things - not just names but opinions about things
| - that people might want to use to connect digital things to
| real things. Stuff like 'larger than average', 'wasteful',
| 'beautiful,' 'sacred', 'durable', etc etc etc
|
| Believing there is one map from the real world to the world
| of symbols is solipsism.
| wdb wrote:
| Wished big companies would stop naming their stuff using existing
| terms, like in this case, scene script.
| Alifatisk wrote:
| This reminds me a little of SegmentAnything
___________________________________________________________________
(page generated 2024-03-23 23:01 UTC)