[HN Gopher] Real-Time Brain-to-Image Reconstructions
___________________________________________________________________
Real-Time Brain-to-Image Reconstructions
Author : taubek
Score : 74 points
Date : 2023-02-10 09:35 UTC (13 hours ago)
(HTM) web link (medarc.notion.site)
(TXT) w3m dump (medarc.notion.site)
| ad404b8a372f2b9 wrote:
| Not much to see there.
|
| I don't have much faith that the stated goal is realizable, I
| don't think fMRI data holds enough information for faithful
| reconstruction. What it will end up being, like all projects like
| it before, is a massive network that contains:
|
| - An implicit multi-class classifier that returns 1 out of n
| previously seen images
|
| - Gigantic model on top to hallucinate a reconstruction close to
| that previous image
|
| In previous projects of this type the reconstruction was blurry
| because it was essentially the mean of the previously seen images
| close to the label predicted by the implicit classifier, in this
| project I imagine they will be photo-realistic since it's a
| stability.ai project but no closer to the content the subject is
| thinking about.
| rprenger wrote:
| I generally agree with the assessment that the limiting factor
| in quality is the signal from the MRI. Just one slight
| correction to the description of previous projects. We first
| built models that predict the voxel responses from images using
| ML. Then recorded the fMRI response on a new image. Then we ran
| many many images (not just those previously seen that have an
| fMRI response) through the voxel models and picked the top N
| images with predicted responses close to the actual response.
| Basically we used the big list of images as an approximate
| "natural image prior". Then we'd often show the average of that
| top N images so you get one picture (which destroys interesting
| multi-model stuff, but we didn't have good stable diffusion
| models of images back then that could do a gradient search on
| or something).
|
| If you wanna see what the reconstructions look like without
| fancy image priors (to get a better idea of how little signal
| there is) check out this figure from one of the early papers:
| https://pubmed.ncbi.nlm.nih.gov/19778517/#&gid=article-figur...
| sillysaurusx wrote:
| I don't know. Now that you explain how it works, it's starting
| to worry me. Imagine being hooked up to an ML model designed to
| classify whether you're thinking about porn or not, and emits a
| beep whenever it detects it. The implications of that are
| astonishing, _if_ it's accurate.
|
| It hinges on accuracy. If it can detect a topic, the image
| reconstruction is almost moot, because you can start
| interrogating people about what they know.
|
| I wish I knew more about biology to know whether this is at all
| feasible, or if it's just smoke and mirrors. (A distressing
| number of projects like this are to secure further funding for
| a lab, not really to advance science.)
|
| Since you seem to know about this, do you have any details
| about the classification process? I assume it requires the
| cooperation of the user, but it's easy to imagine that a
| "primal brainwave" pattern might be common to all humans.
| thedorkknight wrote:
| I'm actually struggling to find the usefulness of this. The
| person needs to be in an fMRI machine, so we're not talking
| about some sort of unobtrusive scanner in a public area. On
| top of that, in cases like what you're describing which
| basically just classify thoughts to a binary level, then it's
| pretty much useless because if someone knows that they're
| being scanned to see if they're actively thinking about pink
| bunny slippers, then they're going to think about pink bunny
| slippers.
| retrac wrote:
| > classify whether you're thinking about porn or not, and
| emits a beep whenever it detects it
|
| Already gone down this road with lower tech. In the 50s - 70s
| it was in vogue to monitor blood flow to the genitals while
| subjecting the subject to erotic imagery of various kinds to
| see what turned them on. [1]
|
| It was then realized that arousal is also correlated with
| pupil dilation, which meant that a much less invasive test
| was possible. Which of course led to the Canadian government
| (and probably other governments) using it in the 1960s to
| detect possible homosexuals in the public service. [2]
|
| The scientific flaws and human rights violations of such a
| test are too many to even begin to list in a short post like
| this. Still, the inconvenient and disturbing thing is: it
| does work. Sort of. Kind of. Not good enough to be confident
| about the result with an individual, but a properly run
| protocol has better than chance odds at determining if a man
| is gay or straight, in terms of aligning with their self-
| reports about what arouses them.
|
| Whether someone is actually erotically aroused by something
| based on a physical response is obviously controversial. For
| example, a repeatable finding is that self-reported-
| heterosexual men with intensely homophobic views are more
| aroused by homosexual male imagery than the average
| heterosexual male. Commonly reported in the news and media
| as: homophobic men are secretly gay. And I guess that's one
| possible interpretation. But someone about to to fly into a
| fit of rage is also aroused. Sometimes literally -- sexual
| arousal during flight-or-fight and disgust responses is a
| thing. So it's tricky to interpret what that result means, if
| anything.
|
| No doubt we will fall into similar traps again, with the
| brain scanning version of it.
|
| [1] https://en.wikipedia.org/wiki/Penile_plethysmography
|
| [2] https://en.wikipedia.org/wiki/Fruit_machine_(homosexualit
| y_t...
| lostmsu wrote:
| > it's starting to worry me
|
| Not any time soon:
| https://duckduckgo.com/?q=fMRI+device&ia=images&iax=images
| cabalamat wrote:
| > Imagine being hooked up to an ML model designed to classify
| whether you're thinking about porn or not
|
| Or whether you're thinking thoughts the government doesn't
| like.
| ad404b8a372f2b9 wrote:
| lol yes, the social implications are horrible for this kind
| of work but I'd say more so for dissidents, journalists, and
| so on rather than porn.
|
| Projects that are at the intersection of life sciences and
| machine learning are rarely interesting along both axes, it's
| always novel on one side and the other side is just there to
| make it seem cool. For example you'll find a lot of ML
| scientists investigating medical issues but lacking the basic
| medical knowledge to actually make an impact, similarly
| you'll see doctors using neural networks where a linear
| regression would do. In both cases the paper seems novel to
| the people in the field it's being published in.
|
| The classifier I was talking about is implicit, if you look
| at their proposed diagram they want to use fMRI embeddings,
| they serve as a continuous equivalent. You put a sufficiently
| large model on top of any embeddings and it will learn to
| reconstruct their modes. The embeddings are trained across
| subjects so it's not specific to a single patient but they
| hope to fine-tune it per patient using latent space
| alignment.
|
| If you look at the papers in their recommended reading list,
| it's apparent that only low-frequency signals are learned,
| for example for face reconstruction the model can learn hair
| color, sex and beard/no-beard, but the actual face doesn't
| look anything like that of the person being looked at.
| Similarly for other benchmarks, the model can tell it's
| looking at a plane, but not much beyond that. This is cool
| from a technological standpoint but I recall papers doing the
| same thing at least 7 years ago. The novelty of their
| approach will be the architecture that comes after the
| embeddings, which can be interesting but won't add any more
| information to the fMRI scans.
| r3trohack3r wrote:
| I'm struggling to find a reference, but I've already seen
| this used in the reverse during interrogations.
|
| You use something like an EEG and show someone a series of
| images to establish a baseline of "familiar" and
| "unfamiliar." Then you show them photographs of the crime
| scene that haven't been made public to test for familiarity.
|
| I question its accuracy but your scenario already seems
| possible.
|
| Edit: for siblings comments about journalists - "identifying"
| a source could probably be "accomplished" using the above
| technique already, no need for AI. Show photos of random
| people mixed in with the photo of the suspected leak in your
| gov. department.
| PaulScotti wrote:
| There's already been preprints released showing fMRI
| reconstructions that appear to do better than an implicit
| multi-class classifier [1] [2]. But also, even if the result is
| an implicit multi-class classifier, if the n is sufficiently
| high then that would still be quite impressive!
|
| [1] https://openreview.net/pdf?id=pHdiaqgh_nf
|
| [2] https://arxiv.org/pdf/2211.06956.pdf
| ad404b8a372f2b9 wrote:
| I see no evidence that they do better than multi-class
| classification, in fact they both work as I described. They
| learn embeddings of fMRI which perform an implicit
| classification of the data (which can be recovered just by
| quantizing the embedding space to get its modes) and put very
| large generation models on top.
|
| The only reason the reconstructions are much better than
| before is because they use the latest generation models.
| Those models have internal models of the classes which allow
| them to fill in the high-frequency details in the
| reconstruction. The only information they get from the fMRI
| is the same low-frequency signal that previous papers already
| had, and indeed the only things the reconstructions get right
| are low-frequency: class of object/scene, broad position of
| object, broad shape of object.
|
| fMRI scans are aggregates of brain information, they act like
| low-pass filters over the brain state. You can put as big a
| model on top as you want it won't make it more truthful a
| reconstruction.
|
| I think, as you say, that detecting as many classes as
| possible is already a pretty good goal, developing new
| embeddings and techniques to see how much juice we can
| squeeze out of the scans. I like the arxiv preprint you
| posted in particular since it does just that and evaluates
| accuracy (although the way it does it is flawed since it uses
| an image classifier on the reconstruction which presents the
| same problems). What I don't like is the misrepresentation of
| what's going on when people put those large generative models
| on top of this kind of data.
| proto-n wrote:
| Here's an old approach doing something similar, for reference:
| [1]
|
| [1] https://www.youtube.com/watch?v=nsjDnYxJ0bo
| trompetenaccoun wrote:
| Without knowing much about (f)MRI scanning, I wonder how they
| get so much detail when the resolution seems so basic.
| LargoLasskhyfv wrote:
| https://en.wikipedia.org/wiki/Die_Gedanken_sind_frei
| drcongo wrote:
| As someone with aphantasia I'd love to see this used to work out
| what it is that our brains aren't doing.
| [deleted]
| quectophoton wrote:
| [dead]
| uoaei wrote:
| fMRI will never have the resolution necessary for something like
| this. But Neuralink or other precision implants might, if they
| can put enough probes in enough of the right places.
| Shaggy2000 wrote:
| This technology one can imagine will be used in the future in
| books about a digital dictatorship , where the author has
| computers monitor the workers brains in society who are operating
| airport scanners and other security/important control jobs ....
| who might be tempted to not follow the dictatorships increasingly
| crazy rules or for example allow someone/something through in a
| critical situation that could lead to an attack by the societies
| people in the name of freedom.
|
| "The culmination of all the above projects. We will consolidate
| our findings to ultimately reconstruct images from brain
| activations in real-time (i.e., reconstruct a seen image in a few
| seconds following perception, while the patient is still being
| scanned)."
| kelseyfrog wrote:
| "I was told that if I wanted to, I could vote with my feet and
| choose a job that didn't use B2I. The only problem was, they
| didn't exist anymore. What good is a choice when only one
| option exists?"
| AstixAndBelix wrote:
| For the love of God stop using Discord to organize your community
| and hold all your discussions, especially scientific endeavors!
| you are bound to lose everything and have immense difficulty
| searching for all the past information in the future
| tbenst wrote:
| I feel this struggle although wonder what you recommend instead
| (besides slack ;)?
| AstixAndBelix wrote:
| For off the record discussions you can host a Matrix server.
| For recorded meeting you can use Jitsi. For recorded
| discussions and knowledge you can set up a wiki or a forum
___________________________________________________________________
(page generated 2023-02-10 23:01 UTC)