[HN Gopher] Event-based camera chips are here, what's next?
___________________________________________________________________
Event-based camera chips are here, what's next?
Author : hliyan
Score : 70 points
Date : 2021-10-17 11:40 UTC (11 hours ago)
(HTM) web link (spectrum.ieee.org)
(TXT) w3m dump (spectrum.ieee.org)
| Lx1oG-AWb6h_ZG0 wrote:
| If the hardware can directly tell us what pixels have changed,
| this should dramatically simplify how we encode and store videos,
| right? Ideally, the enormous amount of time computing P-frames
| would just not be necessary anymore.
| jrockway wrote:
| I think video codecs do much more than just store differences
| between frames. They detect motion and output instructions like
| "move these pixels 8 to the left and 2 up". Representing that
| as a difference would be more data.
| TooSmugToFail wrote:
| Wondering what's going to be the price point for these new Sony
| sensors. All the event cameras that we have been looking at were
| prohibitively expensive for most of the applications.
| amelius wrote:
| What is different about this compared to just grabbing frames in
| software and differencing these frames? It sounds to me that
| doing it in hardware can give a performance improvement, but only
| a minor one.
| Lichtso wrote:
| > What is different about this compared to just grabbing frames
| in software and differencing these frames?
|
| Normal cameras lose information by accumulating the intensity
| of each pixel over time until the next frame "arrives". You can
| not reconstruct that information anymore by diffing the frame-
| based output.
|
| Event cameras on the other hand track the timestamp of the
| intensity change for each pixel individually. Thus they
| experience far less motion blur because they don't average the
| signal amplitude over time.
|
| However, I haven't yet understood why their dynamic range is
| also better.
| avianlyric wrote:
| > However, I haven't yet understood why their dynamic range
| is also better.
|
| If you're sampling intensity changes at high frequency, then
| you don't need to worry about saturating each pixel between
| each sample interval. Instead you would need to integrate
| over all the collected deltas to get an intensity value for a
| given time period.
| toxik wrote:
| The event camera has no frame rate, essentially. Each photon
| sensor fires asynchronously. You should look into some of the
| applications of event cameras, they're near magical for some
| things like high velocity movements and such. They have other
| problems though, not least of which is inaccessibility.
| amelius wrote:
| But wouldn't the frame rate be limited by the complexity of
| the scene, and the available bandwidth? What happens when
| buffers start to overflow?
| staticfloat wrote:
| Imagine instead of getting a grid of pixels once every 30th
| of a second, you instead get one pixel's value, alone with
| its location, along with the time stamp at which the
| pixel's change was noticed. Event cameras can have very
| fine time stamp resolution (orders of magnitude better than
| 1/30th of a second) and so a bright moving pixel can be
| tracked very accurately.
| tuatoru wrote:
| When you explain it like that, existing camera technology
| looks like the stupidest, most inefficient way to do
| things possible.
|
| This seems like one of those "obvious in hindsight"
| discoveries, which are always the best ones.
| amelius wrote:
| But with enough depth (bits), almost every pixel will be
| changing at high frequency due to small light variations.
| dzhiurgis wrote:
| Probably orders of magnitude less power use
| bjornsing wrote:
| Much higher "frame rate" is one thing that comes to mind.
| keskival wrote:
| Here's how they can be used with drones:
| https://youtu.be/BzykucxFddI
| toxik wrote:
| I was wondering if somebody had posted a Scaramuzza paper,
| guess so... lol
| aaaxyz wrote:
| I wonder if this could be used to increase the effectiveness of
| LIDAR in vision applications. Rather than continuously scanning
| the entire FOV, prioritize areas with more brightness changes.
| Kind of like how animal eyes have low resolution in the periphery
| for event detection but high resolution in the center.
| sorenjan wrote:
| If you're not familiar with event cameras, here's a video where
| Davide Scaramuzza explains how they work (at the linked time
| stamp) and what they're good for. If you search for Scaramuzza on
| Google scholar or similar you'll find more papers on the subject.
|
| I think they're going to be used a lot in robotics in the future,
| it doesn't seem to make sense to take a bunch of frames, several
| million pixels each, and spend a lot of compute power to find
| features when consecutive frames are so similar.
|
| https://youtu.be/6Sn9-M7qXLk?t=77
| jcims wrote:
| This feels much more like how biological systems operate. I
| wonder if neural networks trained on this kind of data would work
| any differently than ones trained on standard frame-by-frame
| camera feeds.
| igouy wrote:
| First paragraph -- "The pixels in event-based cameras--a
| technology inspired by animal vision systems..."
| nynx wrote:
| I was just thinking the same thing. This would would only work
| for biologically-plausible neuron variants, where neurons train
| constantly when being used. Maybe something like HTM
| (hierarchical temporal memory) neurons would work.
| termy wrote:
| I did my masters research in this field. I utilized a spiking
| neural network to classify images that I converted to spike
| trains. They are hard to train and as there was no sparse
| method to hold the images in the GPU the memory constraints
| were significant.
| gumby wrote:
| There is a stealthy startup that at least originally was doing
| machine vision based on the human early visual system.
|
| You and I don't see anything like pixels; our vision system
| pulls out relatively high level features like parallel lines,
| motion etc and starts working with that in addition to other
| optical data, muscle feedback on your lens shape etc. This is
| why many animals, including humans, freeze when frightened or
| when they perceive a risk: stationary objects are simply harder
| for most animals to pick out. The theory is (or was -- I don't
| know what they are doing today) is that you can see things
| interesting to humans and reason about them in ways interesting
| to humans.
|
| Not part of that company: there are some wild but interesting
| theories related to this kind of thing. We pretty clearly have
| face-finding hardware (though not in the early visual system I
| presume). Do animals have this? Cats and dogs look at human
| faces.
|
| I have read a theory that reading may hijack some subsystem
| originally used for identifying tracks (footprints).
|
| It does appear that recognizing 2D pictures is learnt, not
| innate, while 1990s/2000 ML went the other way.
|
| Some anthropomorphically-inspired design could yield systems
| that are more comprehensible and more useful. I wouldn't make a
| fetish of it (cars don't run faster than a human does) but
| machines should be better adapted to humans rather than the
| other way around (e.g. cars need special places to move around
| in so can only help humans when that is possible).
___________________________________________________________________
(page generated 2021-10-17 23:01 UTC)