[HN Gopher] Algorithm detected over a hundred asteroids after st...
___________________________________________________________________
Algorithm detected over a hundred asteroids after studying old
telescope data
Author : geox
Score : 133 points
Date : 2022-06-05 15:07 UTC (7 hours ago)
(HTM) web link (www.natureworldnews.com)
(TXT) w3m dump (www.natureworldnews.com)
| lostdog wrote:
| I think this is the paper: https://arxiv.org/abs/2105.01056 .
| It's all open sourced, and Python.
|
| From the paper, previously you needed a sequence of images from a
| single night to generate a "tracklet," which lets you estimate a
| guess of the orbit of the asteroid. The guess can be compared
| with images from other nights, to confirm the existence and orbit
| of the asteroids. Without the tracklet, the asteroid could be
| heading in any direction, meaning that the set of possible orbits
| is huge, and there would be many many possible matches to other
| images.
|
| The key insight of THOR (their algorithm) is that even a mediocre
| guess of the orbit makes the true orbit stand out. From an
| observed object, they predict a test orbit. Then they take all
| other images that could be along that orbit, project all objects
| in each image to the original image according the orbit, and then
| stack the projected images together. If their guess of the orbit
| was perfect, there should be a clear dot in the stacked
| projections, where that single asteroid was seen in all the
| images. But even an imperfect guess is still useful. Any object
| that's somewhat close to the guessed orbit will show up as a line
| in the stacked image. So find the lines, use some math to refine
| the estimate of that objects orbit, and now you've got the
| object.
|
| (I'm not in this field, so take all explanations with a grain of
| salt please).
| kadoban wrote:
| Really interesting. They're using a Hough transform, which is
| used to pick out shapes from an image (lines and circles most
| often), especially popular before ML techniques for object
| recognition existed.
|
| Sounds like a great way to search for asteroids, never would
| have thought of it.
|
| It's essentially just a coordinate transform that lets you add
| up evidence for a particular object, and the objects stand out
| as little bright points.
|
| I'm sure they're doing lots of clever things on top of it of
| course though.
| salty_biscuits wrote:
| Hough was looking for the tracks from charged particles in
| cloud chambers, so the algorithm comes home after a while I
| guess.
| dylan604 wrote:
| Lots(most?) comet discovery is now done off of sequences of
| images and looking for the dots that move in relation to the
| background stars once the stars are aligned. At that point,
| you could do differences with subtraction between the plates
| so that aligned stars disappear and only interesting dots
| remain. No machine learning required.
| dekhn wrote:
| (see for example:
| https://en.wikipedia.org/wiki/Blink_comparator)
| zh3 wrote:
| Previous discussion:
| https://news.ycombinator.com/item?id=31568837
| dontbenebby wrote:
| What's a good way to learn how to do this work, if you have
| scripting knowledge but not physics? I am working on a set of
| images to parse with OpenCV (non astronomical) but could apply
| the same techniques as those in the OP.
|
| Often I find lately the barrier to a project is sociological, not
| technical.
| ad404b8a372f2b9 wrote:
| The hard part of research is the domain/scientific knowledge,
| not the scripting. If you look at the original paper, the
| method is informed by their knowledge of astrophysics, and
| their knowledge of the current state of the field and existing
| methods.
|
| Many (maybe most) researchers are terrible programmers, it's a
| non-issue. If you want to learn how to do this kind of work you
| have to learn the science.
| dontbenebby wrote:
| I guess my question is what you mean by "know the science".
|
| For example I know different types of stars and how their
| compositions differ, but not detailed mathematical models. If
| by "the science" you mean the latter, great, I can drop the
| idea, if the former I might be able to add value.
|
| (For example, I've read a lot about black holes, Hawking
| radiation etc, but not in a detailed mathematical sense, just
| more general ideas like the concept of an event horizon.)
| mhh__ wrote:
| If you know calculus try the Feynman lectures, if you don't
| know calculus you need to learn calculus first.
|
| Aim to understand things for your own pleasure rather than
| being useful.
| greenhorn123 wrote:
| Reading some papers and having superficial knowledge about
| a field is not the same as knowing the science.
| pvg wrote:
| You don't have to 'know the science' to the degree of
| detail the people driving the research part of the project
| are, I think the bigger problem is many of the
| opportunities for grunt work go to undergrads and graduate
| students where the hope (in theory at least) is that it
| also helps advance their educational goals.
| jcims wrote:
| The replies you're receiving seem to be reinforcing your
| original point.
| ad404b8a372f2b9 wrote:
| Having a cursory knowledge of science is not enough, you
| are competing against people who read hundreds of papers on
| extremely specialized topics and spend 1000s of hours every
| year thinking and researching these topics. The first part
| of research is reading the literature, if you are new to a
| field you usually realize all your ideas have been explored
| thoroughly 10s or 100s of years prior.
|
| Research is not a spectator sport, it's highly specialized
| and competitive. That's not to say you can't contribute, if
| you want to participate the prerequisite is being willing
| to put a significant amount of time and effort into
| learning the skills of research, and learning about the
| specific domain you are interested in. You'll need to go
| from the basic theory to more specialized topics, read
| review papers, understand where the state of the art is,
| etc... As you read more and more you'll get insights on
| what's been done and what hasn't, you might find a niche
| that you can contribute to. In some fields the obstacle is
| gathering data and doing field work, you could make a
| contribution there as well. You could also contribute to
| open-source scientific software, I'd imagine it's be less
| competitive than doing research, you'd still need domain
| knowledge though.
| cgriswald wrote:
| The domain knowledge needed for this particular paper is
| literally just math, first semester undergrad physics, a
| chapter or two of an introductory astrophysics text, and
| some image processing--which, if you understand the
| underlying math, is easy to pick up. The novelty here is
| the algorithm, which I imagine, given the prerequisite
| knowledge, most people here could conceivably develop.
|
| Astronomy is currently a highly accessible field.
| Amateurs are still making meaningful research
| contributions. There is also _a lot_ of grunt work
| available using other people 's methods.
|
| Edit to add: You don't even need to gather your own data.
| There are many, many datasets lying around waiting to be
| processed by new techniques.
| ad404b8a372f2b9 wrote:
| That's unrealistic, the final result being understandable
| with an undergrad level of math does not mean it takes an
| undergrad level of math to reach that result. No one here
| could have divinated that algorithm without doing the
| legwork. The basic theory alone, which you list,
| represents a significant time investment and that's only
| to reproduce the work, knowing exactly which concepts you
| need to learn.
|
| A lot of the papers I read are implemented by adding a
| single line of code to a model that someone with an
| undergrad knowledge of probabilities could have written.
| Except they couldn't because to get that line of code you
| need a deep knowledge of the prior work, good fundamental
| knowledge of the discipline to even come up with such an
| idea, knowledge of the datasets that are in use, the
| problems that are interesting to work on and those that
| aren't, etc ...
| cgriswald wrote:
| No, this guy just wants to get involved in stuff like
| this and you're telling him its impossible; in a field
| where amateurs are making meaningful contributions; in
| response to a paper that is more data science than
| astrophysics.
|
| > No one here could have divinated that algorithm without
| doing the legwork.
|
| No one here could understand how to combine rudimentary
| image processing in such a way as to track a moving
| object over time? It's impressive, certainly, but there's
| more data processing here than astrophysics.
|
| Most people with a BA in science or math would have
| enough requisite knowledge to pick up what they would
| need for this paper over the course of a weekend. The
| part they couldn't do that with would be the image
| processing and data science. If they already have that?
| Yes, this is absolutely achievable. Just copying this
| work on a different data set? Also very achievable. The
| knowledge of which datasets are in use? In this field
| those are both mostly public and easy to find.
|
| Yes, those who don't have some of this knowledge will
| have to put in more time than others, but the implication
| you make about the sheer amount of time needed and what
| would actually be required just to get involved just
| doesn't apply here. There was a time half a decade ago
| where someone like the original poster could literally
| get a job doing this stuff. That may still even be the
| case.
| barkingcat wrote:
| You treat it like any other job. The people who are writing
| these kinds of papers are actual career scientists.
|
| You start with a literature review and textbook search.
| read 5-10 different undergraduate astronomy and physics
| text books, then undertake a current literature review. Do
| searches in science journals that you might get free from
| your local library, or the library associated with your
| alma mater if you have connections there still.
|
| That gets you the basic knowledge needed.
|
| Next is the hard part, using current state of knowledge,
| finding current data sets, finding your own algorithms for
| "scratching your own itch"
|
| Writing the script/programming is the last step - the least
| skilled step.
|
| Imagine you are John Carmack writing Doom for the first
| time. He had to know the math, look for "at the time state
| of the art" university dissertations regarding 3d rendering
| and occlusion, etc and combine that with his own knowledge
| about coding, assembly language, and put all that together
| to make Doom run on the hardware he had access to at the
| time.
|
| It's exactly the same for any field. Be humble, learn the
| basics, get caught up on current research, read deeply,
| read other people's disserations and ideas, then using what
| you have, and combine it with other people's ideas, and
| then you start the programming part (or you can hire people
| to code it for you, nothing wrong with that). and that's
| how it works.
|
| We all stand on the shoulders of giants who come before us.
| gammarator wrote:
| There are open positions for software engineers working with
| these teams:
|
| https://b612foundation.org/open-positions/
| https://www.lsstcorporation.org/lincc/job_opportunities
| anovikov wrote:
| If you have 4 dots on a photo or different photos and you want
| to find out if they belong to the same object (shot at
| different times) and find orbit of that object, it's trivial to
| do it. You find centroids of them using opencv, then recognise
| the plate using the like of www.astrometry.net to build WCS and
| convert (x,y) coordinates into (Ra, Dec) coordinates, and then
| you can run orbit determination code and check if resulting
| orbit is sane (that is, is within certain boundaries of
| "possible" asteroid orbits). If these are just random dots that
| don't belong to the same object, or some of them are
| noise/false detections (like hot pixel, meteor, or similar),
| the orbit will be invalid/impossible.
|
| Trick is now, how to put together these possible groups of 4
| for detection. Because they say they have 68 billion
| observations, certainly combining each to each 4 times is not
| possible. There must be some heuristic and/or trick to it.
| hkgjjgjfjfjfjf wrote:
| dontbenebby wrote:
| >Because they say they have 68 billion observations,
| certainly combining each to each 4 times is not possible.
| There must be some heuristic and/or trick to it.
|
| Maybe define heuristics for what ISN'T useful first to pare
| down the data set (looking for negatives), then focus on the
| sort where you're actually looking for positives?
|
| When I'm sorting photos for art purposes, I shoot a lot,
| years at a time, then spend a month or two sorting into
| albums, I actualy lost hours of work because I had to restore
| my phone from a backup recently.
|
| But the general idea, when doing a completely manual process,
| is you grab say, 1kish of 20k images, then from there further
| split them into 3-4 albums.
|
| Now if they were astronomy photos then you could go into that
| set of about 500 and manually look, think what's a heuristic
| to get me from where I started to here more quickly.
|
| (Sorry if articulating myself poorly, this site spikes my
| anxiety -- if I give useful information people just siphon it
| up, but if you give an opinion they don't like, folks get
| real ornery.)
| anovikov wrote:
| In fact, it might be that their method is a lot lot
| simpler:
|
| - Detect any protracted images on the photos which
| shouldn't be protracted
|
| - Interpret ends of them as 2 observations and dots at 1/3
| and 2/3 as 2 others (exploiting the fact that speed of
| angular motion at such a short interval is essentially
| constant).
|
| - Run a (very crude) orbit detection using these
|
| - Look for any suspicious dots on other images within the
| margin of error provided by this orbit (that is, where the
| object with that orbit should've been if it was real, at a
| time each other shot was made).
|
| - If found, use these for orbit refinement, rinse and
| repeat until on all shots where the object should've been
| apparent, it is present, or reject it as a false detection
| if that doesn't happen.
| agomez314 wrote:
| Wait, no machine learning?
| geoduck14 wrote:
| Nope. Just Machine Doing
| ALittleLight wrote:
| Knowing how many asteroids are known is important context to
| determine the significance of this result. Googling, I get
| 1,113,527 current known asteroids. I wonder how many additional
| asteroids they expect to find by applying this technique to other
| existing datasets and what portion of all available data they
| used to discover these asteroids.
| ourmandave wrote:
| How old is the data? You can only forecast an asteroid's future
| path so far, given the snapshot of data you have.
|
| Veritasium video with Dave Jewitt of UCLA explains a lot about
| what were doing about asteroids.
|
| https://www.youtube.com/watch?v=4Wrc4fHSCpw
| mturmon wrote:
| Mostly better source article appeared recently in:
| https://news.ycombinator.com/item?id=31616340
|
| NASA has had a dedicated search for many years:
| https://cneos.jpl.nasa.gov/
|
| Just detecting these objects in past images isn't helpful if the
| orbit can't be constrained enough to continue tracking it!
| gammarator wrote:
| In many cases the new asteroid discoveries can then be used to
| identify other apparitions of the same object that were
| previously not recognized as being asteroids. That enables
| further orbit refinement.
| mturmon wrote:
| Some colleagues have been involved with the NEO detection
| effort, so I felt called to mention that problem, which does
| limit the usefulness of these detections, and illustrates one
| of the system level problems. From the _Wired_ article in the
| post I linked:
|
| > The THOR team has shown the potential to map out the
| trajectories of numerous asteroids in our neighborhood, but
| there are few key caveats. Since their images date from a few
| years ago, asteroids that haven't been reobserved lately have
| effectively been lost at this point, although they could be
| picked up again in newer images down the road, says Paul
| Chodas, director of the Center for Near-Earth Object Studies
| at NASA's Jet Propulsion Laboratory in Pasadena, California.
| xt00 wrote:
| Scott Manley talks about it here:
| https://youtu.be/W4sOABdM9Q0?t=665
___________________________________________________________________
(page generated 2022-06-05 23:00 UTC)