[HN Gopher] How a PhD astrophysicist thinks about data
___________________________________________________________________
How a PhD astrophysicist thinks about data
Author : izzymiller
Score : 48 points
Date : 2022-02-01 17:56 UTC (5 hours ago)
(HTM) web link (hex.tech)
(TXT) w3m dump (hex.tech)
| latenightcoding wrote:
| this is an unnecessarily negative comment but astrophysicists get
| too much credit because of their cool sounding title. I have
| worked/collabed with plenty and their math/programming/modeling
| skills are not better than the average compsci,physics phd.
| ktpsns wrote:
| I obtained a PhD in {astrophysics, theoretical physics, quantum
| physics, computational physics} a few years ago. Choose one of
| these keywords: Which one is most catchy? I second you,
| astrophysics just sounds cool in most cases, whereas quantum
| physics is obviously much of an advantage if you talk to
| quantum computing (and AI) people.
|
| The bottom line is: In this kind of science, you are used to
| "big data", to massively parallel computing, to computation and
| statistics on various levels of abstraction. But the
| engineering skills outcome greatly vary, because from the
| physics perspective, computers are just a tool for getting the
| job done.
|
| For instance, big data in astrophysics is quite different from
| big data in accounting. Complexity in astrophysics programming
| is also very different from complexity in the banking buisness.
| People tend to get arrogant due to their years-long experience,
| but in the end all they have is just years-long experience in
| that particular domain, let it be buisness or physics.
| zamfi wrote:
| ...who is giving them too much credit exactly? Doesn't seem to
| be this piece?
| dokem wrote:
| It would seem that the interesting modern astrophysics theories
| were created, bottom up, from fundamental physics and math. Not
| by looking at the sky. My view of astrophysics is that they
| catalogue the sky and know interesting party facts but I
| wouldn't think a real genius would be an astrophysicist as
| their specialty. It also kind of bugs me that they can't really
| perform experiments through the telescope which kind of brings
| into question what is the actual definition of science, which I
| think is actually much looser than what we are taught in
| school. Sorry to join the astrophysics bashing train..
| madaxe_again wrote:
| I did my masters in astrophysics, specialised from being a
| general physicist for my bachelors.
|
| I was reading the piece, waiting for the punchline of what kind
| of unholy beast of a workstation she was using, and wasn't
| disappointed.
|
| Thing is, physicists, heck, scientists in general, _are not
| programmers_. Fortran 77 and python are pretty much the only
| shows in town - and your usual data crunching script will be
| huge, procedural, in log time, and will eat mountains of memory
| while making disks thrash as hard as humanly possible.
|
| For instance, I helped out a postdoc who I shared a lab with
| with his ephemeris calculator - it'd take in a series of fits
| images, and it'd tell you the ephemeris of whatever object you
| chose - by editing the source, and putting in the x/y of the
| object in the first frame.
|
| Thing was, it took _all night_ to do this for a single object
| from half a dozen frames. Most of the time was spent opening
| and closing each file to read each pixel, and then stuffing
| those pixels into a gigantic array, and writing that array to
| text files, and then re-reading those files, and then doing
| matrix multiplication and all sorts of amazingly baroque stuff
| that must have seemed like a good idea at the time. He was
| running it on a monster (for the time) of a workstation, with
| 64GB of ram and several TB of storage.
|
| I banged together an app in C++ for him with a basic tcl/tk
| gui, and what had taken him a day of setup and a night of
| processing and an ungodly machine instead took about as long as
| it took for him to click on an object and click "go" - on my
| creaking laptop with 128mb of ram and a 1.4gb hard drive.
|
| This was far from singular - after this, I found myself being
| "the guy" to talk to about your slow scripts - and that was
| basically every script in the department.
|
| So no, not better, considerably worse, and I can't help but
| think that having a more cross-disciplinary approach to science
| (embed tech people!) would yield benefits across the board.
| jlg23 wrote:
| Is there any other field that relies on multiple huge arrays of
| sensors, distributed around the world, that generate huge
| amounts of data non-stop? Data that is not only fed into
| processing pipelines but needs to be explored? Meteorology
| maybe. Something else?
|
| She actually explains that in the article:
|
| "one object might be anywhere between roughly 50 gigabytes to
| maybe a terabyte. By the time I'm done reducing that data and
| imaging it, it'll probably have roughly tripled to quadrupled
| in size. I tend to have larger surveys, so by the time I'm done
| I might have several tens of terabytes of data."
|
| > their math/programming/modeling skills are not better than
| the average compsci,physics phd.
|
| No, but the data they handle regularly is just much, much
| larger than in many other fields.
| traceroute66 wrote:
| > this is an unnecessarily negative comment but astrophysicists
| get too much credit because of their cool sounding title.
|
| I thought it was the oldest joke in the book that at the start
| of the academic year, physicists pronounce to all and sundry
| they're superior to all non-physicists. They then they spend
| the rest of the academic year fighting over what the most
| superior branch of physics is.
|
| As they say, the best jokes have an element of truth to them...
| throwawayarnty wrote:
| Exactly. I care about how a PhD astrophysicist thinks about
| data just as much I care how a biologist, psychologist, or
| sociologist researcher thinks about data.
|
| That is, not so much.
|
| At the PhD and research level the data types and questions are
| so specific that the analyses often are bespoke (in academia,
| the more bespoke the analysis, the more you can sell its
| novelty). PhD-level data analyses have very little relevance
| outside of its own field.
|
| There is nothing profound to be had here, even if they have the
| prestigious title of "PhD astrophysicist".
| asdff wrote:
| I think that depends on the phd. In biology at least, a lot
| of the data types and questions seem specific and bespoke,
| but they are really not. Its often just tabular data you are
| working with, and you are generating the same models as any
| other data scientist generates to find significance in
| tabular data. The only difference is in interpreting the
| significance of the model output, but the tooling is often
| the same as in a lot of fields. This is why PhD computational
| biologists have no problems pivoting into all sorts of
| distally related industries from pure biology, to data
| science, to computer science, ad tech, or even management
| consulting.
| serverlessmom wrote:
| It's certainly interesting when we stop to think about the
| elevated value we place on so many different job titles,
| viewing people as inherently super intelligent and failing to
| allow room for them to be humans as fallible as the next.
| exdsq wrote:
| > this is an unnecessarily negative comment
|
| I'd argue this is simply an unnecessary comment altogether.
| qboltz wrote:
| I don't think it's too negative. The actual complexity is high
| for the modeling, but very few of the abstractions are any
| worse than other many body simulations, unless you're studying
| specifically black holes or other theoretical edge phenomena,
| in which case it becomes hard to use the term "astro" instead
| of just theoretical physicist. Arguably magnetohydrodynamics
| has some difficult components, but it's not specific to
| cosmology.
|
| Clickbait titles revolve around sounds and language though, so
| I don't know if there's a way to combat what's profitable and
| not strictly speaking misleading.
| jpeloquin wrote:
| > One of the questions I've had to really grapple with is that I
| just cannot personally host all the terabytes of data I'm
| collecting.
|
| Hosting is a big challenge for open science. Whenever I see
| someone calling for open data, I feel half enthusiasm for the
| movement to cooperate more and half dread at the prospect of
| trying to comply. I used OSF to publish one of my datasets, but
| they instituted storage limits that would prevent doing the same
| in the future. And exposure to thousands of dollars in surprise
| fees from personal archival in cloud hosts isn't acceptable.
| Which leaves the status quo of email the author for data, hope
| they respond, as disappointingly the best option.
|
| Open code is trivial to provide in comparison, but also less
| useful.
| asdff wrote:
| Sequence read archive has been such a boon for developing
| reproducible biological pipelines without having to worry about
| data. A paper references a dataset by ID and I can use it as
| input for my pipelines and keep the raw data locally only for
| as long as its needed to generate analysis within the running
| pipeline. I can even set threshold levels of how much local or
| cloud compute resources should be used at a time if I didn't
| want to exhaust my systems with one job.
| avn2109 wrote:
| Do people ever make a torrent of their open data and link to
| it?
| ChrisLomont wrote:
| Not for these size datasets. Torrents are just too small and
| unreliable, except for the very most popular items. Slightly
| out of mainstream movies, which are small and likely more
| popular than data for an obscure science experiment, are
| nearly impossible to find seeders for.
|
| So I'd guess no seeders want to host tens to thousands of
| terabyte torrents, and then thousands to millions of those
| for all the different datasets grabbed by all the science
| projects all over the world. The odds of being able to
| download one of these on demand is just about zero.
___________________________________________________________________
(page generated 2022-02-01 23:01 UTC)