[HN Gopher] Citation cartels help mathematicians-and their unive...
___________________________________________________________________
Citation cartels help mathematicians-and their universities-climb
the rankings
Author : pseudolus
Score : 184 points
Date : 2024-01-31 11:34 UTC (11 hours ago)
(HTM) web link (www.science.org)
(TXT) w3m dump (www.science.org)
| jampekka wrote:
| When all you have is a counter, everything looks like a bean.
| McDyver wrote:
| This seems like the early days of search engines and seo
| poisoning, where random sites were referencing others to make
| them climb up the search results.
| tokai wrote:
| Nah its older than SEO shenanigans. Interestingly quantitative
| research evaluation was an inspiration for page rank, so it
| make sense that they have similar issues.
| _kb wrote:
| So this essentially appears to be the SEO blogspam playbook
| oozing into academia? I wonder if any (likely well considered)
| techniques developed to combat this context may be able to be
| backported to help reclaim the usefulness of general internet
| search.
| amadeuspagel wrote:
| Google is already way more advanced then this. If anything the
| question is more the reverse: Can Google Scholar publish a
| university ranking?
| eimrine wrote:
| Google is advancing only in the field of showing ads.
| amadeuspagel wrote:
| The techniques described in the article would not work on
| Google. That makes Google more advanced then Clarivate.
| eimrine wrote:
| > I wonder if any (likely well considered) techniques developed
| to combat this context may be able to be backported to help
| reclaim the usefulness of general internet search.
|
| For a level of an individual researcher this is easy, you just
| learn what is true and what is false. For a level of the whole
| society the battle is already lost, your (highly valuable and
| competent) solution to make a gigachad search just will be
| either lost or gamed or sued/prohibited.
| bsdpufferfish wrote:
| SEO spam is worse than ever. The only thing Google figured out
| is to raise the cost of website above 0 such as by requiring
| ssl and well written English.
| lucioperca wrote:
| "All metrics of scientific evaluation are bound to be abused.
| Goodhart's law [...] states that when a feature of the economy is
| picked as an indicator of the economy, then it inexorably ceases
| to function as that indicator because people start to game it."
|
| https://en.wikipedia.org/wiki/Goodhart%27s_law
| throwaway12314 wrote:
| I worked as a research assistant for a bit over a year after the
| graduation and I had an experience that put me off. We submitted
| a paper and got some review feedback and one of them contained
| something like this ... this and that paper also
| worked on this research area and contain this and that stuff you
| can citate ...
|
| The prof I was working with told me that this anonymous reviewer
| was probably the writer of those papers and asking for citation.
| fock wrote:
| well, if it's that transparent... I still wonder what to do
| with a handful of slightly off, but relatively good papers
| suggested to us. 2/5 would augment existing citations, 3/5 are
| related but not really relevant... No author is shared between
| them.
| enriquto wrote:
| Why did that put you off? This is a very common thing for
| reviewers to do. It is useful for you, because it makes you
| aware of other people who work in the same problem as you,
| leading to future job prospects, etc. But more importantly, it
| is useful for the readers of the article!
|
| The alternative is several independent communities of
| researchers that work separately on the same problem, but do
| not acknowledge the existence of each other. Now, _that_ would
| put you off!
| mycologos wrote:
| > The prof I was working with told me that this anonymous
| reviewer was probably the writer of those papers and asking for
| citation.
|
| I think this is mostly just a kind of fun way of complaining
| about reviewers. Having been a reviewer for many computer
| science conference, I've often had the names of other reviewers
| visible to me. Most requests for citations are not for the
| reviewer's own work, and truly irrelevant citation requests
| seem to be pretty rare.
|
| I do perceive journal/lab-centric fields to be worse about
| this, though.
| chriskanan wrote:
| It depends on context. Often reviewers are chosen because they
| work specifically in that area, and of course they know their
| work best. I've often done this as a reviewer in appropriate
| situations, where I point out closely related work that I've or
| others have done.
|
| Of course, I've also seen EDITORS email me saying they would
| like me to cite some papers from their journal after I
| submitted my paper. That was definitely a turn off.
| maranas wrote:
| This has been an issue with university rankings for a while. A
| lot of top US universities engage in this practice too -
| anecdotal, but I've heard a lot of professors force students to
| add/remove some citations, or even add their names to the list of
| people who worked on a paper to help the numbers for their
| university.
|
| It would be good to see what the criteria is for deciding if a
| journal is "to be taken seriously". I imagine for example that
| Chinese or Arabic language journals wouls be published and citsd
| in journals of those languages. That doesn't necessarily mean
| that they arent to be raken seriously in the field, it's just
| that they aren't Western publications.
| xqcgrek2 wrote:
| I've seen it go in the other direction too. Groups deliberately
| not citing other competing groups because it might help them.
| It's like the other groups don't exist.
| chriskanan wrote:
| I've observed this in multiple AI niches. In some cases I've
| emailed people saying they ignored very similar work and
| failed to cite it, and in at least some cases, they were
| apologetic and said they would update the arXiv version.
| Although of those times, they do that 50% of the time. Kind
| of tells you that the reviewers at top AI conferences
| themselves aren't that familiar with the breadth of the
| literature.
| michaelmior wrote:
| To be fair, there are so many publications in fields such
| as AI that it's really hard to stay on top of things,
| sometimes even within your own specific subarea. I'm not
| saying that to give reviewers a total pass, but I think
| it's reasonable that sometimes a group of reviewers might
| miss a relevant paper.
| generic92034 wrote:
| Yes, there are just too many publications, even with a
| very narrow focus. I am reminded of this article:
| https://slatestarcodex.com/2017/11/09/ars-longa-vita-
| brevis/
| bsdpufferfish wrote:
| Of course they aren't familiar with the literature. Most of
| the papers rediscover existing math and physics in a worse
| way (harmonic analysis, etc)
| chriskanan wrote:
| Regarding inflating the number of authors, it is especially bad
| in medicine, where I've observed a lot of names being added to
| papers for "political" reasons, despite the "author" playing no
| role in the paper.
|
| Some journals now require an "Author Contributions" section to
| at least partially address this issue.
| kjkjadksj wrote:
| The worst part of this in the biomedical field is the
| conferences. Thats because sometimes you get toddlers with an
| advanced degree and a chair position picking the conference
| presenters, who will unilaterally reject or accept people on
| the grounds of whether they like them or see them as a
| competitor, even within the same department, no regards to
| what the poster or talk might be. At least with journals you
| have the editor who can sometimes mediate a hotheaded
| reviewer dispute in a level headed manner.
| oytis wrote:
| Is it just about mathematicians? Is it something new? I though
| that was pretty much how academia is done.
| Sirizarry wrote:
| I've never been a part of academia but I know it's been fraught
| with all sorts of fraud for the last few decades (maybe forever
| idk) but this is a new type of weirdness to me. Is anything
| trustworthy is that space?
| User3456335 wrote:
| It's not really fraud though. If we get paid to upvote each
| other, the problem is not that we upvote each other but that
| there's someone paying us to do so.
| brodo wrote:
| I've been told who and what to cite in articles (even if it was
| only very tangentially relevant) as a Ph.D. student in HCI as
| far back as 2014. I also had to put my advisor as a co-author
| on papers that he didn't even bother to read...
| zensayyy wrote:
| same. I read through publications of my institute for my MS
| (EU, pretty big tech-focused university) and they pretty much
| also cite in circles (same institute and other related
| institutes of our university).
| bsdpufferfish wrote:
| There are also ethnic/cultural circles as well (Chinese
| students come to us to study with Chinese professors who only
| cite other Chinese papers).
| butler14 wrote:
| Backlink farms for universities
| ajsnigrutin wrote:
| Academia wrote the playbook and now they're abusing the rules.
| Nothing new here.
| questinthrow wrote:
| I wonder what percentage of academic work is based on cargo
| culting and practices like these. There are more "scientists"
| than ever and yet we dont really have a quantitative increase of
| people that are as productive as Euler, Einstein, Newton, Neumann
| or whoever you want to pick as a luminary
| mjburgess wrote:
| Would any of them work in academia today? It seems unlikely.
|
| There are research problems to solve in industry, with more
| prestige, more pay, and so on.
| bsdpufferfish wrote:
| Companies hire few of these people too (bad culture fit).
| tutfbhuf wrote:
| I think that also has to do with the amount of complexity.
| Henri Poincare is often said to have been the last man who knew
| all of math. Today, you need research teams across the world
| working on a very narrow topic to make tiny progress
| iteratively in the best case. I think the time when one man
| could bring us a century forward like Newton or Einstein is
| gone.
| questinthrow wrote:
| Yeah the quantity and complexity of what you need to know now
| is daunting. But even so I dont think Einstein for example
| had to know the entirety of physics and math in order to
| produce a groundbreaking revolution in physics.
| mandmandam wrote:
| > I think the time when one man could bring us a century
| forward like Newton or Einstein is gone.
|
| I'm certain his contemporaries said the same about
| Pythagoras.
|
| You're not wrong about how much more complex things have
| gotten, certainly.
|
| But you don't know what you don't know. Another Ramanujan or
| von Neumann could be right around the corner.
|
| Just look at all the wasted potential in our education system
| - it's impossible to quantify just how much effort is
| profoundly wasted.
|
| This article points directly to that wastage. We've tolerated
| an academic system that's hated by just about _everyone_
| except publishers, in an era where publishing is as close to
| free as it could be.
| lupire wrote:
| There is not room to make education a million times more
| efficient and effective for one single person. A brain has
| extreme physical limits. Computers can do it because they
| can combine millions of computers to work together, limited
| only by asymptomatic constraints like the Amdahl's law.
| mandmandam wrote:
| You don't know what the limits of brains are. People like
| von Neumann show that we really have no idea what the
| upper bound on braininess is.
|
| Now, start extending that limit by integrating computers?
| The options are _literally_ unimaginable.
|
| But that all kinda misses the point, which is, there's a
| lot of low lying fruit that we're leaving unpicked.
| thaumasiotes wrote:
| >> I think the time when one man could bring us a century
| forward like Newton or Einstein is gone.
|
| > I'm certain his contemporaries said the same about
| Pythagoras.
|
| Pythagoras is best known as the center of a mystery cult.
| What did he bring a century forward?
| Someone wrote:
| I think he's best known for his _alleged_ mathematical
| discoveries.
|
| https://en.m.wikipedia.org/wiki/Pythagoras#In_mathematics
| :
|
| _"Although Pythagoras is most famous today for his
| alleged mathematical discoveries, classical historians
| dispute whether he himself ever actually made any
| significant contributions to the field.
|
| [...]
|
| The Pythagorean theorem was known and used by the
| Babylonians and Indians centuries before Pythagoras, but
| he may have been the first to introduce it to the Greeks.
| Some historians of mathematics have even suggested that
| he--or his students--may have constructed the first
| proof. Burkert rejects this suggestion as implausible,
| noting that Pythagoras was never credited with having
| proved any theorem in antiquity."_
| epolanski wrote:
| Problem is as always greed for fame, power and money.
|
| Labs and researchers are given funds based on "impact".
|
| As we all know, when a metric becomes the goal, the metric gets
| gamified.
|
| This is very hard to fix.
|
| Gonna give you an example. As soon as you move in any direction
| in science you're entering a niche. Pretty much everything is
| its own niche where there's a limited number of people really
| able to review your paper.
|
| No, there's not thousands of experts dedicating their life to
| helicoidal peptides interactions with metal layers. It's an
| extremely small number of people. They all know each other and
| are gonna review each other's papers regularly.
|
| Solar-powered water splitting to produce hydrogen and oxygen?
| Again, extremely small club.
|
| Perovskite? Graetzel solar cells? Bigger club, but still, at
| the end of the day the relevant luminaries are a handful again.
|
| You think it's much different for quaternions or advanced
| complex numbers analysis?
|
| And those small clubs set the rules and standards, and there is
| really not much way to have oversight on those small clubs.
|
| Plenty of terrible science is published every day, I can
| confidently say 90 to 95% of experiments in Physics or
| Chemistry are impossible to reproduce (numbers people get by
| taking outlier results or even more often by tweaking the
| data).
|
| When it comes to softer sciences like psychology it's even
| worse. It's crap.
|
| It's sad, but I think the world desperately needs a *free*
| alternative to the biggest publishers out there like Nature or
| ACM or all these things. That free alternative has to be funded
| by universities and governments globally. This entity should
| only allow papers that present clear experiments that are
| reproduced elsewhere or under supervision.
|
| This would greatly enhance the quality and reliability of
| papers.
| questinthrow wrote:
| Yeah its very true, the incentives are screwed. I think
| you're touching on the key difference here. Someone like
| Euler would have had to been forced not to do math, it was
| his passion. But you have all these "scientists" going
| through the motions doing "science" because its their job.
| Passionate individuals will always be rare and the worse part
| is that the current scientific system in place is tailor made
| to root these passionate individuals out. A good modern
| example is Grigori Perelman.
| lupire wrote:
| Outside of math, even passionate people need funding.
| TotalCrackpot wrote:
| I know that at least in AI/ML there are reproducibility
| challenges, I even had a course during my masters where we
| had to reproduce a paper. Not perfect, but some disciplines
| try to address that. AI/ML has a nice feature of publishing
| almost only in open conferences/journals, there are not that
| many closed publishing venues in this field.
| SkyBelow wrote:
| >You think it's much different for quaternions
|
| It feels like it should be. Maybe the following is just the
| naivety of a novice looking at experts they don't understand.
|
| Learning numbers, integers, rationals, and real numbers,
| introduced so much power with each step, to the point that
| every child is expected to know the basics about real
| numbers. Maybe not what they are theoretically, but the basic
| ability to do something like .5 * pi or root(2)^4. They
| likely never dig any deeper, most don't really get the idea
| of never ending never repeating decimals, but they are able
| to work with them on a simple level. For most fields, an
| introduction only requires this level of math.
|
| When one does move to the complex numbers, it opens up far
| more possibilities. Fourier transform is everywhere, to the
| extent many use it without having the math to understand how
| it does what it does. More complex problems, and I mean
| outside of the field of math, only have general solutions if
| you allow for complex numbers to be used. These are
| difficult, mapping from C to C requires 4D to represent and
| thus are much harder to visualize, but people still struggle
| through it.
|
| When I realized that complex numbers weren't the end, but the
| second step in a tower of infinite height, I wondered what
| otherwise unsolvable problems needed higher levels from that,
| just like how many problems needed complex numbers. The
| difficulty working with them grows, though somewhat given C
| to C is already 4D, we have already reached the limit of 3D
| viewing power.
|
| Yet they are rarely used, and higher levels used even less.
| Maybe there is something fundamental that makes them less
| useful, far weaker than complex numbers. But from a viewpoint
| of a novice, I find that surprising.
| lupire wrote:
| Like the Eiffel Tower or Burj Khalifa, the algebra tower
| only has 4 levels (Real, Complex, Quaternion, Octonion)
| before it shrinks to degeneracy, and level 2 (Complex) has
| the best balance of dimensionality and power. Adding
| dimensions adds power, but linking the dimensions adds
| constraints on allowed structures.
|
| https://en.m.wikipedia.org/wiki/Composition_algebra
| thaumasiotes wrote:
| > Learning numbers, integers, rationals, and real numbers,
| introduced so much power with each step, to the point that
| every child is expected to know the basics about real
| numbers.
|
| The amount of additional power you get from real numbers is
| zero, and in fact it isn't possible to work with them
| because specifying an irrational number takes an infinite
| amount of time and space.
|
| They give you additional power in the sense that they let
| you give names to the exact answers to certain equations,
| but not in the sense that you can manipulate those
| equations any better than otherwise.
|
| > When I realized that complex numbers weren't the end
|
| But... they are the end. That's the point. We call that the
| Fundamental Theorem of Algebra: every solution to every
| equation takes the form of a complex number.
|
| Quaternions do not generalize from complex numbers in the
| sense of the progression of the numeric tower from integers
| to rationals to reals. That is known to stop at the complex
| numbers. They generalize from complex numbers in the sense
| of functions that scale and rotate two-dimensional images.
| Quaternions are a way of expressing functions that scale
| and rotate three-dimensional images. They are familiar to
| most people who have taken calculus under the name "vector
| cross product".
| thaumasiotes wrote:
| (And for completeness, while there _are_ octonions, the
| full generalization of "functions that scale and rotate
| arbitrary space" is matrices, which again people tend to
| be familiar with.)
| epolanski wrote:
| Not sure what the point is you're trying to make.
|
| All I said is that as soon as you release a paper you fall
| in a niche of some topic (unless the paper is super
| generic) where the number of people with the expertise
| allowing them to review it is super small.
|
| Quaternions or complex analysis where just examples where,
| even if not strictly experimental and more theoretical, if
| you're pushing the boundaries of knowledge the number of
| people able to review what you wrote and concluded isn't
| that big.
| ants_everywhere wrote:
| These big names are usually promoted beyond their actual
| achievements [0] because of the tendency to make stories about
| great people who changed the world.
|
| The reality is murkier. Huge names like Lorentz and Poincare
| had worked on relativity before Einstein. Gregory and Barrow
| had proved the fundamental theorem of calculus before Newton,
| not to mention Leibniz's work.
|
| If you mythologize the past it's easier to look around and
| wonder why we don't have immortals like Zeus or incredible
| warriors like Achilles anymore. But the truth is science always
| proceeds by steps that look small at the time and it's often
| only in retrospect that things seem amazing and unprecedented.
|
| Semi-relatedly, I used to attend a seminar with a well-known
| Russian mathematician who would often chime in with
| Russian/Soviet priority over historical results mentioned by
| visiting speakers. The cold war created two mathematical
| cultures that had limited contact. So famous European and
| American mathematical results from, say the 40s to the 90s
| often had Soviet versions worked out independently in journals
| nobody here had ever heard of and written in a language they
| can't read.
|
| So this is all just a way of saying that empirically, the big
| kahuna theory of mathematical development seems more fiction
| than reality. And it should be treated with skepticism when you
| hear things framed in those terms.
|
| [0] Not that their achievements aren't great. They are all
| incredible. I just mean they seem more incredible than they are
| if we forget all the other incredible achievements. Shoulders
| of giants etc etc.
| lupire wrote:
| Where all of today's Lorentzes and Poincares?
| ants_everywhere wrote:
| Realistically, a lot of them are probably training AI
| algorithms to get people to click on ads and questioning
| their life choices.
|
| But math keeps advancing at a good clip, there are tons of
| brilliant young people. Physics IIUC is largely gated on
| waiting for new experimental results. Einstein, Poincare,
| Newton and many others arrived at a time when the
| experimental physics data was perplexing and theoretical
| advances were needed to clear up the picture.
|
| The reverse is true now if I understand. People are just
| waiting for more funding to build larger accelerators or
| whatever big expensive astronomy things they need to test
| the many predictions we have from the various mathematical
| attempts to unify quantum mechanics and relativity.
|
| Biology seems to be moving pretty briskly. I can't speak to
| chemistry or any of the other sciences.
|
| EDIT: I should add that when I left academia about 8 years
| ago, the mood was that academia was increasingly hostile to
| smart people who just wanted to do research. The move
| toward running universities like businesses has pushed out
| a lot of people who would be candidates for the next
| Einstein or Poincare. Poincare happened to be independently
| wealthy, so wealth may again be more of a factor than it
| was during the last century.
| bsdpufferfish wrote:
| > mood was that academia was increasingly hostile to
| smart people who just wanted to do research.
|
| Exactly these people are more likely to be spending their
| time speed running Mario.
| LegitShady wrote:
| working for ad companies hiding as tech companies?
| eimrine wrote:
| I have a sad guess that attention economics kills them.
| Imagine being Poincare but having a smartphone addiction.
| jackcosgrove wrote:
| It sounds to me like the who of science doesn't matter as
| much as the where and the when. There are ideas floating
| around in the ether, and the discovery of these ideas
| generate more ether for future discoveries. To a large extent
| being a scientific luminary is being in the right place at
| the right time.
|
| Which doesn't seem at odds with the grandparent post, given
| that we shouldn't expect throwing more people at the problem
| to accelerate discoveries.
| ants_everywhere wrote:
| > It sounds to me like the who of science doesn't matter as
| much as the where and the when. There are ideas floating
| around in the ether, and the discovery of these ideas
| generate more ether for future discoveries. To a large
| extent being a scientific luminary is being in the right
| place at the right time.
|
| I think it's really an empirical question how it works. My
| own take is that it takes both. You need the ideas floating
| around and you need an obsessive synthesizer.
|
| A music example might be Bob Dylan. Folk music could have
| been a passing local fad, but Dylan studied all of it,
| distilled it down, and produced a version that seems
| different in kind than what came before. Without Dylan, we
| could look back on that era as being like the swing revival
| of the 90s. Just a period where a kind of music was popular
| again but with no real lasting change.
|
| Similarly, there could be areas of math that are ripe for a
| Newton figure to come in and clarify everything. But that
| field could wait 100 years in the backwaters of math
| journals before anyone took notice. So you can't directly
| jump from the ideas floating around to the synthesis, you
| do need some sort of figure who will read and understand
| everything and put all the pieces together into a coherent
| piece of writing that people can read.
| User3456335 wrote:
| No offence to them of course, but they already took the lower
| hanging fruit. It's hard to come up with fundamental axioms and
| laws when they have already been found. And there are
| definitely people around that have the same potential.
| ronald_raygun wrote:
| In mathematics, this is just called a clique!
| _kb wrote:
| I thought it was a semigroup.
| User3456335 wrote:
| Now I realize why it's called a field. I always thought of a
| physical field but this makes way more sense. If there's
| enough division in a group, it is considered a field.
| ants_everywhere wrote:
| These citation games are at least decades old. The PageRank
| algorithm came out of the literature on ranking academic papers
| by citation, and many of the SEO manipulation techniques work in
| both cases. I'd guess it's being done on an increasing scale
| given the article.
|
| As a semi-related phenomenon, I also feel like mathematicians are
| culturally prone to under-citation. I don't just mean that nobody
| cites older mathematicians like, say, Leibniz or Euler, for
| techniques they invented. But pretty frequently I'd read papers
| that _really_ seemed like they were heavily indebted to a paper
| that came only a few years before but didn 't appear in the 5-10
| citations. Maybe there it wasn't mandatory to cite those papers,
| but it made it hard to chain backwards into the literature to get
| more context on methods and ideas. And sometimes it came off as
| the author trying to make a tool appear ex nihilo. I'm sure this
| happens in other fields too.
| _the_inflator wrote:
| SEO and CO (citation optimization) maybe rhyme for a reason.
| fluoridation wrote:
| My understanding is that citations are used to avoid having to
| argue for the truth value of a proposition, not to give credit,
| nor to establish some kind of chain of trust. A sort of "I
| assume this is true, if you want to argue about it take it up
| with this guy". If two papers are very similar in structure,
| but neither uses the conclusions of the other to support the
| truth of a dependent claim, then I don't see why the older
| paper should be cited by the newer paper.
| michaelmior wrote:
| It's somewhat field dependent. I would generally expect
| citations to be used for both purposes. That is, if someone
| is proposing work substantially similar to existing work,
| they should be expected to cite that other work and identify
| differences with the new work since novelty is a key
| component generally (although not always) required for
| publication in my area.
| michaelt wrote:
| Academic papers often include an introduction which
| summarises the context in which the paper is written.
| Consider https://arxiv.org/pdf/2201.03545v2.pdf where the
| paper says "For many decades, this has been the default use
| of ConvNets, generally on limited object categories such as
| digits [43], faces [58, 76] and pedestrians [19, 63].
| Entering the 2010s, the region-based detectors [23, 24, 27,
| 57] further elevated ConvNets to the position of being the
| fundamental building block in a visual recognition system."
|
| This shows the authors are familiar with the field, avoids
| inadvertent plagiarism, allows them to make it clear
| precisely how their contribution contributes to knowledge,
| shows people from funding bodies that this is a cutting edge
| and important field of research, and points readers in the
| right direction if they want to see earlier ideas, or ideas
| in a more applied context.
|
| If among those 92 citations should be a few papers from your
| boss and your colleagues, nobody will see that as unusual -
| as long as they're at least marginally relevant.
| fdupress wrote:
| This is a lot less true in mathematics research, where a
| question existing and not being trivial is enough
| motivation to investigate.
|
| I would in fact argue that the trend, in "applied" fields,
| of justifying the importance of a piece of work by pointing
| out that a lot of people are doing similar work is in fact
| self-fulfilling. That makes it somewhat useless as a
| measure of importance.
|
| Scientific context should be critical, not just
| descriptive.
| davidgay wrote:
| > I would in fact argue that the trend, in "applied"
| fields, of justifying the importance of a piece of work
| by pointing out that a lot of people are doing similar
| work is in fact self-fulfilling.
|
| There is one filter on this, though: you have to get
| funding for the work, which is a different process than
| getting a paper accepted. There is of course some
| overlap, as "people are working on this so it must be
| important" can be useful in arguing for funding, but
| funders at least potentially have other criteria in play
| too.
| fluoridation wrote:
| You would argue for the importance of research verbally,
| or in some kind of proposal document, though. There's no
| need to include part of that document in the final
| publication.
| abdullahkhalids wrote:
| Your claim about applied fields is not correct. When you
| do your research, you survey what other people have done
| in some subarea. You determine what has not been done.
| And then you do it.
|
| When you write your paper you say exactly this in the
| introductory context. Here is what other people have
| done, here is a problem not yet solved in this area, we
| solve it. It saves the reader an enormous amount of time
| by not having to do a literature review themselves to see
| how the paper fits into the larger thrust of the field,
| and what the novelty is.
|
| How can you justify the novelty other than by comparing
| to other people's work?
| canadiantim wrote:
| Qualifications of truth are often written into the paper too
| tho, so e.g. in biology paper people will cite studies, but
| then explain their mitigating circumstances and context and
| how they need to be balanced with other studies.
|
| So while I agree that generally citations are a statement of
| claim and the reference given allows people to see if that
| statement of claim is actually supported, within the paper
| itself good authors also explicitly weigh the "truth value of
| a proposition".
| fdupress wrote:
| You're arguing against practice in mathematics based on
| practice in an empirical field, though. Truth in biology is
| fuzzy in a way truth in most mathematical fields is not.
| nickpsecurity wrote:
| "My understanding is that citations are used to avoid having
| to argue for the truth value of a proposition, not to give
| credit, nor to establish some kind of chain of trust."
|
| People do this in science, politics, and church. It's a call
| to act only (or mostly) on that person's testimony. An act of
| faith. It's both a red flag to watch for and a necessary aid
| for people.
|
| A red flag because it's often a sign of fallacy. I default on
| looking that person up along with folks that might disagree
| with them. Every success adds credibility. Failures mean I'll
| default in a different direction if I hear their name come
| up. Might even dig into what other claims they have which are
| spreading.
|
| Those sources who keep getting the job done in specific areas
| can graduate from red flag to reliable enough to default on.
| Only in that area for claims not to far from what they've
| been good at. If outside or a big claim, still double check.
| Also, be able to provide verifiable evidence of why you trust
| the source.
| Karellen wrote:
| Are you suggesting it's a red flag for science/mathematics
| to build upon previous results, to "stand on the shoulders
| of giants" (as it were)? What are you proposing, that
| researchers should reproduce all of the results leading up
| to the current state of the art in their field before
| embarking on their own original experiments?
|
| Edit: So, if anyone wants to build on Fermat's last
| theorem, it's not enough that they cite Wiles' paper
| proving it, but they should have to re-derive the proof
| themselves?
| nickpsecurity wrote:
| I answered your first question in my original comment.
| People can build reputations with good science. I'll
| answer the other one now.
|
| I think humans operate on faith in others' testimonies
| way more than actual science. As a follower of Jesus
| Christ, this doesn't bother me since I believe we learn
| the truth in multiple ways. However, science is rooted in
| empiricism which says any belief must be proven,
| independently reviewed by skeptics, and we keep using it
| while it seems correct. Dissent is always allowed in real
| science since our models (beliefs) might be useful but
| incorrect on some level. Whatever survives the most
| review with the most utility is what we should keep
| using.
|
| What happens is often different. People believe something
| just because it was reported somewhere or had a name on
| it. That's pure faith (or argument from authority). Many
| works aren't just unreplicated: they _can't_ be
| replicated. Many institutions have political biases
| affecting their work. Many don't allow dissent on some
| topics. People good at one thing often fail at another. I
| also see big jumps in extrapolating a small success to
| future, larger successes.
|
| While common (even on HN), none of that is the scientific
| method in action. It's more like a mix of science,
| unproven dogma, speculation, political maneuvering, and a
| popularity contest. The ratios of these vary by field and
| institution. Don't call anything science or "proof"
| unless it fully follows the scientific method. And builds
| on premises that did, too.
| PaulHoule wrote:
| My impression is that math is strange and disconnected. When I
| was a grad student the (large) physics department had a
| colloquium series that brought in speakers that were
| interesting to the whole department despite that department
| having major splits between astrophysics, condensed matter,
| accelerator physics, high energy physics, biophysics, etc.
|
| There was a chemistry colloquium that I went to occasionally,
| partially because I had some friends in the chemistry
| department who were into the same quantum chaos stuff I was
| into, but it was clear that a certain faction of the chemistry
| department showed up there and the rest of them went to other
| talks.
|
| The math department didn't have any lecture series which was of
| broad interest to the department.
|
| Physics brought in senior people who had something interesting
| to say at the colloquium, but more than half of the people the
| CS department brought in for their colloquium (which I attended
| a lot later when I had a software dev job at the library) were
| job talks and often the people did not know what they were
| talking about and it could get embarrassingly bad. I was lucky
| to see Geoff Hinton speak before he got famous but there were
| times I would team up with "the meanest physicist in the world"
| (for real, not my evil twin) to ask questions at the end that
| would call out the weakness of the speaker because the audience
| was just not holding therm accountable.
| sfpotter wrote:
| All of the math departments I've been involved with have had
| department-wide lecture series. Actually, all of them even
| had regular lectures meant to apply to the entire college.
| BobaFloutist wrote:
| >The PageRank algorithm came out of the literature on ranking
| academic papers by citation
|
| I never made this connection, but it makes so much sense.
| bArray wrote:
| The situation is out of control. I've also seen that during peer
| review a reviewer may give a list of publications they 'suggest'
| for the literature review, sometimes completely unrelated fields.
| If you choose not to, you will likely see your paper rejected.
|
| The truth is that almost all papers published across almost all
| fields cannot be replicated, full stop. For those that can, the
| results rarely match up, indicating that the tests are chosen to
| boost the performance of the method.
|
| If you want to boost citations legitimately, purposefully leak
| your own paper to get past the publisher pay wall, make it
| colourful and accessible (if it's too dense they are unlikely to
| understand it), give examples on how to replicate your method,
| the abstract is likely the only part that will be read so make it
| count.
| eimrine wrote:
| > The truth is that almost all papers published across almost
| all fields cannot be replicated, full stop.
|
| This is not true for Mathematics.
| sukruh wrote:
| Sure, but most Maths papers are read by like 3 people in
| total globally, so there's that.
| psychoslave wrote:
| Citation needed. XD
| lucioperca wrote:
| The pareto effect!?
| User3456335 wrote:
| If that's true it's only because most people don't
| understand these pure maths fields. But their results are
| generally very useful to the applied mathematicians (whose
| results are and should be used in most other fields) and in
| this way there is still an impact although it's not always
| reflected in the citations.
|
| The fact is that these pure mathematicians could almost
| surely write the more applied papers but don't because it
| would feel to repetitive. You don't need to read Euler's
| work in order to indirectly benefit from it. It's the same
| for a lot of pure math.
| lupire wrote:
| Applied math does not use much pure math. It hasn't for
| decades.
| bsdpufferfish wrote:
| This is not even close. Pure math is often complete
| clueless about how it applies. For example Developing
| algorithms to compute properties used in the most basic
| pure math is a massive pursuit.
| itointegral wrote:
| As an applied mathematician, I'd say this depends on the
| field. Pure maths more often than not comes with
| assumptions that make the theory more elegant, but by
| doing so they also drastically limit its application to
| real life scenarios.
| nxpnsv wrote:
| It is not true for Physics either.
| josephg wrote:
| Or computer science, at least when the code is also
| published.
|
| (That said, performance measurements in most papers are
| complete junk. Most code written by academics is missing
| trivial, obvious optimisations which could completely
| change any comparative benchmarking results.)
| PaulHoule wrote:
| I'm not sure that the average CS paper really tested the
| algorithm they thought they were testing. That comes from
| a lot of commercial projects where people weren't using
| the algorithm they thought they were using (even simple
| algorithms) and a few cases of working with code from CS
| researchers.
|
| There was that time I got a C program from a famous ML
| researcher, tried to run it and it segfaulted, I pointed
| gdb at it and realized it was crashing _before it got
| into main()_. Understanding why was an educational
| experience (don 't allocate a 8 GB array _you never use_
| on a 32-bit machine... It worked for the researcher
| because he had a 64-bit machine) but it did not increase
| my faith in the code quality of academic projects.
|
| Some CS researchers are brilliant programmers but the
| fact is that ordinary devs make a living writing _code_
| and CS researchers make a living writing _papers._
| eimrine wrote:
| Allocating a big struct for everything and not using
| pointers is one of possible techniques of dealing with
| data in C programs. Anyway having a requirement of 8G RAM
| is not a good example of something "can not be
| replicated".
| PaulHoule wrote:
| I know all about alternate ways to deal with memory, if I
| was just programming purely for fun I'd write assembly
| and not waste time moving the stack pointer around and
| other meaningless activities associated with calling
| conventions.
|
| It's a sign of poor quality code.
|
| He probably was using the array for something at one
| point, quit using it, and never bothered to delete it.
|
| And of course it is C so there are no unit tests despite
| it being the kind of code that is eminently unit testable
| (if it wasn't written in C, not like you couldn't write
| unit tests for C but who does?)
| matthewdgreen wrote:
| > Most code written by academics is missing trivial,
| obvious optimisations
|
| I agree with this, but I would pose it the opposite way.
| Academics shouldn't be prematurely optimizing their code:
| they don't have the resources for it, nor will their
| results be representative across all hardware. I
| routinely see (and publish) papers with novel
| cryptographic protocols where the reviewers will say
| things like "compare the running time of your algorithm
| to a state-of-the-art production system that Apple spent
| $50m developing and optimizing for their silicon, using
| hand-tuned assembly." We're not going to win in that
| benchmark, and we don't have the resources to even
| compete. _Moreover, the resulting code wouldn 't make for
| good reference code._ But that doesn't mean we shouldn't
| publish our work, so that e.g., Apple (or other
| resources) can begin the process of optimizing it for
| production.
| josephg wrote:
| > Academics shouldn't be prematurely optimizing their
| code
|
| While I agree with that in principle, the question of
| whether or not a given algorithm is fast enough to be
| practically usable is very useful information for the
| reader of your paper. Do I need to read your paper,
| understand it, implement it myself, and only then figure
| out that its 3 orders of magnitude slower than existing
| algorithms? I know its a lot of work, but as the reader I
| don't want to do that work either.
|
| I've spent the last few years working on a novel
| collaborative editing algorithm for text editing. If the
| algorithm was orders of magnitude slower than existing
| CRDTs, it would never get adopted. And I want my work to
| be used. _I_ want to use it, but again - only if its fast
| enough to be practically usable. I ended up writing a
| very highly optimised implementation of the algorithm -
| which took a massive amount of time - to answer that
| question. For the paper, I spent another couple of days
| writing a much slower reference implementation that
| people can read to understand how it works. (The
| optimised version is thousands of lines, and the
| reference code is a few hundred lines).
|
| I understand how much work and expertise is needed to do
| things like this, but I still want someone to do this
| work. Even if its "We wrote a straightforward
| implementation of <competing algorithm> and <our
| algorithm> in plain C code. They perform at roughly
| similar levels of performance <see chart>. We suspect
| highly optimised versions of both approaches would also
| perform at roughly similar levels of performance."
| PaulHoule wrote:
| Who's going to replicate an experiment that takes a $5
| billion particle accelerator?
|
| For that matter, who's going to replicate a condensed
| matter experiment for which a grad student took three years
| to build the apparatus at a cost of $250,000?
|
| Peer review is an absolute joke. In principle researchers
| could publish their data sets and reviewers could see if
| they could replicate the analysis but they don't do it
| because they know if they tried they'd get different
| answers, see
|
| https://www.nature.com/articles/d41586-023-03177-1
| eigenket wrote:
| > Who's going to replicate an experiment that takes a $5
| billion particle accelerator?
|
| In the case of the LHC, which I assume is what you're
| talking about, this is why we have 4 pretty much
| independent detectors around the LHC looking at different
| but at least partially overlapping stuff. The Higgs boson
| was detected by the ATLAS detector, but wasn't considered
| to be "discovered" until it had also been detected at the
| CMS detector.
|
| > Peer review is an absolute joke
|
| That strongly depends on the field and the journal in
| question.
| PaulHoule wrote:
| If they flew out the peer reviewer to interview the
| researcher and take a look at their apparatus or if the
| peer reviewer loaded up a Jupyter notebook and rechecked
| the analysis there might be some value in peer review. If
| peer review is just somebody looking at the paper
| superficially than it's of very limited worth.
|
| When I wrote my first paper somebody else had done a very
| similar experiment at another university and we wound
| method up peer reviewing each other's papers. Turned out
| we were using the same bogus data analysis that was being
| used in 1000s of other papers. I was vaguely aware of
| this when I wrote the paper but did not feel
| psychologically safe at all in an environment where the
| American Physical Society claimed a PhD had a 3% chance
| of having a permanent career in the field.
|
| Someone who was later to become a titan in the field was
| at our lab and was crying some nights because he had no
| idea where he was going to get his next postdoc wrote a
| paper about this problem more than a decade later after
| he had gotten tenure in _a statistics journal_ that
| physicists probably won't even read.
| eigenket wrote:
| That sounds (as I said before) quite specific to the
| field you found yourself in. That isn't how things work
| everywhere.
|
| For a relatively recent example - in my field of quantum
| information someone found in the course of reviewing some
| newer work that there was a mistake in one of the big
| foundational papers that this newer work was based on.
| They shared it with some of their colleagues and together
| they did some work finding out exactly where the error
| was and what the impact is on work that is based on it.
|
| The response from the community was they got a talk slot
| at a pretty prestigious conference (QIP 2023, you can
| watch the video here:
| https://www.youtube.com/watch?v=2Xyodvh6DSY) and the work
| they did has been published here https://quantum-
| journal.org/papers/q-2023-09-07-1103/ and in a followup
| paper here
| https://www.nature.com/articles/s41567-023-02289-9.
|
| My point is essentially that it isn't a big shock that
| this error wasn't found by peer review when the
| foundational paper was published in 2008. Even when it is
| working as well as possible peer review can only detect
| most errors, not all errors. _However_ when your field is
| functioning "normally" and everyone does their jobs
| properly this doesn't matter because the errors that get
| past peer review get detected by other people building on
| the work later. On the other hand if your field is _not_
| functioning properly and people are knowingly doing stuff
| that is wrong then peer review _can 't help you_. If the
| status of the field is so broken that people who know
| stuff is wrong are still pushing for it to be published
| then you aren't in a place where peer review can save
| you.
|
| My opinion is you are incorrectly putting the blame on
| the (highly imperfect) process of peer review here where
| it doesn't really belong. Peer review as we perform it
| today sucks for many reasons, but it isn't to blame for
| the problems you saw.
| PaulHoule wrote:
| It's not to blame for these problems but it _fails to
| counter them_.
|
| In medicine see
|
| https://www.cochranelibrary.com/
|
| where there is a systematic process to search the
| literature and select papers with usable results. I've
| never seen more than 50% of papers get accepted and the
| other day I saw one where they found 80 invalid papers
| and 2 valid papers. It seems to me that peer review isn't
| accomplishing very much if it is passing through papers
| that can't be built on. For that matter, why is this work
| even being funded?
| eigenket wrote:
| Yeah I wouldn't expect it to counter such problems. Peer
| review is by its nature review by other people in the
| same field. If you have a field where "bogus" methods are
| being used in thousands of papers then you aren't going
| to be able to get things reviewed properly.
|
| This isn't an indictment of peer review as it is
| practised in every field.
| PaulHoule wrote:
| I'll say a good field is an island in an ocean of bad
| fields.
|
| Look at hep-th. If in 100 years people conclude there is
| no evidence for supersymmetry and extra dimensions
| something like 90% of that field will be "not even
| wrong". There is so much of the drunk looking for his
| lost keys under a lamp. I mean, there is a huge industry
| in 2-d gravity where it is possible to calculate things
| but gravity doesn't attract in 2-d so the line between
| that and Newton's apple is unclear. I've talked to astro-
| ph refugees who think that current theories about
| accretion disks are "not even wrong".
|
| Then there are all the folks who tell me firewalls make
| no sense because there is no such thing as the event
| horizon (locally) but from the viewpoint of people who
| fall in it is a tiny difference getting killed at the
| apparent horizon meanwhile the nonsensical "black hole
| information paradox" (sorry, no unitarity = no quantum
| mechanics) still gets breathless and insufferable
| articles written about it just because somebody famous
| for their disability made a bet. If young people were
| able to get established in the field you wouldn't see
| people so fixated on bad ideas.
| pa7x1 wrote:
| And the same was done for LIGO and the discovery of
| gravitational waves. 2 observatories, one in Hanford and
| one in Livingstone.
| analog31 wrote:
| >>> For that matter, who's going to replicate a condensed
| matter experiment for which a grad student took three
| years to build the apparatus at a cost of $250,000?
|
| That was my thesis. To make it worse, the laser I used
| became unavailable as I was finishing up. My work was
| adopted by 2 labs but required extensive adaptation.
| EvgeniyZh wrote:
| > For that matter, who's going to replicate a condensed
| matter experiment for which a grad student took three
| years to build the apparatus at a cost of $250,000?
|
| If the results are interesting to anyone and there is
| more science to do, people will eventually replicate (or
| fail to do so, like with Majoranas). If not, then why
| would we care about replication?
| BeetleB wrote:
| Sorry, what?
|
| It's extremely true in physics. I've published in (good)
| physics journals.
|
| Computational papers: Authors leave out details because
| they don't want to give up the goose that lays the golden
| eggs. They broadly describe the technique, ("solve this PDE
| computationally"), and anyone who wants to replicate has to
| craft his/her own solution. If they fail to replicate, the
| authors just say "your simulation was poor and probably
| numerically unstable".
|
| This was the norm.
|
| Experimental papers: Same thing. Experimentalists are
| inventors. They don't buy off the shelf equipment - they
| build their own. They _don 't_ provide details of how they
| built it in their papers.
|
| At conferences, PIs would openly discuss whether they
| "believed" a highly cited paper. When I was a new grad
| student, this troubled me. Why should belief play a role?
| Just replicate it! A year later: "Oh, of course these
| studies can't be replicated."
|
| It's one reason I left academia. I was doing
| theoretical/computational physics, but the whole game
| reminded me of English literature: The value of your work
| was always seen with subjective eyes. Your paper could be
| rejected simply because the reviewer doesn't "believe" it.
| abdullahkhalids wrote:
| > Why should belief play a role? Just replicate it!
|
| If you are talking about theory or numerical simulation
| papers, just replicating the calculations in the paper
| are not enough to believe the conclusions. There are many
| assumptions that go into any calculations, in terms of
| how much of spherical cow you treat reality as. Changing
| those assumptions will change the conclusions. The
| assumptions are not always clearly stated, and even the
| authors might not realize all their assumptions. It can
| sometimes take many years or decades for people in the
| field to figure out how a paper was wrong.
|
| So yes, experts do have to depend on their subject
| intuition to determine how much they believe some
| surprising result. Not to mention, if you go through
| historical papers in your field you will inevitably come
| across decades long fights where one group is coming to
| one conclusion in their paper and other is coming to
| another, before the situation is resolved one way or the
| other.
| BeetleB wrote:
| Except it's the same problem for experimental papers. My
| thesis was based on a highly cited experimental study. My
| role was to "support" that paper with numerical
| simulations. And yet, at conferences, people would openly
| discuss if the original, highly cited paper, was fact or
| fiction.
|
| The guy built/fabricated a device, and reported
| interesting measurements. Want to know if it's true?
| Build it yourself! Except when you do and get differing
| results, there will be quibbling over whether you built
| it properly. There are always details left out of the
| original paper that can be appealed to. The original
| author is king of the field, and people will simply
| ignore your contrarian findings.
|
| For theoretical/numerical results, it is for sure valid
| to criticize assumptions. But in the longer time frame,
| the question shouldn't be "Are these assumptions valid?"
| but "Is there experimental evidence to support these
| calculations?" With a lot of the papers published
| (including mine), I can assure you that no one will ever
| be able to construct an experiment to (in)validate my
| results. And that was true for probably over half of the
| theoretical/numerical results in my field. We were all
| publishing things that most of us believed could never be
| connected meaningfully to the physical world. We're not
| talking string theory or high energy stuff - more like
| material properties.
|
| The goal was to publish, and convince others. Not to
| understand reality. Hence, more like literature.
| PaulHoule wrote:
| I dunno. When I was in grad school I regularly would find
| papers in _Annals of Mathematical Physics_ that would go on
| for 50 pages and have critical errors on page 17 and 45. I
| did a calculation like that which I know what right at the
| equation-by-equation level because I made unit tests for
| everything, but the whole thing was still useless (published
| in my thesis but not _Annals_ ) because the coordinate system
| we were using converted a periodic orbit into something like
| a spiral so the we couldn't compute a topological factor
| because there was no topological invariant...
| alafactor wrote:
| I've published in some well-respected, but not highest-
| tier, journals (think impact factor ~ 5-7), where this was
| absolutely the case. You could very often identify the
| reviewer by who the corresponding author of the twenty
| papers they asked you to cite was. Perhaps it's field
| dependent, my field (chemistry) was not always as, shall we
| say, ethical as some of the other fields.
| PaulHoule wrote:
| ... I thought people in physics were pretty sportsmanlike
| but my understanding was that biologists would stab you
| in the back for the slightest advantage. I would complain
| about how hard it was to make it in physics but in
| biology it seemed almost impossible to get established
| since young researchers just couldn't get grants period.
| Balgair wrote:
| I had on old physics professor who said that by the second
| page of any math/physics/equation-heavy paper, assume all
| signs are randomized.
|
| I've discovered in the intervening years since graduation
| that he was wrong, you should start assuming by the bottom
| of the first page.
| PaulHoule wrote:
| I told my thesis advisor that!
| Verdex wrote:
| I don't know about that. IIRC the creation of HoTT was
| because someone was notified that one of his important papers
| from earlier in his career had a non-trivial flaw in it.
|
| To be sure, reproduction is a bit easier in mathematics
| because you normally don't need exotic equipment, just time.
| However, that doesn't mean that people are necessarily taking
| the time to double check everything with sufficient rigor.
| nickpsecurity wrote:
| In the article below, they claimed one study said a third of
| the papers had a false theorem.
|
| https://news.ycombinator.com/item?id=17430577
|
| A number of papers on formal verification reported that
| restructuring their specifications for proof revealed
| problems they missed in the specification. Other works found
| through empirical testing that the specs didn't match
| reality. Many theories in physics changed that way.
|
| So, we should assume they're all false by default until the
| specs are reviewed, the proofs are checked, and there's
| empirical validation. If theoretical, its component theories
| must be built on observed truths instead of imaginary
| constructs. Then, we might trust the paper (i.e. mathematical
| claims).
| etrautmann wrote:
| this is overstating the case. Throwing out all publications is
| ridiculous. Yes, many fail to replicate, but not nearly all. A
| larger concern may be that many fail to generalize (e.g. mouse
| cancer cures fail to work in humans).
|
| It's true that it's hard to ignore all of the citations a
| reviewer suggests, but that's part of the role of an editor,
| and the response to reviewers (while being a frustrating
| exercise) can make it clear what is and isn't an appropriate
| citation, which serves as communication to the editor. Yes,
| this process is frustratingly low bandwidth
| stathibus wrote:
| How do you know? It's rare that anyone even tries to
| replicate scientific results outside of clinical medicine
| patall wrote:
| Exactly. What kind of editors does your field have? In my
| field, that kind of oversight would have been called out on
| twitter immediately.
| eigenket wrote:
| > I've also seen that during peer review a reviewer may give a
| list of publications they 'suggest' for the literature review,
| sometimes completely unrelated fields. If you choose not to,
| you will likely see your paper rejected.
|
| You are severely misrepresenting how things work here. A
| reviewer doesn't accept or reject a paper. A reviewer gives
| their response to an editor and the editor makes a decision.
| The editor knows who the reviewer is, and has access to the
| reviews and the author's response to them.
|
| If the editor sees the reviewer has suggested adding bunch of
| references to their own work, and has suggested rejecting the
| paper after the authors refuse this then the editor is going to
| know what is going on. They aren't completely stupid.
|
| > purposefully leak your own paper to get past the publisher
| pay wall
|
| Many papers in maths, physics and computer science appear on
| arXiv before being submitted for publication. In the fields I
| have worked in you can replace the "most" with "pretty much
| all".
| jwsteigerwalt wrote:
| I think you're giving the editors too much credit.
| eigenket wrote:
| I guess it depends on the journal. If you're publishing in
| some low tier or even predatory journal then yeah there
| might be problems like this with the editor but in good,
| respected journals the editors usually do their jobs
| properly.
| vladms wrote:
| > The truth is that almost all papers published across almost
| all fields cannot be replicated, full stop.
|
| Isn't it ironic that we can't replicate your assessment?...
|
| I personally have no idea how much papers can be replicated or
| not, but replicating an experiment would not guarantee that the
| idea/conclusion in the paper is sound (ex: paper extracting
| some data, observing a correlation and concluding it is an
| implication).
|
| I do agree that it should be made easier to replicate for the
| fields where this is possible, but there are many other
| problems.
| asah wrote:
| Is replication a big problem in computer science?
|
| If anything, I'd think replication has gotten _easier_ with the
| advent of open source and the expectation that top papers will
| include git repos which make it easy to reproduce. The advent
| of docker, locked-version-dependencies etc have all contributed
| as well. 10-20 years ago, we didn 't have this - but I also
| didn't see * widespread * problems even then: if someone
| published a result saying that technique <x> led to <y>
| performance gain, I thought it usually did?
|
| (that said, computer science research doesn't have the same
| money on the line as medicine and the "perish" part of the
| publish-or-perish equation isn't the same when the fallback for
| failed academics is still six figure salaries at tech
| companies)
| nradov wrote:
| What are the odds that in 2044 we will still be able to run
| the code published with CS papers today? If it's pure Java SE
| or ANSI C with no dependencies then it will probably still
| work, but anything more complex tends to decay pretty
| rapidly.
| bsdpufferfish wrote:
| Most CS papers are theoretical, and this is another reason
| why HN is wrong about requiring papers to publish code.
| newzisforsukas wrote:
| > The truth is that almost all papers published across almost
| all fields cannot be replicated, full stop.
|
| And yet we still have advances in technology. I wonder how this
| academic nihilism matches reality?
| mycologos wrote:
| Note that the article specifies that these cartels mostly feature
| participants from bad universities in countries that have really
| whacked-out academic reward mechanisms that arose because some
| powerful central government force that doesn't understand
| research productivity came up with a dumb way to incentivize it.
|
| Prestige is not a failsafe guarantor of quality, but it kind of
| is, in the sense that somebody with a reputation to protect is
| going to be more careful about coming across as "unserious", and
| doing stuff like this is a good way to get a bad reputation
| quickly.
| lupire wrote:
| "Prestigious" people just do it in a more refined way, managing
| perception instead of raw numbers.
| alephnerd wrote:
| We can safely assume MIT has a much higher quality output
| than King Abdulaziz University back in Jeddah.
|
| In general, State Flagships and Ivy Leagues+Ivy Tier Privates
| are largely comparable research output wise.
|
| The GP is right about prestige minimizing bad practices. Look
| at how diluted Northeastern's brand has become by trying to
| artificially inflating their prestige [1]
|
| [1] - https://www.bostonmagazine.com/news/2014/08/26/how-
| northeast...
| jampekka wrote:
| It happens everywhere to some extent.
| ineedasername wrote:
| > _powerful central government force that doesn 't understand
| research productivity_
|
| [1]*
|
| I'm sure they understand it. But their goal in doing this isn't
| really research productivity (though it would be a happy
| byproduct) it's to obtain prestige, or its illusion. And/or be
| able to tell the people of your country how great you're doing.
| "Our country is a rising powerhouse in field X, in the top 3 in
| the world and still rising!"
|
| It's internal propaganda. And externally targeted towards
| populations of other countries unfamiliar with the academic
| publishing system, which is most people. So for example the
| vast majority of people in the US that hear these claims by
| China, which are then echoed in click-bait headlines or simply
| taken at face value after a journalist checks something like
| the various HCR lists.
|
| It can even be propaganda, not from China, but from other
| countries. One of the studies was funded by the US State
| Department to an Australian group, thus obscuring a primary
| influence on the results of the study.
|
| Then even a single newspaper article can spark a dozen others
| citing it, propagating the message. All of these directly or
| indirectly rely on citations to support their headlines [2]
|
| https://www.science.org/content/article/china-rises-first-pl...
|
| https://www.wsj.com/articles/american-universities-continue-...
|
| https://money.usnews.com/investing/news/articles/2023-03-02/...
|
| https://www.reuters.com/article/idUSTRE72R6FQ/
|
| [1] The quality of this comment will be measured by metrics
| that I made up myself.
|
| [2] https://www.reuters.com/technology/china-leads-us-global-
| com...
|
| *This comment was funded by a grant from its author.
| maxglute wrote:
| >isn't really research productivity (though it would be a
| happy byproduct)
|
| Or the goal is research productivity and citation gaming is
| just the unhappy byproduct. Yes some motivated actors to
| playup PRC capabilities (ASPI/US think tanks) exist, but
| unless you think every western index on science and
| innovation, controlled for quality of citations (like Nature)
| are incentivized/coordinated to carry water for PRC's massive
| increase in research productivity in the last few years, then
| the parsimonious answer is PRC research productivity has
| gotten really good and world leading in some domains.
|
| Which should not be surprising because what OP fails to
| understand incentivizing citation gaming is a smart way to
| incentivize output, FAST, and at PRC scale output quantity
| has quality of it's own. If system spams research, and start
| off with only 2/10 good research, but leader is 3/5 good
| research, then PRC has nearly caught up. Refine to 3/10 and
| there's parity. Refine system to 4/10 and PRC leads. Emphasis
| being fast because PRC started focusing on seriously
| improving tertiary and R&D/S&T ~10 years ago and the goal was
| to develop capabilites fast by customary overproduction. You
| don't get that by slow careful growth, you do that by
| incentivizing easy KPIs everyone in sector can coordinate
| around and the product of said KPI is ENOUGH good research
| after byproduct of citation gaiming. Having a bunch of chaff
| leftover to get wheat is the point.
| blantonl wrote:
| A classic textbook example of perverse incentives and unintended
| negative consequences.
|
| Someone could write a paper on this.
| ulucs wrote:
| It's not like the "principal-agent problem" already exists in
| the literature
| achenet wrote:
| Goodhart's law: When a measure becomes a target, it ceases to be
| a good measure.
| xqcgrek2 wrote:
| Citations are simply a flawed metric for anything useful.
|
| A prominent paper with a false hypothesis can get many citations
| because people reference it when they refute it.
| bachmeier wrote:
| That's the example I often use. There are papers in prominent
| journals that draw negative citations.
|
| A related but much larger problem is that citations are a
| measure of the number of researchers working on a particular
| problem and nothing else. There was one topic in particular
| that had many researchers making small contributions, but all
| of those papers cited all the others, simply because they were
| written on that topic. Maybe the methods weren't the best or
| the conclusions weren't robust, but the only reason to cite a
| paper is because it is prior literature. It's not like you can
| refuse to cite papers because you believe they are of low
| quality.
| lkdfjlkdfjlg wrote:
| Once a measurement becomes a target, it ceases to be a good
| measurement.
| Tycho wrote:
| Can some billionaire not fund some sort of 'Academic Integrity
| Institute' non-profit
|
| - bust citation rings
|
| - catch plagiarism
|
| - flag statistical malpractice/mistakes
|
| - attempt replicate published studies/experiments
|
| Could give jobs to loads of young scientists and be a huge net
| positive for society.
| alephnerd wrote:
| What's the point?
|
| Most R&D is driven by Government Procurement or
| Commercialization anyhow, which can validate if shit works or
| not.
|
| A lot of the subfields with a replication crisis are those that
| aren't exactly "tangible", and as such don't have a
| quantitative way or vested interest in validating that stuff
| works as expected.
|
| If I'm researching a compound as a MatSE on a grant from the
| DoD or Applied Materials, it's on the assumption that tangible
| results will happen, and any attempt at fraud will be detected
| once commercialization or productionizing kicks in.
|
| At the end of the day, a field like Pure Math or Anthropology
| just doesn't have a commercial incentive that forces
| reproducibility because there isn't a tangible impact.
| lupire wrote:
| Why are taxpayers paying these people to tell us lies about
| things that don't matter? That's doubly wrong, even worse
| than truths that don't matter.
| alephnerd wrote:
| > Why are taxpayers paying these people to tell us lies
| about things that don't matter
|
| Research that gets federal grantmaking attached to it
| essentially gets validated at the production stage.
|
| Fields dealing with a severe replication crisis are also
| those which aren't validating or productionizing research.
| j7ake wrote:
| The solution is to abandon quantitative university rankings.
| They're obviously not useful.
| vladms wrote:
| What about taking them with a grain of salt? Just because
| something is approximate that does not mean it is not useful.
|
| I never got this obsession with detailed rankings. Sure "above
| average" and "under average" makes sense, but position x or x +
| 1 out of 10k does not make much sense anymore (and by
| similarity if you are 1 or in the first 1k out of 10k could be
| similar)
| dev1ycan wrote:
| I can't wait until AI is used to take away thousands if not
| dozens of thousands of masters and PHDs
| WhyOhWhyQ wrote:
| Why's that? Just because you're vindictive or something?
| kzz102 wrote:
| Citation numbers doesn't provide any useful information for a
| serious researcher. It's all a game for administrators.
| lupire wrote:
| With computer and AI search, the obvious fix is to cite
| _everything_ related to the new work, up to thousands of
| citations.
| attractivechaos wrote:
| This is also why I think paid review won't work although many in
| academia advocate that. If paper mill remains a problem now,
| review mill will be much worse when journals start to pay
| reviewers.
| jampekka wrote:
| Do many in academia advocate that? I agree that paying for
| reviewing would make it even worse.
|
| Money in general is a bad incentive if honesty and rigor are
| wanted.
| attractivechaos wrote:
| > _Do many in academia advocate that?_
|
| That was my impression whenever paid review was mentioned.
| Nonetheless, it is also possible that more who dislike the
| idea were not voicing themselves.
| amadeuspagel wrote:
| Google has a better university ranking[1], with more experience
| in withstanding this kind of manipulation and no need to conform
| to expectations that Ivy League universities must be at the top.
|
| [1]: https://www.google.com/search?q=university
| russfink wrote:
| Is the reward of doing this worth the risk of being blacklisted
| by the metrics keepers?
| nsagent wrote:
| It's not just cartels, but also self-citations. For example, the
| original preprint of this paper [1] has 5 self-citations, with
| the version accepted to EMNLP increasing that to 10. This kind of
| behavior is sadly pretty common now and when you mention it to
| conference organizers they just shrug.
|
| [1]: https://arxiv.org/abs/2304.02210
| kzz102 wrote:
| Speaking as a researcher, I think the whole citation business was
| invented by and consumed by administrators. Researchers don't
| need citation numbers to discover new works and evaluate people.
| They are perfectly fine asking their circle of friends, using
| journal quality and author fame as a crude proxy, and (most
| importantly) just read the damn papers and reach their own
| conclusions.
|
| Administrators needed some objective metric that allow them to
| make decisions themselves without relying on the personal
| opinions of researchers, whom they don't trust. This is magnified
| by rankings that like these metrics for the same reason,
| publishers trying to promote journals, and database access
| sellers.
|
| Unfortunately, this has now circled back to the researchers,
| because the administrators are making hiring / promoting
| decisions based on these metrics, and people started to game the
| system. However, I think in many circles, there is a healthy
| culture of rejecting this nonsense.
| zelphirkalt wrote:
| I find it kind of ridiculous and depressing, that we are playing
| these kind of games in academics and risk careers of bright mind
| and scientific progress. What a waste of potential and time.
| rcpt wrote:
| I left academia a long time ago and now I buy ads for one of my
| old papers for fun. I still get citations from time to time which
| is nice.
| cherryteastain wrote:
| I have seen all sorts of crooked behaviour in academia, such as
| reviewers demanding authors to cite their unrelated papers.
| However, this article's main premise seems to be predicated on
| another assumption that is also terrible for academia - elitism.
| Specifically, this quote
|
| > There were people that published in journals that no serious
| mathematician reads, whose work was cited by articles that no
| serious mathematicians would read, coming from institutions that
| nobody knows in mathematics
|
| is really bad. It suggests one can be a 'serious' mathematician
| only if you went to XYZ universities and published in ABC
| journals. This elitism is the main reason why people try to game
| the system in the first place, because they know that they can't
| advance in their careers unless they build that perception of
| prestige. The single minded focus on prestige is then what
| smothers the real science.
___________________________________________________________________
(page generated 2024-01-31 23:01 UTC)