[HN Gopher] Argonne National Lab is attempting to replicate LK-99
___________________________________________________________________
Argonne National Lab is attempting to replicate LK-99
Author : carabiner
Score : 306 points
Date : 2023-07-29 02:36 UTC (20 hours ago)
(HTM) web link (www.science.org)
(TXT) w3m dump (www.science.org)
| hmaxwell wrote:
| EEVblog's input on this is actually hilarious, if you haven't
| seen: https://www.youtube.com/watch?v=QHPFphlzwdQ
| harrid wrote:
| Being loud and obnoxious is at best orthogonal to being
| correct. I'm not on either side on the LK99 question)
| hmaxwell wrote:
| What part of the video I linked to was "loud" or "obnoxious"?
| I am not sure if you even watched the video...
| ehnto wrote:
| Are you Australian by chance?
|
| He's really laying on the sarcasm during the intro, I could
| totally see that getting misconstrued by someone not
| familiar with our flavour of sarcasm.
| hmaxwell wrote:
| Despite not being of Australian origin, I can understand
| how one could interpret his initial remarks as being
| tinged with sarcasm. In my personal analysis, however, I
| found his comments to be injected with a considerable
| amount of wit and humor.
| csydas wrote:
| as a non-australian but native English speaker, I found
| the video annoying because it's padded and the
| jokes/sarcasm weren't really all that funny. If you trim
| out his YouTube-OverReactions, I think the video goes
| down to like 1.5 minutes in my mind and still conveys the
| same information, perhaps even better.
|
| I get probably this is a popular video maker in whatever
| sphere he's in, but as an outsider, it's difficult to get
| into the video because of what I attribute as the
| presenter's YouTube personality.
|
| (also it very much so bugs me to see people using Chrome
| and see all the nags for notifications, the random
| extensions/features, etc. And any browser without uBlock
| origin throws me as it takes me a minute to realize why
| all the websites look so bad, but that's just a personal
| issue :) )
| ArnoVW wrote:
| Having skipped to his analysis of the video, I find him
| pretty convincing.
|
| He explains why the video is bunk using information provided
| by the lab, and basic physics. And then 'replicates' using
| his own sheet of metal and a magnet.
|
| Also, he raises the fact that it's a commercial lab.
|
| I've downgraded my level expectations a couple of orders of
| magnitude. But hey, let's hope for a miracle.
| seiferteric wrote:
| Just watching it. He brought up something I noticed also,
| showing the magnet moving the hanging copper plate with the
| material deposited on it... In like basic physics class you
| go over eddy currents induced by moving magnets, isn't that
| all we see there? I can't understand how this showed
| anything other than that?
|
| Thunderfoot comment though seems ridiculous. It would be
| irrelevant because it is a ceramic??? lol. They proposed a
| mechanism how it works! Obviously this would just be the
| first step. That's like saying the first transistor was
| irrelevant because we don't use that design anymore.
| harrid wrote:
| I liked his actual point, too! But it's being devalued by
| his demeanor
| tux3 wrote:
| It shouldn't devalue the point. As you said, it's
| orthogonal.
| kykeonaut wrote:
| Gotem
| mikenew wrote:
| I've been wondering what the "hanging copper plate"
| demonstration would look like if you were just seeing eddy
| currents. I kind of assumed you wouldn't see much movement
| because otherwise that makes the whole demonstration seem dumb.
|
| ...and based on that demo, it really does seem like a dumb way
| to demonstrate your miracle material. Really doesn't inspire
| confidence.
| guerrilla wrote:
| I find it very disappointing if not worrying that this wasn't
| picked up by more people... This is something any of us could
| have noticed.
| willis936 wrote:
| Most people haven't seen that video. Most people have only
| seen the levitation video.
| Khaine wrote:
| [flagged]
| einpoklum wrote:
| I asked NotReallyChatGPT to evaluate this comment and it told
| me to downvote.
| carabiner wrote:
| It's over for this compound I'm pretty sure, but it is cool to
| see the range of responses from professionals in this field. Some
| like Jorge Hirsch (superconductivity researcher and came up with
| the h-index) think it's a joke (and his work is cited by the
| Korean scientists), and then some guys at a national lab are
| trying to synthesize it.
|
| Hirsch is also apparently a bit of a shitposter at age 70 and was
| banned for flaming people on arxiv last year.
| ethanbond wrote:
| That's how you know you're doing actual groundbreaking science
| doctoboggan wrote:
| I forget the actual quote, but there is a saying that if an old
| esteemed scientist says something is possible, they are almost
| certainly right. If they say something is impossible, they may
| be wrong.
| iancmceachern wrote:
| "1. When a distinguished but elderly scientist states that
| something is possible, he is almost certainly right. When he
| states that something is impossible, he is very probably
| wrong. 2. The only way of discovering the limits of the
| possible is to venture a little way past them into the
| impossible. 3. Any sufficiently advanced technology is
| indistinguishable from magic." A. C. Clark
|
| I particularly love #3
| the-dude wrote:
| Related : "Science progresses one funeral at a time"
|
| https://en.wikipedia.org/wiki/Planck%27s_principle
| lamontcg wrote:
| You'll see a bunch of old scientists saying a legitimate
| revolutionary breakthrough is bullshit, but you didn't see
| the other 99 times they called bullshit correctly.
| Evidlo wrote:
| That's why it's called LK-99
| jcranmer wrote:
| Actually, the 99 apparently comes from its year of
| discovery, 1999.
|
| No, I don't know why it took over two decades for their
| discoverers to figure out it was a room-temperature
| superconductor.
| jacquesm wrote:
| One to go then.
| ghaff wrote:
| Don't know if it's the original source but Arthur C. Clarke
| had an even more definitive quote along those lines.
| valine wrote:
| It's not over until multiple labs fail to reproduce it. A
| definitive no will take months.
| Animats wrote:
| > It's not over until multiple labs fail to reproduce it. A
| definitive no will take months.
|
| Yes. Some may remember the problems with cold fusion. I once
| went to a packed talk at Stanford where some physicists and
| chemists were trying to reproduce the effect. They mentioned
| that in the first attempts, they had radiation alarms set up
| in case the thing suddenly started emitting substantial
| amounts of radiation. After a while, it became clear that the
| effect, if it existed at all, was small, somewhere below
| twice background radiation, so it wasn't high risk. They
| discovered that people moving around the apparatus affected
| the results; humans have a lot of water and are neutron
| reflectors. Finally they did the experiment inside a "neutron
| cube", a box made of lead bricks, to eliminate stray
| neutrons. No neutron generation measured inside the cube.
|
| They worked hard for months getting to a definitive "no".
| xorbax wrote:
| Right. But this is social-media science, so there needs to
| be an answer by Saturday or - at _worst_ - Sunday.
|
| Entertain our sudden fickle intellect! We're paying
| attention _now_ , so you should all hop to, regardless of
| theory or evidence!
| lordnacho wrote:
| But would a yes be immediately obvious? Is there a test for
| superconductivity that doesn't easily show false positives?
| rendang wrote:
| Unless in the meantime the authors realize they made some
| mistake in their analysis & retract their claims.
| zarzavat wrote:
| Even if it doesn't turn out to be a superconductor, it is
| at least strongly diamagnetic, so it opens avenues for
| future research either way. There's a potential scenario
| here where this doesn't turn out to be a superconductor,
| but it nonetheless leads to a superconductor further down
| the line.
| XorNot wrote:
| The one wrinkle here is if the sample we saw actually was
| pyrolytic carbon and someone simply mislabeled a vial
| when looking for something to shoot the video with.
|
| At the extreme end of big news, stupid things can happen
| (I've worked in a lab: people suck at labelling things
| they make).
| willis936 wrote:
| Where would this carbon come from? I think being able to
| produce an element out of thin air is more impressive
| than STP superconductors.
| jtriangle wrote:
| If it is pyrolytic carbon, they've somehow significantly
| enhanced the dimagnetism of it, which is remarkable.
| Usually that stuff only barely floats, and only when it
| is very thin.
| carabiner wrote:
| https://i.kym-
| cdn.com/photos/images/original/002/555/116/b2e...
| konschubert wrote:
| This feels wrong. The burden of proof should be on the one
| making the extraordinary claims.
| krashnburn200 wrote:
| They are literally willing to mail out samples of their
| claimed superconducting material to labs for independent
| testing and verification.
|
| Exactly how much more proof than "Here, have a super
| conductor we made" do you want exactly?
|
| The only thing required is a little patience.
| anabab wrote:
| [dead]
| molticrystal wrote:
| While I am very skeptical, when working with what seems to be a
| fluke, it is more important to test the claim then to test a
| reproduction.
|
| Figure out how to send an independent lab's equipment and
| personal there or a sample of the substance to an independent lab
| that verify and if true. Also ensure that they can also do
| material analysis such as X-Ray spectroscopy & diffraction , and
| a battery of other stuff if a miracle did occur.
|
| For all we know, this could be one of those accidents of
| sloppiness that introduces a particular containment that makes
| everything line up as a super conductor that nobody else will
| easily reproduce.
| hooande wrote:
| why don't they just repro their own experiment? If they can do
| it twice, then it isn't a fluke. at worst it's something
| specific to their lab. if they can't do it a second time, then
| the issue is settled.
|
| I'm sure it will take a lot of time and money to run everything
| again. but all of earth seems willing to give them whatever
| resources they need.
| anonylizard wrote:
| It doesn't particularly matter if it is reproducible, as long
| as they have 1 working superconductor that they can hold in
| their hand (Room temperature/pressure), its a nobel prize.
| Other people can figure out how to reproduce it since it is
| clearly possible.
|
| The challenge is proving they have even that 1 sample.
| highwaylights wrote:
| This.
|
| If it was some kind of fluke and is not easily reproducible
| _but_ the sample exists and _is_ a room-temperature super-
| conductor then it becomes a reverse engineering problem.
| delecti wrote:
| Considering the magnitude of the discovery and the relatively
| easy steps to reproduce the process, they would have to be
| outrageously lazy to have put the paper out without doing it
| a second time. In absence of a statement that they only tried
| it once, I think it's fair to assume they at least verified
| they can reproduce it locally.
| generalizations wrote:
| > For all we know, this could be one of those accidents of
| sloppiness that introduces a particular containment that makes
| everything line up as a super conductor that nobody else will
| easily reproduce.
|
| Maybe we should call this the Hyde Phenomenon.
|
| > Jekyll's involuntary transformations increased in frequency
| and required ever larger doses of the serum to
| reverse....Eventually, the supply of salt used in the serum ran
| low, and subsequent batches prepared from new stocks failed to
| work. Jekyll speculated that the original ingredient had some
| impurity that made it work.
|
| https://en.wikipedia.org/wiki/Jekyl_and_Hyde#Plot
| kurthr wrote:
| I think all they have to do is send a sample to another lab to
| analyze.
|
| Apparently, people feel it's pretty easy to make, but in any
| case, if they have a sample they can see if it has the claimed
| properties and analyze what it's made of.
| sanxiyn wrote:
| Condensed Matter Theory Center (a Twitter account affiliated
| with University of Maryland) says they will send samples.
|
| https://twitter.com/condensed_the/status/1684960318718406656
| Simon_O_Rourke wrote:
| > For all we know, this could be one of those accidents of
| sloppiness that introduces a particular containment that makes
| everything line up as a super conductor that nobody else will
| easily reproduce.
|
| I get that terrible sense too - either the original studies
| were in error, or some one-off fluke makes it difficult to
| reproduce.
|
| On the upside however, and I'd caveat this as not being fully
| in the know about the art of the possible in solid state labs,
| it seems that the material isn't that awfully difficult to
| produce given an appropriate lab and equipment. Hopefully we'll
| know one way or the other shortly.
| cryptonector wrote:
| I believe MIT researches are already on the ground to do just
| this, and probably other teams are there too.
| Tade0 wrote:
| To me the main takeaway is that there has had to be _some_
| advancement in this field over the past two decades if claims of
| sufficiently-close-to-ambient superconductivity are currently not
| dismissed outright.
|
| Maybe we'll have maglev everything by the time I'm old and gray.
| ehnto wrote:
| Not room temperature but very cool to see practical
| implementations making progress in the form of the SCMaglev
| line in Japan.
| azernik wrote:
| Despite some criticism of the original work, they seem to have
| been very conscientious about making replication easy.
|
| ===========
|
| Nadya Mason, a condensed matter physicist at the University of
| Illinois, Urbana-Champaign says, "I appreciate that the authors
| took appropriate data and were clear about their fabrication
| techniques." Still, she cautions, "The data seems a bit sloppy."
| aaron695 wrote:
| [dead]
| shusaku wrote:
| > On the other hand, he says, researchers at Argonne and
| elsewhere are already trying to replicate the experiment. "People
| here are taking it seriously and trying to grow this stuff."
|
| The submitted title has been heavily editorialized. That's the
| only relevant part of the article, and that's far from implying
| that there's a concerted effort at Argonne.
| Dylan16807 wrote:
| > The submitted title has been heavily editorialized. That's
| the only relevant part of the article, and that's far from
| implying that there's a concerted effort at Argonne.
|
| What's wrong with the submitted title (other than being very
| narrow)? It just says they're attempting to replicate, which
| they are.
|
| If anything I'd say that "taking it seriously" is _stronger_
| language, and the submitted title is slightly _underselling_
| it.
| mhb wrote:
| Being _very narrow_ is exactly what 's wrong with it. That's
| not what the article is about.
| Dylan16807 wrote:
| That has nothing to do with the sentiment I was addressing,
| though. That is a different complaint.
|
| I agree that it was too narrow, but it wasn't exaggerating.
| mhb wrote:
| The article says many things. Cherry-picking one
| incidental detail to use as the title is editorializing,
| regardless of whether it is accurate. Do you think an
| equally good title might be "If you've ever had an MRI,
| you've lain inside a big electromagnet made of
| superconducting wire."?
| Dylan16807 wrote:
| I didn't say it wasn't "editorializing".
|
| I objected to shusaku implying that this headline was
| _exaggerating_.
|
| I don't know how to make this clearer.
|
| > Do you think an equally good title might be "If you've
| ever had an MRI, you've lain inside a big electromagnet
| made of superconducting wire."?
|
| Well, if you really want to get into this, _even though I
| acknowledge it 's cherry-picking_:
|
| If that fact is what the submitter cared about, and this
| was the only page on the internet talking about that
| fact... it wouldn't be a terrible idea.
|
| The HN guidelines aren't great here. You can make your
| own blog post about something and link that, but it will
| probably get replaced to the "original" link even if the
| "original" link had a completely different focus.
|
| Sometimes you have to pray that whoever wrote the
| original splits it up into different articles themselves.
| Or that they write on twitter, so you can link to a
| specific tweet.
|
| Sometimes a newspaper will put three completely separate
| stories into the same article, and trying to link the
| second or third story on HN risks the title getting
| replaced with something utterly unrelated.
| mkl wrote:
| From the HN Guidelines [1]: "please use the original title,
| unless it is misleading or linkbait; don't editorialize."
|
| The editorialised title here is misleading, as it's not what
| the article is really about, it's just mentioned in passing.
|
| [1] https://news.ycombinator.com/newsguidelines.html
| tigershark wrote:
| I would say that the best part of the article is: "Some of you
| haven't had blisters from overusing your pestle and it shows."
| carabiner wrote:
| Is the original title better? "A spectacular superconductor
| claim is making news. Here's why experts are doubtful."
| Standard nothing title, "a thing happened." I zeroed in on the
| pithy part of the article, which is SOP on HN. The subtitle is
| just "skepticism abounds" which we already know. What some,
| including me, didn't know is that legit USG labs are studying
| it, and that's what makes it news.
| mhb wrote:
| Yes. And article is also two days old. Ancient for this.
| nemo44x wrote:
| My young son came to and excitedly asked, "Daddy, is it true that
| LK99 is an ambient pressure, room temperature superconductor?"
|
| I replied, "It might be or it might not be..." and I looked at
| him and grinned and said "but you still have to clean your
| bedroom!"
|
| "Oh dad!" He said as he scampered off.
| sanxiyn wrote:
| I wrote a comprehensive thread on the current situation here. It
| is somewhat long and hard to summarize, sorry.
|
| https://twitter.com/sanxiyn/status/1685094029116297216
| gausswho wrote:
| I love the self-deprecating humor of the introduction, esp the
| ice cream bit. Well done.
| doctoboggan wrote:
| Can you post a link to one of the mirrors that lets logged out
| people read the thread?
| krasin wrote:
| Here: https://nitter.net/sanxiyn/status/1685094029116297216
| amai wrote:
| Reading all this I'm starting to believe the paper was
| written by chatGPT.
| callalex wrote:
| Paywall
| krasin wrote:
| Same thread on nitter:
| https://nitter.net/sanxiyn/status/1685094029116297216
| carabiner wrote:
| Yes this is an excellent thread. You have done some of the only
| on the ground reporting from Seoul. The takedowns you linked of
| the paper, CMTC giving it an F, are damning. There's one legit
| professor Kim from William & Mary who had a tiny part of this,
| but more and more, it looks like it's the output of a few
| cranks out of a "Q-Centre." The bulk of the work took place
| before Kim's involvement and he's listed on 1 of 3 papers so
| far. He's the only thread of legitimacy in this whole thing,
| and it's a thin one. The 5 other guys are just loony.
| brucethemoose2 wrote:
| Why didn't the researchers ship their "working" sample to another
| lab? I can't imaging _any_ institution turning down the chance.
|
| Is it unstable?
| sanxiyn wrote:
| They are willing and eager to ship samples. (Search for my
| other comments.)
| brucethemoose2 wrote:
| >
| https://twitter.com/condensed_the/status/1684960318718406656
|
| Oh wow. If true, then that's that.
|
| If they are stable, the original samples will validate, or
| they will not.
|
| The researchers could wave their hands and claim the sample
| deteriorated, but I think that would still end optimism for
| replication.
| scythe wrote:
| Norman's complaints that lead atoms are too heavy do not seem
| consistent with the composition of other known superconductors.
| As far as I know, the most widely used cuprate superconductor is
| bismuth strontium calcium copper oxide (BSCCO), which was used
| for the world's first superconducting power line:
|
| https://en.wikipedia.org/wiki/Holbrook_Superconductor_Projec...
|
| >What's more, lead and copper atoms have similar electronic
| structures, so substituting copper atoms for some of the lead
| atoms shouldn't greatly affect the electrical properties of the
| material
|
| The lead (II) ions claimed have an even number of electrons.
| Copper (II) has an odd number. Or if copper (I) is present, then
| the charge itself is different. Again, this is just a very
| confusing argument to hear from a physicist.
|
| >First, the undoped material, lead apatite, isn't a metal but
| rather a nonconducting mineral.
|
| The cuprate and iron superconductors are not metals either. In
| fact some are Mott insulators (materials with unexpectedly high
| resistance) under normal conditions.
|
| There is a case for skepticism about LK-99, but it isn't this
| one.
| bawolff wrote:
| One of the common threads seems to be that other scientists think
| that lk-99 is not a super conductor but just strongly
| diamagnetic.
|
| As a non physicist i wonder if that is useful in and of itself?
| Skimming wikipedia it doesn't seem like there are that many
| strongly diamagnetic materials. Would discovering a new one still
| be a big discovery (just not earth shattering)
| jacquesm wrote:
| That depends on how the diamagnetism comes about. One way in
| which it could come about which would still make it a big
| discovery is if the material is locally superconducting but not
| globally. That would give you diamagnetism without
| superconductivity on a usable scale and might in turn open the
| door to modifying the material to increase the size of the
| superconducting regions. Pure speculation, obviously, the
| bigger chance is that if the material is 'just' diamagnetic
| that it is simply diamagnetic in the same way in which other
| materials are (all particles of which the samples are made are
| constructed such that all electrons are paired).
| tux3 wrote:
| It would be cool, but there are no world-changing implications
| to a new diamagnet made of lead. It's just kinda neat if you
| find one.
| fnordpiglet wrote:
| I don't know how that explains the resistances measurements
| though? Unless they're complete fabrications don't they imply
| superconductivity?
| tux3 wrote:
| I truthfully don't know.
|
| A few people suggested they might have measured an
| insulator, the threshold where resistance changes being the
| breakdown where current starts flowing.
|
| I'm skeptical either way, I think it's too early to say.
| fnordpiglet wrote:
| I see. V=IR=0 means R can measure as 0, but they did
| measure small current. Presumably they know the voltage.
| That seems odd if they can't measure the resistance
| accurately if their instrumentation is rated to measure
| that current.
| adrian_b wrote:
| It is not possible to determine that a resistance is
| exactly zero through direct measurement, because any kind
| of probes that can be used to contact the material will
| have a high contact resistance, which will vary from one
| probe contact to another.
|
| Even when the resistance is measured with a 4-point
| probe, the only conclusion that can be reached is that
| the resistance is smaller than the experimental error.
|
| However it should be possible to determine that the
| resistance is zero through various indirect effects, for
| instance by establishing a closed current in the material
| and verifying that it does not decrease in time.
|
| Even such experiments must be done carefully because some
| materials with unusual magnetic properties may behave
| apparently in a similar way to superconductors in some
| experiments, e.g. both a magnetized ferromagnetic
| material and a superconductor ring carrying a current
| will have a remanent magnetic field that is constant in
| time and in both cases the magnetic field will disappear
| when the temperature is raised above a threshold.
| highwaylights wrote:
| My understanding is that taking the paper at face value it
| seems like they're saying the diamagnetism was an encouraging
| piece of supporting evidence beyond their other rationale for
| what they believe they've found. I'm not enough of a
| physicist (or one at all) to speak to that though so I guess
| I'll just sit here wit ma popcorn for the next week with
| everyone else
| carabiner wrote:
| Whatever happens, this goan be a great 4,000 word New Yorker
| article, or Netflix kdrama.
| evrimoztamur wrote:
| ...or the next BobbyBroccoli special like the one on Hwang Woo-
| suk, the man who faked human cloning
| (https://m.youtube.com/watch?v=ett_8wLJ87U).
|
| Not claiming the researchers of LK99 are intentionally faking
| their results, by the way!
| jimkoen wrote:
| I find it really cool that this whole process didn't involve
| scientific publishers at all. Paper was submitted to Arxiv, there
| was Turmoil. Now a national lab is reviewing it, all without a
| Journal submission.
| oldgradstudent wrote:
| > I find it really cool that this whole process didn't involve
| scientific publishers at all.
|
| IIRC, the 1989 cold fusion fiasco did not involve scientific
| publishers as well.
|
| Not saying this is a similar fiasco, just that 34 years ago it
| was done via a press conference and circulating drafts rather
| than arXiv.
| xorbax wrote:
| I don't think LK99 is a superconductor
|
| But as an experimental chemist who got a PhD and works at a
| national lab, if the twitch/Twitter streamers _don 't_ produce
| a semiconductor, it says nothing
|
| Experiments and processes are hard and particular. You're
| expecting to produce a nanoscale material through macroacale
| processes. Chemistry can be like producing a microchip with a
| ballpeen hammer. "I hit it seventy times in thirty seconds in
| 3.8mL acetone and it just forms a single crystal structure. Do
| not hit it seventy-one or sixty-nine times."
|
| Chemists are experts at clearly explaining their procedure
| while leaving out the meaningless keystone detail that will
| only take a year or two to suss out if you're a clever
| experimentalist.
|
| And this isn't a clearly outlined procedure.
|
| But, again, I think the best case scenario is to watch an
| ambitious amateur do materials synthesis and see it work less
| perfectly than he intended. Maybe he and they stick around for
| the other 5 or 10 or 150 weeks as he does the same thing again
| to make sure the problem is nature rather than technique.
|
| If bet-placing is the fashion, shall we place bets about
| tenacity and thoroughness?
| swader999 wrote:
| Breaking bad showed us all how hard chemistry is. Still
| rooting for the after hours twitch/Twitter hackers.
| iancmceachern wrote:
| Thank you for this, especially the hammer analogy.
|
| For a humble mechanical engineer its very helpful to make
| hammer analogies.
| wrp wrote:
| My dad was an Organiker and he used to go on about the huge
| amount of lab-craft knowledge you needed beyond what is
| communicated in reports.
|
| One of his favorite movies was _It Happens Every Spring_
| (1949), about a chemist who produces a non-reproducible
| compound.
| wahern wrote:
| I wonder if there's a language barrier. I can't judge for
| myself--neither the substantive content nor the biographies--
| but hypothetically the principle investigators might not have
| strong (or any) English language proficiency, relying on a
| younger, less experienced peer to write the paper.
|
| As someone who sucks at languages, I can appreciate that
| learning new languages does not come easy to everybody, or
| even to most people, especially in older age. If it seems
| like all reputable scientists have some English proficiency,
| that's a selection effect--within the English-speaking world
| in particular you're not going to bump into or even hear much
| about those who cannot speak or write English. AFAIU there's
| a substantial amount of high quality research in, e.g., Japan
| that is invisible to non-Japanese speakers. Japan's
| scientific establishment long predates English as a global
| language, unlike China; and unlike many European countries
| but similar to the United States there was never a widespread
| norm of learning foreign languages even among the
| professional classes, AFAIU. Though Korea is a younger
| industrialized country, I wouldn't take for granted that the
| principal discoverers have any English proficiency,
| especially considering that they would be late middle-age or
| older if they were working in the lab in 1999.
| porknubbins wrote:
| There might be a body of historical purely Japanese
| scientific research, but I know some Japanese physicists
| who all speak English well enough, and have heard others on
| podcasts who do also. It would be serevely limiting to not
| speak English when that is the language of modern science.
| I don't know how far back you would have to go to find
| monolingual scientists but I would guess several decades at
| least.
| Sharlin wrote:
| You realize there's an incredibly obvious selection bias
| in your sample?
| xorbax wrote:
| I find it sort of implausible that somebody high in their
| field is totally ignorant in English, if only because so
| many papers are only published in English.
|
| I've known so many researchers who apologize every other
| sentence because they say they've only read English and I
| would never had known otherwise.
|
| Which is to say - it'd be wierd to say "well I don't know
| English and I don't know anybody who does oh well". But,
| anyway, the papers were published on aeXive in...
| rtpg wrote:
| There are plenty of Japanese scientists who are _at
| least_ extremely uncomfortable in English. Plenty of
| people who struggle to write their own stuff in English
| and get it corrected a lot. Though I think that beyond a
| certain level you have to be interacting internationally
| to get anywhere in your career so you're gonna be forced
| along.
|
| Plenty of people would learn Latin to consume some
| content, but doesn't mean they are proficient in it
| itself, would be another example.
|
| Having said that, I think that in the research field that
| merely ends up with "very confusingly written papers with
| mistakes large enough to hide process errors" more than
| "cannot communicate any info in English". But there are
| plenty of people who get their doctorate and their only
| English-language output is their final papers in
| themselves.
| peyton wrote:
| It's really hard to see a language barrier affecting the
| sample synthesis instructions. It's a few short paragraphs
| of "mix two things together and put them in the oven."
| bohadi wrote:
| It is amusing so many commentators cannot communicate
| novel materials science in Korean to the layperson.
| Where's your zero resistivity sample?
|
| The paper is just a guide. Who would replicate likely
| needs to contact the authors to understand the implicit
| details of the recipe. That is process knowledge. There's
| a good reason world-leading lithography happens in that
| region. With luck, researchers and engineers will be able
| to refine and industrialize the process these coming
| years.
| xorbax wrote:
| I dunno, I'd be suspect of the argument was that the
| directions were perfect in Korean and merely totally hidden
| from industrious Americans.
|
| Japanese and Korean professionals speak English. There will
| be one researcher for whom explaining the language is a
| minor dalliance.
|
| I work in a very minor polymer materials field. There are
| two Japanese researchers who cheerfully banter. If somebody
| had hidden some transformative work behind their language,
| even as a student, they would have talked about it
| straightaway, if only for the conversation and a bit of
| competitive intellectual football. Can the foolish American
| grad student reason around it? Let's see!
|
| The twitch channel was packed with interested Koreans. If
| there had been the tiniest bit of confusion there would
| have been 200 translations from which to utilize the
| Translation Of The Crowd.
| wahern wrote:
| Well, Japan is the _only_ mature industrial nation other
| than the United States where a majority of science papers
| are published in domestic journals[1][2], many of which
| are predominately or solely Japanese language[3], whereas
| international journals--including international journals
| in Japan--are invariably English.
|
| _Of_ _course_ the Japanese scientists you 've met speak
| English. People who don't speak English _well_ _enough_
| are not generally going to participate in events or
| situations where they would need to rely on English-
| language speaking skills. That someone apologies for
| their bad English is, of course, irrelevant; if it were
| truly bad you wouldn 't have encountered them. Likewise,
| Japan has many international science programs, forums,
| and conferences; and they'll be filled with proficient
| Japanese English speakers, but Japan is a huge nation so
| it says little necessarily about broader patterns.
|
| I don't doubt that a majority of professional research
| scientists in Japan have some proficiency in English.
| Especially the top scientists; even in Japan "impact"
| would undoubtedly be heavily biased toward publication in
| prestigious international journals. But there's plenty of
| literature on the internet that describes the
| _relatively_ poor English-language proficiency
| (especially conversationally or with verbiage outside
| that in their technical field) and poor English-speaking
| research community integration of the broader Japanese
| scientific establishment. See, e.g., "Japanese materials
| scientists' experiences with English for research
| publication purposes".
|
| I first encountered this cultural dynamic when
| researching the tilting of the Millennium Tower. While
| friction piles weren't first used in Japan, Japan was
| where friction piles were first heavily researched and
| employed for large buildings, including skyscrapers.
| People kept saying friction piles were idiotic in
| earthquake-prone, sandy soils, but that's precisely the
| environment where they were first researched and from
| whence the global engineering community's confidence
| sprouted. However, AFAICT much of that research was
| published in Japanese, even through the late 20th century
| and (IIRC) even early 21st century. Was that Nobel-worthy
| science? No, but it was very useful applied science; and
| applied science which was highly consequential to global
| engineering and architecture, notwithstanding that much
| of the substance of it existed on the other side of a
| Japanese language barrier.
|
| [1] See
| https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8024886/
|
| [2] But see, e.g., China and Brazil where domestic
| publication dominates and even outpacing international
| publication: https://www.digital-
| science.com/blog/2018/11/japan-collabora...
|
| [3] E.g. many of the publications by the The Japan
| Institute of Metals and Materials are only or
| predominately in Japanese. Some domestic publications
| require summaries and tables in English but permit the
| text to be in Japanese; e.g. the Journal of the Society
| of Materials Science.
| jacquesm wrote:
| > People kept saying friction piles were idiotic in
| earthquake-prone, sandy soils, but that's precisely the
| environment where they were first researched
|
| Friction pilings have been in use in NL for 100's of
| years.
| TheAceOfHearts wrote:
| Something I've also been thinking about is that we don't
| appear to have any data on the success rate. Maybe the
| samples of LK99 are the best that the team has been able to
| produce, but the process requires further refinement and
| experimentation.
| xorbax wrote:
| It's a little strange that they don't do multiple
| grinding/crystallization when their _best_ sample
| produces...gimpy diamagnetism
|
| If it's been 1000 experiments, do another. I'm not
| convinced that 1000 experiments over 24 years is all that
| intense if you actually do the math.
| cryptonector wrote:
| 1000 / 24y ~= 41/y, which after vacation and other things
| is probably closer to 1/wk. If all their experiments
| involve easy-to-synthesize substances (where easy == no
| more than 3 or 4 days) then it's not that intense, no.
| But you really have to be quite confident that this
| approach will pay off to put 2+ decades into it.
|
| I suspect if they really did 1,000 experiments then they
| often did 2/wk because I don't think they ran all of them
| continuously over 20+yrs.
|
| Speaking of which, can we see their notes on the other
| 999 experiments that didn't work out? That would be very
| useful, especially if LK-99 turns out not to be awesome
| but is good enough to hint that this approach is worthy
| of more research -- there's 999 compounds not worth
| testing, right? (Well, maybe maybe. If the physics of
| LK-99 come to be understood, then maybe some of the other
| 999 experiments might need to be re-examined.)
| emtel wrote:
| > if the twitch/Twitter streamers don't produce a
| semiconductor, it says nothing
|
| If you're referring to Andrew McCalip, he's been very clear
| that he is not claiming to be doing a serious replication
| attempt, but he also does zero-g/orbital chemical synthesis
| for a living so its not quite fair to characterize him as a
| random twitch streamer.
| godelski wrote:
| As an academic, I have not so secretly been enjoying the shit
| storm of revelations lately. Science got along fine for
| thousands of years without Journals and Conferences deciding
| what's "correct" and not. It's been a failed experiment.
| Journals started as a way to help distribute works to other
| researchers to improve upon the old method of just mailing it
| to one another. Arxiv is great for peer review. Your work is
| actually fucking available and not behind some pay wall, where
| your peers can... review it. Only thing would be better is if
| we were using OpenReview so we could track discussions, but
| I'll admit that could get messy real quick as anyone that's
| open sourced their research work will tell you (lots of
| questions like "I trained my model, how do I test it?" and "I'm
| getting a cuda out of memory error, how do I fix this?").
|
| Arxiv/preprints are peer review
| fsckboy wrote:
| > _Science got along fine for thousands of years_
|
| doesn't upend your point, but just for the record, what we
| call science has only been around for about 500 years
| xorbax wrote:
| The thing that established journals do better than arXiv is
| clarity
|
| You can upload whatever trash methods you want, but a normal
| journal will have at least one guy who tells you to wipe your
| ass and make your bullshit presentable in public, if only
| because they're expected to be gatekeepers for a minimum
| standard of supposed reproducibility.
| dheera wrote:
| > make your bullshit presentable in public
|
| Triple integrals and sigmas over sets, instead of clearly
| stated limits, and that are not analytically solvable, is
| not "presentable". Scientists do this stuff on purpose to
| make their papers harder to read.
|
| > expected to be gatekeepers for a minimum standard of
| supposed reproducibility
|
| They don't enforce this at all. Otherwise they would demand
| code and training data be released for ML papers.
| godelski wrote:
| > Scientists do this stuff on purpose to make their
| papers harder to read.
|
| Not just that, but if your paper has math that the
| reviewer doesn't understand they're more likely to think
| the work is good and rigorous. It's not like they read it
| anyways.
|
| > and training data be released for ML papers.
|
| Other than checkpoints and hyper-parameters, what do you
| want? The wandb logs? I do try to encourage people to
| save all relevant training parameters in checkpoints (I
| personally do). This even includes seeds.
| dheera wrote:
| > what do you want?
|
| Hyperparameters yes, but also the data used for training.
| I should be able to reproduce the checkpoint bit-for-bit
| by training from scratch. If their training process is
| not deterministic, also release the random seed used.
| godelski wrote:
| Oh yeah, that I agree. I'm kinda upset Google is
| frequently pushing papers with JFT and 30 different
| versions of it and making conclusions based on pre-
| training with it. This isn't really okay for publication.
| Plus it breaks double blind! I'd be okay if say CVPR
| enforced that they train on public datasets and can only
| add proprietary after acceptance (but you've seen my
| views on these venues anyways).
|
| All ML training is non-deterministic. That's kinda the
| point. But yeah, people should include seeds AND random
| states. People forget the latter. I also don't know why
| people just don't throw args (including current iteration
| and important metrics) into their checkpoints. We share
| this frustration.
| Dylan16807 wrote:
| Being pseudorandom is often the point. That's very far
| from deliberately being nondeterministic.
| zamalek wrote:
| > if only because they're expected to be gatekeepers for a
| minimum standard of supposed reproducibility.
|
| There are decades of beta-amyloid papers that show that
| this is false. Like any for-profit entity, premium
| journals' primary purpose is to satiate shareholders - it
| is their feduciary responsibility to get away with as much
| as possible in the name of profit.
|
| That's not to say that open access doesn't have equally as
| concerning issues: which of the two is better or worse is
| an extremely difficult call to make.
| lazide wrote:
| Just because someone wiped their ass doesn't mean they're
| not still full of shit.
|
| It just means it's less stinky/noticable.
|
| I think I did take this analogy too far though.
| godelski wrote:
| In practice I'm not seeing much of a difference. Maybe it
| is just being in ML, where if you wait for conferences
| you're far behind. If a paper is that shitty, it is usually
| very apparent. Like if a paper isn't in latex you know... I
| mean there's a lot of garbage in conferences and journals
| too, I just haven't found it to be a meaningful signal.
|
| And it is still silly that people call it "peer review."
| Peer review is not 3-4 randos briefly glancing my work in
| an adversarial setting who say my work is not novel because
| it is the same as some unrelated work that they didn't read
| either; peer review is the grad student building on top of
| my work, peer review is lucidrains rewriting my work from
| scratch, it is Ross Wightman integrating it into timm and
| retraining, it is the forks that use my work for projects
| (hobby or professional). Peer review is peers looking and
| reviewing. More peer review happens on Twitter than these
| conferences. You can say these conferences and journals are
| a form of peer review, but we gotta stop saying that just
| because something is a preprint that it isn't peer
| reviewed. That's just incorrect. Peer review is when peers
| review.
| DoctorOetker wrote:
| I agree with the core of your message, and you are far
| from alone with this opinion.
|
| The history of science is a sequence of profound
| platitudes, each time refining the prior ones. It is a
| part of awakening to pick the lowest hanging fruits
| first, just to get a foot in the door. While its not
| theoretically impossible to go straight from ignorance to
| -say- the postulates of quantum field theory, its
| incredibly unlikely to scale such a huge step at once,
| both for scientists at the frontier as well as each
| generation of students catching up. In this sense the
| profound platitudes are quasi necessary to build up an
| understanding of the world around us. Individually each
| outdated platitude may appear so very wrong when looking
| back with 20 / 20 hindsight, and most of these individual
| platitudes could plausibly have been skipped by scaling 2
| or 3 steps at a time, but no one could scale all of them
| at once.
|
| Literally respect and review mean the same thing. To view
| or observe anew (re-).
|
| One may objectively claim that each step or correction of
| a prior platitude was an improvement of our worldview,
| even though in hindsight each step of correction also
| contributed the next erroneous stumbling block to be
| scaled.
|
| Throughout history policy decisions (from household
| decisions, to court decisions, to budget decisions, to
| research direction decisions, etc...) humans sought ways
| to _settle matters_.
|
| The desire for Finality is universal and justified.
| However the expectation that this universe came with a
| manual is not.
|
| A core tenet of the scientific mindset is the recognition
| that this finality is a temporary illusion.
|
| Selling this illusion of a recipe for finality is
| profitable and what journals and gatekeepers have
| gravitated toward, especially since information became
| dirt cheap to duplicate.
| godelski wrote:
| I think you missed my point.
|
| I have no problem citing, and actively encourage it. I do
| have a problem with someone rejecting my work because I
| failed to cite a work which would have been impossible
| for me to cite. The key is that it was released (as a
| preprint) AFTER my paper. Not only is it in bad taste to
| make concurrent works a requisite (happy to update btw),
| but it is ludicrous to require that I have a time
| machine.
|
| This isn't about respect and me complaining about pomp
| and circumstance. This is me complaining about the
| frequency that I have been criticized for not having a
| time machine.
|
| The point is that a system that allows people to reject
| works due to lack of access to a functioning time machine
| is not a system we should support.
| xorbax wrote:
| Sure, peer review is peer review. It depends on your
| peers.
|
| I'm a chemist, who has considerable established
| competition. So our reviwers know the systems and studied
| the method in grad school and sweated over it as a
| postdoc and had to innovate for cash in a professional
| capacity while doing something other than what we were
| doing.
|
| They'd excoriate for vague methods, or poor explanations,
| or general nonsense. You had to explain everything and
| make sure to refence the Big boys you were close to but
| definitely not stealing from.
|
| I think ML is unformed enough that there isn't that sort
| of public stricture as a random field in chemistry. Feel
| lucky. We don't even get to hide behind beautiful LaTeX,
| because there's too much benchwork for anybody's boss to
| ever give a fuck about it.
| godelski wrote:
| > You had to explain everything and make sure to refence
| the Big boys you were close to
|
| That sounds problematic
|
| > there isn't that sort of public stricture as a random
| field in chemistry. Feel lucky.
|
| It's pure noise over here. I don't feel lucky, I feel
| frustrated. I get 3 reviewers and 3 completely different
| reasons to reject, often nonsensical (not joking,
| recently was rejected because a broken reference link to
| the appendix). Then submit to another conference and it
| is another 3 reviewers with 3 different reasons to reject
| that are completely different from the previous ones.
| Often asking me why I didn't cite arxiv papers that were
| released after the submission deadline. ACs don't care as
| long as reviewers agree because they gotta keep that
| acceptance rate low. It's playing the fucking lottery
| except your graduation and career depends on it despite
| people frequently saying conferences don't matter they'll
| still critique on lack of top tier publications.
|
| Your system doesn't sound great and also sounds
| frustrating, but I'll trade you. Regardless, I don't have
| faith in either system to significantly determine if a
| work is good (I do believe they can identify bad works,
| just not good works).
| YeGoblynQueenne wrote:
| >> I don't feel lucky, I feel frustrated. I get 3
| reviewers and 3 completely different reasons to reject,
| often nonsensical (not joking, recently was rejected
| because a broken reference link to the appendix).
|
| The people who review your papers in conferences and ask
| you why you didn't cite future arxiv papers are the same
| people who put their work on arxiv and cite each other's
| preprints. You can't rely on the process of "peer-review"
| on arxiv any more than you can rely on the conference
| peer-reviews because they're performed by the same
| people, and they're people who don't know what they're
| doing.
|
| The sad truth is that the vast majority of the
| researchers in the machine learning community haven't got
| a clue what the hell they're doing, nor do they
| understand what anyone else is doing. The typical machine
| learning paper is poorly motivated, vaguely written, and
| makes no claims, nor presents any results, other than
| "our system beats some other systems". As to
| reproducibility, hell if we know whether any of that work
| is really reproducible. Everybody who references it ends
| up doing something completely different anyway and they
| just cite prior work as an excuse to avoid doing their
| job and properly motivating their work. The people who
| write those papers eventually get to be reviewers (by
| sheer luck), or sub-reviewers. They have no idea how to
| write a good paper, so they have no idea how to write a
| good review, either. And they couldn't recognise a good
| paper if it jumped up and bit them in the cojones.
|
| I love to cite Geoff Hinton on this one:
| GH: One big challenge the community faces is that if you
| want to get a paper published in machine learning
| now it's got to have a table in it, with all these
| different data sets across the top, and all these
| different methods along the side, and your method
| has to look like the best one. If it doesn't look
| like that, it's hard to get published. I don't think
| that's encouraging people to think about radically
| new ideas. Now if you send in a paper that
| has a radically new idea, there's no chance in hell
| it will get accepted, because it's going to get some
| junior reviewer who doesn't understand it. Or it's
| going to get a senior reviewer who's trying to
| review too many papers and doesn't understand it first
| time round and assumes it must be nonsense.
| Anything that makes the brain hurt is not going to
| get accepted. And I think that's really bad.
|
| https://www.wired.com/story/googles-ai-guru-computers-
| think-...
|
| So the problem is not arxiv or not arxiv, the problem is
| that peers in peer review lack expertise and knowledge
| and they can't do their job well.
| godelski wrote:
| > The people who review your papers in conferences and
| ask you why you didn't cite future arxiv papers are the
| same people who put their work on arxiv and cite each
| other's preprints.
|
| I don't have a problem citing arxiv works. I mean I have
| no faith in the official system, so why wouldn't I? But
| this is a clear indication of an impossible bar to pass
| and demonstrates the ridiculousness of the system. It
| isn't just that there's a malicious or dumb reviewer, it
| is that the other reviewers don't call them out, the AC
| doesn't call them out, and the metareviewer doesn't call
| them out. The best case was that broken link person and
| the AC made them update their response, but this was in
| the response to my 1 page rebuttal (for 4 reviewers)
| where they got to say whatever they wanted and I had no
| chance of response. (Their initial review was literally 2
| lines) I don't know how anyone can see this and not think
| that the system absolutely failed at every level. You're
| probably unsurprised to hear that this is a common
| occurrence rather than uncommon.
|
| > the vast majority of the researchers in the machine
| learning community haven't got a clue what the hell
| they're doing
|
| I'm well aware. __I__ have no idea what I'm doing, but at
| least I can tell you the difference between probability
| and likelihood or that tuning on the test set is
| information leakage. Research often requires venturing
| into the unknown and unexplored. That's fine. I don't
| care if we're all stumbling around in the dark. I do care
| when people are not just unable to admit it, but unable
| to recognize this. But that's the classic "tell a lie
| enough times and you'll start to believe it" situation.
|
| I am in full support of that Hinton quote. This is the
| first I've heard it (or recall at least), but I often say
| quite similar things (in fact, just did in another
| thread). I do mean it when I say that our current system
| harms us and I'm confident that we won't get to AGI with
| this system.
|
| > the problem is that peers in peer review lack expertise
| and knowledge and they can't do their job well.
|
| I won't disagree with this point, but I believe that this
| is a systematic problem rather than an individual. The
| system encourages this behavior rather than stamps it
| out. So in that sense, I think people are doing their job
| very well. It is just that I don't think their job
| actually aligns with the intent of the job. Classic case
| of irony, that the group of people that highly discusses
| alignment is one of the worst at this. But I guess we
| shouldn't be surprised given that lately we've seen how
| unethical we've seen people who write about ethics are.
|
| I do want to add one thing though. A good paper is hard
| to recognize. A bad paper is easy, but a good paper may
| be indistinguishable from a bad paper. This is the
| "paradox" of research and something people need to take
| to heart. That is if we want to align our job
| descriptions with our actual jobs.
| mechanical_bear wrote:
| As to your very last point, it isn't my "job". It's yet
| another task that I take on for no recognition, nor
| additional pay - as is much of academic life.
| godelski wrote:
| Jobs come with a lot of shitty aspects. Don't get me
| wrong, I generally don't enjoy reviewing either. But I
| put a lot of work into it because regardless of what I
| think, this has a significant effect on real people and
| their entire livelihoods can depend on this task.
| Especially those in their early career. One or two
| publications in a top tier journal can land them that
| internship or job which snowballs.
|
| So I'd ask you do one of two things, either:
|
| - Review a work with the diligence and care that you wish
| someone would give to you
|
| or
|
| - Don't review
|
| I'd also appreciate it if you openly recognized how
| stochastic the system is and that if/when you become in a
| position where you need to evaluate someone, that you
| remember this and take it into consideration. It has a
| lot of value to you too, since if the metric is extremely
| noisy it doesn't provide you value to heavily rely upon
| that metric. Look for others.
| cryptonector wrote:
| > > You had to explain everything and make sure to
| refence the Big boys you were close to
|
| > That sounds problematic
|
| Bingo, it _is_ problematic. It 's part of how "the Big
| boys" get their academic and compensation (investment in
| their firms) rewards. You have to play the publish-or-
| perish, paper rank game to get ahead and stay ahead in
| academia, and this leads to all sorts of problems.
| Authors don't want to question "the Big boys" because
| that leads to their papers not getting published because
| the "the Big boys" and their bootlicker wannabes _are_
| the reviewers and they will exact their tribute. Make it
| to "the Big boys" club and now you're a _gatekeeper_ and
| now you 're also responsible for perpetuating this
| system.
|
| It's why Nature-style peer review needs to become a thing
| of the past.
|
| I'm not saying that "popular" (for a value of "popular"
| that involves peers at large, not the public at large)
| peer review is / will be without problems. But it seems
| to me that it will -at least for a while- be less
| corruptible.
| godelski wrote:
| Yeah, this is a hill I will die on. I love researching,
| but once I grab my PhD I do not plan to push to
| journals/conferences unless it is requested in a job. It
| just holds no meaning and I'm tired of pretending it
| does. Perpetuating the system harms my fellow
| researchers, kills innovation, and just kicks the growing
| can down the road.
|
| The other hill I will die on is that we shouldn't refer
| to journal/conference publishing as "peer review." This
| is one form of peer review, but there are MANY more. And
| as far as I'm concerned, 3 randos that briefly skim my
| paper in an adversarial setting (zero sum) looking to
| reject works barely constitutes peer review. Peer review
| is what happens when your peers read your work, test it,
| build upon it, replicate it, etc. We need to stop this
| language because it helps no one.
| [deleted]
| bschne wrote:
| > I get 3 reviewers and 3 completely different reasons to
| reject, often nonsensical
|
| Could one problem with these kinds of reviews be that the
| reviewers have absolutely zero skin in the game? Like,
| you can write complete nonsense and nobody will blame you
| personally for it, you have no stake in the outcome of
| the submission/paper what so ever, and you're much more
| incentivized to come up with reasons to reject than to
| say "good enough". I realize the latter part can be good
| in theory, as it sets a high bar, but it also often feels
| like this goes awry when not kept in check somehow.
| godelski wrote:
| > Could one problem with these kinds of reviews be that
| the reviewers have absolutely zero skin in the game?
|
| Actually, it is worse. They probably have skin in the
| game, and incentives to reject! Most CS fields publish
| through a conference system. This is a zero or one shot
| system (if you get a rebuttal phase, _if_ you want to
| count that). These conferences have acceptance rates that
| they "need to maintain" to keep their rankings. In other
| words, you are competing against all other papers being
| submitted to the same conference, not just papers with
| similar topics. Even if it is only a little, rejecting a
| paper actually increases the odds that your submission
| makes it through.
|
| But yeah, you're in the right ball park. This is also why
| you will quite frequently see conference review
| guidelines blatantly violated and why you will see area
| chairs and metareviewers just not care. They all have
| incentives to encourage rejection, let alone be
| impartial.
|
| I think the system works when the community is small and
| there is accountability among peers. Accountability
| creates a dampening effect on bad behavior. But at scale,
| you only need a few bad actors to setup a feedback system
| and to not just spoil the entire barrel, but the entire
| shipment.
|
| Edit: I should also add that there's an additional
| negative incentive. You are not judged by how often you
| review, how many reviews you perform, or how good your
| review is (hard to measure). So reviewing ends up taking
| away time from the very limited time you have to do work
| that you are actually evaluated on. This is likely why
| reviews are so rushed. There's a feedback loop too, since
| many will see that others rush when reviewing their work
| so they get tired and end up rushing when reviewing the
| work of others. "If they aren't going to give my work
| their time, why should I have their work my time?"
| thinking grows.
| bschne wrote:
| > (...) make sure to refence the Big boys you were close
| to but definitely not stealing from
|
| Much as giving credit is important, this is unfortunately
| also how you end up with intro sections littered with
| references to the same ten papers everyone's read already
| just to back up some extremely vague/general statement.
| godelski wrote:
| I hate this. Honestly, I'd love to see short papers. Just
| cite what is relevant and not much else. But I see
| citations exploding. Look at this paper[0], it is 34
| pages in total and 5.5 of those are the bibliography! It
| has almost 100 references! ~14 pages are images (not
| figures... images). Nearly an entire page worth of
| material is just the citations! Not the text referencing
| the citations, but the citations themselves. It is
| absolute chaos.
|
| No, this isn't a survey paper. So it is serving no
| purpose other than greasing their peers.
|
| [0] https://arxiv.org/abs/2303.11435
| xmcqdpt2 wrote:
| Eh, not really.
|
| There are exceptions [1] but most journals don't expect the
| reviewers to even attempt to reproduce results, which makes
| sense given how specialized and expensive scientific
| experiments often are. As a reviewer on open code papers I
| would usually try to run the provided code, it didn't
| always work and that wasn't always addressed before
| publication. (I was also usually the only one who even
| tried.)
|
| Usually peer review is more about making sure the work is
| novel and interesting, fits the journal's audience and
| doesn't have any glaring flaws. Not entirely unlike code
| review: if it builds, merge it, and we can address problems
| in a future PR. Those are basically the reviewer
| instructions you get from most journals IME.
|
| [1] OrgSyn famously requires a reproduction from one of its
| editors lab before it accepts any paper,
|
| http://www.orgsyn.org/about.aspx
|
| It has a very high reputation amongst chemists, even if
| it's "impact rating" is low. High impact journals are not
| usually considered the most accurate.
| godelski wrote:
| > As a reviewer on open code papers I would usually try
| to run the provided code
|
| You're only one of two people I've ever heard make this
| claim. Which I'm sure you're aware, but many people
| probably aren't. Fwiw, I'm often called diligent because
| I read the code (looking at main method and anything
| critical or suspicious. Might run if suspicious). Even
| reading supplementary materials will earn you that title
| (which is inane). According to this informal survey, ~45%
| of neurips read the supplementary material <13% of the
| time and less than a third always read it[0] (I'm in that
| third, and presumably xmcqdpt2).
|
| > Usually peer review is more about making sure the work
| is novel and interesting
|
| This is why I find peer review[1] journal/conference
| reviewing highly problematic and why this system is at
| the root of our current existential crisis: the
| reproduction crisis. Reproduction is the cornerstone of
| science. And many MANY good works are not novel in the
| slightest. See the work of Ross Wightman (timm) or Phil
| Wang (lucidrains). These people are doing critical work
| in the area of ML but they aren't really going to get
| "published" for these efforts. Many others do similar
| work, but just not at the same scale and so you'll likely
| not hear of them, but they are still critical to the
| ecosystem.
|
| But with your next point: if it builds, merge it; I'm all
| for. The system should be about checking technical
| soundness and accuracy, NOT about novelty and how
| interesting it is. Of course we shouldn't allow
| plagiarism (claiming works/ideas that aren't your own),
| but we should allow: replications, revisiting (e.g. old
| methods, current frameworks (see ResNet strikes back)),
| surveys, technical studies, and all that. Novelty is a
| sham. Almost all work is incremental and thus we get
| highly subjective criteria for passing the bar.
|
| Which is probably why high impact journals are not
| considered the most accurate. Because they don't
| encourage science so much as they encourage paper
| milling, rushing, and good writing.
|
| [0]
| https://twitter.com/sarahookr/status/1660250223745314819
|
| [1] We need to stop calling journal/conference reviewing
| "peer reviewing." Peer review is when your peers review.
| Full stop. This can come in many forms. Similarly
| publishing is when you publish a paper. Many important
| works come through open publishing.
| hn_throwaway_99 wrote:
| I don't think you are really arguing against what the
| parent poster was saying. That is, I interpreted the
| parent commenter as saying that journals require that
| submissions _at the very least_ be in a clear,
| understandable, "your paper must be at least verifiable
| (or falsifiable)" format, _not_ that they actually
| attempt to reproduce the results.
| davidktr wrote:
| (not OP) Verifiability/falsifiability are big words,
| mostly it is not clear what that means in a specific
| case. Crucially, that is not what
| journals/editors/reviewers do. They check if they find
| the contribution convincing, novel, and in line with the
| discipline's community standards, nothing more.
| willis936 wrote:
| [dead]
| crucialfelix wrote:
| That could be an LLM prompt. Cleanliness and structure
| analysis checks for papers would be very useful tool.
| whimsicalism wrote:
| > Science got along fine for thousands of years without
| Journals and Conferences deciding what's "correct" and not
|
| The 'republic of letters' was much more similar to the
| journal-style approach than to modern arxiv, IMO.
| godelski wrote:
| That was one way, but it definitely was not the only way. A
| lot was people writing letters to one another and sharing
| papers by word of mouth. There will always be elitist
| groups, yeah, but we shouldn't support them. It's about
| doing good work, not gatekeeping. Reviewing is supposed to
| be critical, but that's not the same as adversarial.
| Science is about progressing human knowledge and we need to
| ask if this system is accomplishing that or if we're just
| caught it Goodhart's nightmare.
| SiempreViernes wrote:
| The work of keeping spurious results out is pretty
| important for the building of reliable knowledge
| though...
| godelski wrote:
| I'm not seeing good evidence that journals or conferences
| do any better of a job at this than pre-print servers do.
| They do have a slight edge, but that is not that great.
| The edge is because the default position is to reject and
| so identifying just a small percentage of good papers
| does affect this outcome, but that's not a meaningful
| signal. Have an extremely high false positive reject rate
| and a high false positive accept rate doesn't make the
| signal meaningful, it just means it is very noisy.
|
| I don't give this very low signal any meaning because the
| truth is that good works filter to the top regardless of
| the journals. They rise because peers share them, not
| because they got the stamp mark from a journal.
| heliophobicdude wrote:
| Has there been any serious discussion about using GitHub for
| peer reviews? Specifically the pull request functionality?
| rsfern wrote:
| JOSS does something like this and it's awesome.
|
| I'm not sure about the pull request workflow though, how
| would that work for review? Is your concept more for
| collaboration?
|
| https://joss.theoj.org/
| einpoklum wrote:
| GitHub would not be relevant in this respect because:
|
| * It's owned by a (single) commercial corporation,
| Microsoft.
|
| * There is censorship both by content and in some respects
| by country of origin.
|
| * The code is closed.
|
| but otherwise it's an interesting idea.
|
| The C++ standardization committee uses GitHub to track
| papers submitted to them, see:
|
| https://github.com/cplusplus/papers
| godelski wrote:
| Yeah, but more in the CS world and specifically with ML.
| There are still plenty of CS groups that don't make their
| source available.
| tensor wrote:
| I don't understand how pull requests would help with
| reviews. Peer review is generally looking at the
| methodology and analysis, not simple typos or wording that
| might be corrected by a pull request.
| efficax wrote:
| science in the modern sense is far less than 1000 years old,
| and emerged in tandem with systems of publication which no
| did not use the contemporary review system but did have
| editors who decided whether or not to publish submissions.
| not defending the peer review process as it currently exists
| but it's not a bizarre outlier in the history of science
| danielbln wrote:
| Also, the scientific method took shape in what, the 17th
| century? Thousands of years, that's quite the stretch for
| "science".
| xorbax wrote:
| Eventually somebody will say "what if arXive, but we had
| really smart people who were volunteer experts read this
| bullshit first and decide if it was good enough? It would
| be like arXive but way more efficient for the end user!"
| godelski wrote:
| That's not far from how it actually works. 90% of what I
| read is from arxiv, and before it has been accepted into
| a journal or conference. Most of the time that I'm
| visiting a conference website is to submit a paper or get
| the proper citation for the paper I'm writing.
|
| Like I'm not sure what you expect, just dumb people to
| read arxiv? If that were the case then no one would be
| submitting works to arxiv. The reason works go there is
| because researchers are reading them. They then talk
| about the works to their peers, post on twitter, blog, or
| whatever. And guess what, that's also exactly how works
| that are published in journals and conferences get passed
| around because there's a million of them too and no way
| for people to sort them. No one is sitting around ranking
| papers. That process happens by citations.
|
| Who do you think the end user is? Laymen?
| SiempreViernes wrote:
| Are you aware that the origin of Arxiv was as a pre-print
| archive? People read stuff on there because it was freely
| accessible drafts of _published_ work, and in my field
| that is still the most common practice.
| godelski wrote:
| Yes, but I think you're being too strict with the
| definition of draft. It is not "draft" as in "first draft
| of my paper" it is "draft" as in "we submitted this to
| journal/conference and are just making widely available"
| or "here's the same copy that is behind a paywall, fuck
| jstor."
|
| There's a reason most papers that end up in journals or
| conferences don't get a revision, or if they do only get
| minor ones. In general it is the same exact work which
| got sent to a journal or conference. So that is either
| the final draft or near. I do not want people to confuse
| this with "early" or "first" draft.
|
| The history of arxiv is a few physicists getting annoyed
| that, like many of their peers, that sending per-prints
| (papers awaiting review by a journal) by email was
| burdensome. This was about just making access easier. It
| was about literally making peer review easier. Because
| let's be real, your colleagues reading your work closely
| is better review by peers than a few random people who
| are annoyed about having to review and know it is easy to
| just dismiss and reject.
| SiempreViernes wrote:
| Arxiv _is_ moderated[1], so that 's basically exactly how
| it currently works though the standards for what is let
| trough are lenient.
|
| [1] https://blog.arxiv.org/2019/08/29/our-moderation-
| process/
| EvgeniyZh wrote:
| > Science got along fine for thousands of years without
| Journals and Conferences deciding what's "correct" and not.
|
| And was available only to rich bored people who could self
| teach themselves
| _aavaa_ wrote:
| Right, because a 40$-per-article paywall has done wonders
| for accessibility.
| EvgeniyZh wrote:
| I guess you're just trolling, but in case you're serious,
| yes, being able to group together as scientists
| (university) and pay a single (or small amount) of
| entities to get all the scientific results, peer
| reviewed, has done wonders. Being able as
| university/funding agency to evaluate someone's
| scientific output without actually reading their papers
| (because other experts did it) allowed scaling. And to be
| honest, I do not know a single person how pays $40 for a
| paper, there are so many ways to get it for free
| (including just asking the authors). There is space for
| improvements, but "get rid of journals and conferences
| and hope scientists will somehow figure it out" is not a
| solution
| godelski wrote:
| And pre-print servers have even accelerated science.
| Because frankly this is a MCMC system because the nature
| of the game is everyone stumbling around in the dark.
|
| I am a scientist and I'm all for "get rid of journals and
| conferences BECAUSE we already do figure it out." It
| isn't "somehow" we're already fucking doing it. We aren't
| going to journals and searching them, we go to arxiv. We
| read what our peers share with us. We read what comes out
| of other groups that we know that are doing similar work
| to us. Journals and conferences aren't helping us do
| science, they are helping us advance our careers because
| we live in a publish or perish ecosystem that quantifies
| our work based on the prestige of a conference who is
| prestigious because they reject 80% of works. Nowhere in
| here is a measure for the actual quality of a work. That
| is often a difficult task and can only be done by peers,
| which is already happening. Identifying bad works is easy
| but identifying good works is hard.
|
| I'm sorry, but we already live in the system that you're
| suggesting is silly and it's been this way for centuries.
| Journals are just people capitalizing on our work and
| labor. Conferences at least put us in the same room, but
| they have no business claiming that they can accurately
| ascertain the quality of a work.
| evouga wrote:
| Arxiv has its own problems.
|
| You can find dozens of proofs on the arxiv that P=NP and that
| P!=NP. What does peer "review" look like here? Most people
| won't bother to write an article specifically rebutting one
| random incorrect proof.
|
| Now somebody tomorrow posts a proof that P!=NP. Will people
| pay attention? If it's a "big name" or somebody at a
| university with a good PR office then yes; otherwise no.
| Doesn't seem ideal to me.
| godelski wrote:
| How many of those works actually hit the front page of HN?
| How many do you hear of? The reason it is little to none is
| because peer review happens. If someone posted that to HN
| it would likely be trashed very quickly and fail to make it
| to the front page. That's peer review in action. Now does
| shit get through sometimes? Yeah. But I'm not convinced
| this is at a meaningfully higher rate than any top tier
| journal or conference, if at all.
| drtgh wrote:
| The popular Journals have an strange business,
|
| * By one side the Journal reclaim a high amount of money for
| being able TO TRY to publish on them (APCs fee), more if one
| want to be published for to allow free access, as to read the
| publication in such Journals require subscription, are pay
| walled.
|
| * By other side, the reviewers of such papers, academics and
| researchers from other universities, don't receive economical
| compensation for doing the review, while the taken APCs fee
| its supposed for doing it? only in few cases the review is
| compensated by some programs of the reviewer's university or
| other 3rd parties. Then, what is for all those pay-walls and
| publish-fees?
|
| Sounds like a big filter... for what? for who? Because can be
| seen it's not working just as a mere "quality" filter.
|
| Being a paper published in one of those popular Journals, or
| being done directly in medias like arXiv, both sources needs
| the same amount of "grain of salt" about what is being read,
| until others replicate experiments, or contrast theories.
|
| So IMHO, outsider point of view, from time to time I have to
| use Sci-Hub in fact, I understand if they decide to don't
| publish in such popular Journals. I consider absolutely
| legitimate when they do it in distribution channels like
| arXiv.
|
| What I really would like it were exists an open research
| platform without pay walled papers, like happens in arXiv,
| and in addition, peer reviews through the platform were
| possible, as to read such reviews, and also, equally
| important, to have available debates about the paper, like if
| it were hacker news format.
| blablabla123 wrote:
| > Sounds like a big filter... for what? for who? Because
| can be seen it's not working just as a mere "quality"
| filter.
|
| To me it seems it keeps not only a wall around papers and
| journals. But also about whole academia which is probably
| an unwanted side effect since inside and outside of
| academia there seems to be consensus that knowledge
| transfer could be better. Also it seems everybody who can
| rather does research outside (AI/ML, Chemistry, Pharmacy).
| still_grokking wrote:
| > [...] and in addition, peer reviews through the platform
| were possible, as to read such reviews, and also, equally
| important, to have available debates about the paper, [...]
|
| Someone here mentioned:
|
| https://openreview.net
|
| Looks exactly like what is asked for.
|
| Never heard about it though. Would need some more
| visibility (and word of mouth, I guess).
| godelski wrote:
| I mostly agree with you. But I absolutely despise debates.
| They are more often than not worthless. They are based on
| charisma and how good you are at language, not facts.
| Discussion is the right word. I've often thought of pushing
| directly to OpenReview rather than just arxiv, but
| discussions can get off base quickly and novices can
| dominate (while academic Twitter is great, this happens
| there a lot). It is not our job as a researcher to
| communicate our ideas to laymen, that's the job of science
| communicators. Sure, it is great and a bonus, but please
| stop asking on my github how to do something that is in the
| pytorch intro tutorial. Read the tutorial or attend my
| class. Open discussions are nice in theory, but HN and
| Reddit are good examples at how these systems often get
| over burdened with noise. There's a catch-22 and so it is a
| tough call.
| daniel-cussen wrote:
| [dead]
| systemvoltage wrote:
| Somewhere I read that this is how things used to be in Einstein
| era (or maybe earlier than that). There was little bureaucracy
| in getting the idea out.
| ggm wrote:
| Richard Rhodes' monumental work on the A and H bomb mentions
| that from Thompson onward, "being heard" Was a problem.
| Juniors often struggled to be taken seriously, sometimes
| seniors ostentatiously "spoke for them" to have new ideas
| given credence and women were doubly disabled on the "taken
| seriously" front.
|
| Right up until Murray Gell-Mann and beyond, speaking outside
| the current limitations to knowledge was hard. I don't want
| this to descent into AGW and antivaxx denialism, this is
| inside classic science but considering radical
| theory/paradigm shifts which were still testable
| propositions. New models are hard on early stage career.
|
| Blinded peer review was partly designed to help some of that.
| In a narrow enough field it's impossible for reviewer and
| submittor not to know each other. There may only be 3-4
| people who understand your niche fully. Rueben Hersh
| discusses that a bit in "what is mathematics really" (I
| think, could be another book of his, "the mathematical
| experience")
|
| Rhodes discusses Michael Polyani's theory of science as old
| fashioned apprenticeship. Journeyman scientists publish
| reproducible, testable work. Theoreticians.. harder to test
| sometimes.
| nextos wrote:
| I agree with what you say but the problem is that journals
| such as _Nature_ do not have blinded peer review.
|
| Reviewers know who you are. This is quite shocking to
| discover if you come from Math or CS.
| ggm wrote:
| Yes. The system broke down. I don't know anyone who
| thinks it works how it should
|
| I've had pretty hard bounces which were deserved and I
| know how to get work over the threshold, but some review
| feedback has been petty, passive aggressive ignorance,
| and suspiciously similar stuff pops up from time to time
| in "3 papers and your phd is done" which makes me wonder
| if copycats are getting softballed through for career
| development.
|
| the rules behind length, word count and mark-up are
| pretty silly too. I've seen some rather odd latex tricks
| to compensate for length inside word count because
| typographical checkers were bouncing.
|
| I don't do this for a living, barely act as maybe helpful
| co author these days: it's hard work.
| jacquesm wrote:
| I think in part this is because a lot of people started
| to be interested in the meta-metrics of scientific work
| and like any metric once you start tracking it you
| influence the system you are tracking. Publishers and
| various scientific actors then made things worse by
| making those very metrics (a symptom) a goal in its own
| right. That's what broke the system.
| SiempreViernes wrote:
| Well, the blinding is supposed to allow the juniors to
| speak up when the seniors are making mistakes, so it
| makes sense its only one way.
|
| And while double blinding sounds nice in theory I'm not
| so convinced it is that useful in practice, because it
| requires the reviewer to play along and pretend they
| can't figure it out from the text alone: if they are able
| to do that reliably they can probably be trusted to keep
| an open mind anyway.
|
| Reliable blinding of the author would mean having them
| consciously copy the style of others and avoiding citing
| their own previous work, which would be very hard in a
| small sub-field since they are by definition a sizeable
| fraction of it!
| lloydatkinson wrote:
| Why should experiments wait to be submitted to some arbitrary
| academic body before trying to to reproduce them? This is
| what's wrong with modern science, everything is academic.
| kykeonaut wrote:
| The only thing valuable that academic journals and
| conferences bring to the table is the peer-review process.
| Which, to be fair, is needed.
|
| In this case, the peer review is being conducted without the
| need for publishing venues. However, I don't think that is
| usually the case. Take a look at the countless number of
| ArXiv papers that lack any sort of peer review.
___________________________________________________________________
(page generated 2023-07-29 23:02 UTC)