[HN Gopher] GPTZero Case Study - Exploring False Positives
___________________________________________________________________
GPTZero Case Study - Exploring False Positives
Author : sungage
Score : 141 points
Date : 2023-02-19 15:22 UTC (7 hours ago)
(HTM) web link (gonzoknows.com)
(TXT) w3m dump (gonzoknows.com)
| panarky wrote:
| As millions of people interact with ChatGPT, their writing will
| subtly, gradually, begin to mimic its style. As future versions
| of the model are trained on this new text, both human and AI
| styles will converge until any difference between the two are
| infinitesimal.
| onos wrote:
| Interesting idea, but isn't there variance in the output? Eg
| I've seen people ask it to "write in the style of x" etc and
| different people also clearly have different writing styles.
| Filligree wrote:
| You can try, but it isn't very good at that. The style
| remains very ChatGPT.
| zdragnar wrote:
| So long as ChatGPT is forbidden from communicating in certain
| ways (swearing, speaking ill or positive of controversial
| people or topics, etc), convergence will never happen. People
| interact with other people more than they do ChatGPT, so the
| majority force will remain dominant.
| fullshark wrote:
| Sounds accurate and horrifying, I don't get the enthusiasm for
| this at all beyond a desire to be there first and make a ton of
| money. All manuscripts get a run through an AI editor, all
| business writing is even more soullessly devoid of purpose
| beyond accomplishing task X, all blogposts are finetuned for
| maximum engagement and therefore ad/referral revenue.
|
| That's already happening I know but it will be amplified to the
| point that all humanity in writing in lost. All ideas in
| writing will be a copy of a copy of a copy and merely resemble
| something once meaningful. Time to go touch grass.
| visarga wrote:
| Too much AI for you? You can fix your problem with even more
| AI! Get your own AI, running on your hardware, loyal only to
| you. It will act like a firewall between you, the vulnerable,
| hackable human, and the wild internet full of other AIs. To
| go out on the internet without an AI is like going for a walk
| during COVID without a mask, or browsing without AdBlock.
| Their AI will talk to your AI, and that's how you can be
| safe.
| moralestapia wrote:
| +1
|
| We train AIs but they also train us.
| ominous wrote:
| Some related idea, in case you like to see that thought
| explore: https://medium.com/@freddavis/we-shape-our-tools-
| and-thereaf...
| brookst wrote:
| One of the big complaints with LLMs is the confident
| hallucination of incorrect facts, like software APIs that don't
| exist.
|
| But the way I see it, if ChatGPT thinks the Python list object
| _should_ have a .is_sorted() property, that's a pretty good
| indication that maybe it should.
|
| I work in PM (giant company, not Python), and one of these days
| my self-control will fail me and I will open a bug for "product
| does not support full API as specified by ChatGPT".
| visarga wrote:
| > if ChatGPT thinks the Python list object should have a
| .is_sorted() property, that's a pretty good indication that
| maybe it should.
|
| Hahaha, Python language fixing itself!!!
| [deleted]
| dr_dshiv wrote:
| > it, if ChatGPT thinks the Python list object should have a
| .is_sorted() property, that's a pretty good indication that
| maybe it should.
|
| Yes! And when it hallucinates references for articles, often
| times those articles probably _should_ exist...
| visarga wrote:
| And if they don't exist, you ask the model to write them
| from title and link.
| moffkalast wrote:
| The year is 2145.
|
| When a new person is born their entire life is
| hallucinated in its entirety by the all great and
| powerful GPT. Deviation from His plan is met with swift
| and severe consequences.
| howlin wrote:
| > LLMs is the confident hallucination of incorrect facts
|
| This is a very common feature of delirium in people. Chatting
| with an LLM seems a lot like what it would be to talk to a
| clever person with encyclopedic knowledge, who is just waking
| up from anaesthesia or is sleep talking.
| moffkalast wrote:
| Or just the average person on reddit's
| r/confidentlyincorrect.
| xiphias2 wrote:
| Or rather you can just catch method missing in the runtime
| and patch it with a chatgpt call
| brookst wrote:
| Love it. Get ChatGPT to write the missing method, execute
| it this once, then store it in a file, update the current
| source file with the include to cache it for next time.
| matthewdgreen wrote:
| ChatGPT also thinks certain coding algorithms should violate
| well-known information-theoretic bounds ;) I'll put a ticket
| in with Claude Shannon.
| brookst wrote:
| I'm sure he'll be excited to learn about this gap!
| wankle wrote:
| AI will turn half the people into Eloi
| (https://en.wikipedia.org/wiki/Eloi) while the rest of us will
| become the Morlocks.
| joe_the_user wrote:
| An opposite possibility is that the commonness of ChatGPT will
| cause people to adopt a style as distinct from it as possible.
|
| Of course, this might mean future Chatbots would successfully
| emulate that. But it's not impossible an "adversarial style"
| exists - this wouldn't be impossible to emulate but it might be
| more likely to cause the emulator to say things the reader can
| immediately tell are false.
|
| One idea is to "flirt" with all things that people have come up
| with that AI chokes on. "Back when the golden gate bridge was
| carried across Egypt..."
| andyjohnson0 wrote:
| Prediction #1: Once enough ChatGPT output gets posted online,
| it will inevitably find its way into the training corpus. When
| that happens, ChatGPT becomes stateful and develops episodic
| memory.
|
| Prediction #2: As more people discuss ChatGPT online, by late
| 2023 discussion of Roko's Basilisk exceeds discussion of
| ChatGPT. (half /s)
| ClumsyPilot wrote:
| Or. ChatGPT will overtrain on it's own data and go to shit
| the way google search did
| visarga wrote:
| Training on its own data is a tradition already. For
| example RLHF example pairs rated by humans are generated by
| the model. So even our best models trained on their own
| outputs + rating from human labellers. The internet is a
| huge rating machine, AI will distill this signal and
| improve even while ingesting its own text.
| andyjohnson0 wrote:
| Meta-ChatGPT's loss function optimises for ChatGPT
| generating training data that maximises the shittyness of
| Google's LLM.
| janalsncm wrote:
| GPTZero cannot work because they don't have all of the logits
| used to judge perplexity. They only have the top five or so. Even
| OpenAI which has the full set of logits is not able to correctly
| classify all texts. It's a futile effort.
| wankle wrote:
| "Tell me more about your perspective customers."
|
| https://projects.csail.mit.edu/films/aifilms/AIFilms.html
| dsign wrote:
| I had some fun yesterday when ChatGPT hallucinated a
| bibliographic reference to an article that didn't exist. But the
| journal existed, and it had plenty of articles that made
| ChatGPT's hallucination plausible. I think that at least this use
| case can be fixed with some pragmatic engineering[^1].
|
| [^1]: Which may take a bit to happen, because our current crop of
| AI researchers have all taken "The bitter lesson"[^2] to heart.
|
| [^2]: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
| FPGAhacker wrote:
| Did chatgpt post the reference as a footnote (or
| parenthetical)?
|
| At least for now,I was thinking it didn't do that, and maybe
| the lack of references would be an indicator of unedited gpt
| output.
| dsign wrote:
| It did:
|
| ...
|
| > However, according to a study published in the Journal of
| Dairy Science, the diacetyl content of butter can range from
| approximately 0.5 to 14 parts per million (ppm)
| (https://doi.org/10.3168/jds.S0022-0302(03)73775-3).
|
| The doi I could not find, so I'm pretty sure is bogus.
|
| I asked it to produce a full reference:
|
| > Sure, the full reference for the study I mentioned is:
|
| > Yvon, M., Chambellon, E., & Bolotin, A. (2003). Effect of
| pH on diacetyl and acetoin production by Lactococcus lactis
| subsp. lactis biovar diacetylactis. Journal of dairy science,
| 86(12), 4068-4076.
|
| I went to the index of the journal 86(12), and that article
| is not there.
| wand3r wrote:
| I saw this[1] interview with Sam Altman touching on interim AI
| impact. I really agree with his point that basically detecting
| output from LLMs is basically going to be futile and only really
| relevant in the near term. Accuracy is obviously going to improve
| in models and detection isnt that difficult now but will be in
| the future, especially if output is modified or an attempt to
| obfuscate origin is made.
|
| [1]https://youtu.be/ebjkD1Om4uw
| martinflack wrote:
| It's probably a short-term social phenomenon. We don't bother
| detecting mathematical output from calculators or spreadsheets;
| we just like that folks give us the right answer, even if they
| had easy tooling to produce it. However, watching someone do
| things the old way would seem bemusing. If you watched a
| manager notating all over a physical spreadsheet with a pencil
| (as was commonly done at one time) it would seem quaint or
| backwards depending on context. Likewise, waiting for someone
| to write a letter and taking more than 90 seconds because they
| didn't co-author it with AI might seem slow.
| mtlmtlmtlmtl wrote:
| "Write an email that says I did X and they should do Y, but
| if Z then W, and we should schedule a meeting with P and Q."
|
| I feel like for most emails I write, information density is
| close to a maximum. This means there's no actual gain to be
| had from a language model. The email I would write myself is
| going to be about the same length as the prompt I'd have to
| write anyway.
| robocat wrote:
| There is a huge gain if English is your second language,
| and you use ChatGPT to rewrite, or translate.
|
| Even if English is your mother tongue, if your written
| English is crappy or you need to write in a style you are
| unfamiliar with (e.g. formal), then ChatGPT can help.
| mtlmtlmtlmtl wrote:
| If you're writing in a style you're unfamiliar with, how
| do you know the model is doing it correctly?
|
| I also think writing yourself might be far better
| practice. This tool can easily become a crutch. This is
| unlikely to be free anytime soon. In fact it's likely to
| be quite expensive.
| simonh wrote:
| ChatGPT is free now, although there is a paid tier, and
| MS and Google are building similar capabilities right
| into their search interfaces.
| robocat wrote:
| I think we all tend to be better at picking out a correct
| answer than generating a correct answer from scratch.
|
| I can ask ChatGPT to rewrite an email in the style an
| American news reporter from 1950, and I can judge whether
| some of the cliches it generates feel correct. I cannot
| write in that style at all.
| scotty79 wrote:
| I hope gpt detectors will evolve into general bs detectors.
| sebzim4500 wrote:
| >detection isnt that difficult now
|
| I would have thought this, but every attempt I've seen at
| detecting chatGPT generated text has failed miserably.
| Closi wrote:
| It fails on false positives, but you don't often get false
| negatives, which might be enough at the moment for quite a
| few use-cases.
|
| Also false positives are typically "this text is likely to
| contain parts that were AI generated" rather than "This text
| is higly likely to be AI generated" (which is what GPT-
| generated content generally produces).
|
| When I've tried to prompt-engineer GPT to produce text that
| GPTZero will flag as negative it has been pretty tough!
| theptip wrote:
| Of course false positives matter; the naive heuristic
| "everything is AI generated" has zero false negatives, and
| mostly false positives. In the OP half of the positives are
| false. That's not a useful signal IMO. You couldn't use
| that to police homework for example.
| Closi wrote:
| I said they might not matter for _quite a few use cases_
| , not that they don't matter for all use-cases.
|
| e.g. If you were Sam Altman at OpenAI and your use-case
| is mostly looking for training data and wanting to tell
| if it is AI-Generated or not (so you can exclude this
| from training data), you probably care much more about
| false negatives than false positives (false positives
| just reduce your training data set size slightly, while
| false negatives pollute it).
|
| Of course they matter if you are marking homework (where
| conversely false negatives aren't actually that
| important!), but it's pretty trivial to think of use-
| cases where the opposite is true.
| RC_ITR wrote:
| > Accuracy is obviously going to improve in models
|
| Well, to be clear, they can put rules based filters and other
| things on top of the neural net, but the core GPT will _never_
| get more accurate since it has no mechanism to understand what
| words mean.
| valine wrote:
| GPT3 is far more accurate than GPT2. Seems reasonable that
| larger models trained on more data will continue to improve
| accuracy. I'd also expect larger models to be better at
| summarizing text, ie potentially fixing the Bing issues where
| it hallucinates numbers.
|
| Our models sizes are a product of our scaling and hardware
| limitations. There's no reason to believe we are anywhere
| near optimal.
| flir wrote:
| > Seems reasonable that larger models trained on more data
| will continue to improve accuracy.
|
| It also seems reasonable to assume that they will
| eventually encounter diminishing returns, and that the
| current issues, such as hallucinations, are inherent to the
| approach and may never be resolved.
|
| To be clear I don't have a clue which statement is true
| (though I don't see why scaling would solve the
| hallucination problem).
| int_19h wrote:
| The biggest problem is that scaling is non-linear. The
| returns might well be non-diminishing wrt model size, but
| if we have to throw N^2 hardware at it to make it (best-
| case) 2N better, we'll still hit the limit pretty
| quickly.
| mtsr wrote:
| Might turn out that for rules based systems such as
| prescriptive grammars (for grammatically correct language
| rather than natural spoken) there is still use for a
| system that explicitly represents those rules.
|
| Then again, we are a big old bulb of wetware and we can
| generally learn to apply grammar rules correctly most of
| the time (when explicitly thinking about them, anyway).
|
| Maybe what we need is some kind of meta cognition: being
| able to apply and evaluate rules that the current LLMs
| can already correctly reproduce.
| ozy23378 wrote:
| [dead]
| wongarsu wrote:
| This is testing on abstracts of medical papers. I wouldn't be
| surprised if the way abstracts are carefully written and reviewed
| by > 3 people is very different from normal text, and in some
| ways more similar to LLM output.
| jmfldn wrote:
| It's important to shine a light on the limitations of AI
| detection software, and this case study on GPTZero does just
| that. False positives can have serious consequences, particularly
| in sensitive areas such as healthcare.
| MonkeyMalarky wrote:
| Thank you for your heroic effort in copying and pasting a chat
| log, I am left in awe by it.
|
| ...nice edit...
| jmfldn wrote:
| Lighten up mate.
| photochemsyn wrote:
| Just wrote this myself, although I did try to chatGPT-style it a
| bit. I thought the final third would serve to identify it as non-
| AI as it goes off on a tangent about isotopes...
|
| > "The periodic table is a systematic ordering of elements by
| certain charcteristics including: the number of protons they
| contain, the number of electrons they usually have in their outer
| shells, and the nature of their partially-filled outermost
| orbitals."
|
| > "Historically, there have been several different organizational
| approaches to classifying and grouping the elements, but the
| modern version originates with Dmitri Mendeleeve, a Russian
| chemist working in the mid-19th century."
|
| > "However, the periodic table is also somewhat incomplete as it
| does not immediately reveal the distribution of isotopic variants
| of the individual elements, although that may be more of an issue
| for physicists than it is for chemists."
|
| GPTZero says: "Your text is likely to be written entirely by AI"
|
| Now I'm feeling existential dread... perhaps I am an AI running
| in a simulation and I just don't know it?
| catchnear4321 wrote:
| How has your "success" been with chatgpt? Qualitatively,
| generally, positive/negative.
|
| Being able to speak to machines would likely correlate with
| "sounding like one."
|
| There could be a different (mis)categorization here but also
| por que no los dos.
| robocat wrote:
| Perhaps you just write very predictably.
|
| A partisan writing cliched slogans and regurgitating tired
| political statements has a "temperature" closer to 0, likewise
| an engineer stringing together typical word combinations. A
| poetical writer with surprising twists and counterintuitive
| mixtures of words has a temperature closer to 1.0.
|
| I can see a few cliches in your writing, also I did a search on
| some fragments of sentences which showed a number of results.
|
| If you want to be less "robotic" then you could add whimsical
| or poetic wording, and less common turns of words (be a phrase
| rotator).
| LoveMortuus wrote:
| I mean, technically speaking you are a man made intelligence,
| thus it would be fair to say that you are artificial
| intelligence. ^^
| thomastjeffery wrote:
| Technical writing, in order to be relatively unambiguous - the
| "technical" part - defines itself as a subset of English with a
| constrained grammar and vocabulary.
|
| You just illustrated technical writing. Naturally, your writing
| style is very similar to that of other technical writing.
|
| Take one guess what kind of writing exists in most of the text
| GPT is trained on.
| sdwr wrote:
| You made a few unforced errors that move it away from the
| quietly authoritative, mirror-sheen AI voice.
|
| The colon in line 1 is clunky, the combination of "but" and
| "with" in line 2 reads as passive, and line 3 is full person.
| corrupto wrote:
| main.2019115999.com.dts.freefireth.obb 1 El archivo comprimido
| esta corrupto Freef
| Maken wrote:
| Alternative title: Most academic papers are indistinguishable
| from AI generated babble.
| [deleted]
| sigmoid10 wrote:
| ...to AI. It's kinda funny how this is yet another area where
| these models suck very much in the same way that most humans
| do. LLMs are bad at arithmetic? So are most people. Can't tell
| science from babble? I already wouldn't ask a non-expert to
| rate any aspect of an academic paper. Trusting the average Joe
| who has only completed some basic form of education would be
| tremendously stupid. Same with these models. Maybe we can get
| more out of it in specific areas with fine tuning, but we're
| very far away from a universal expert system.
| brookst wrote:
| The best was the Ted Chiang article making numerous category
| errors and forest/trees mistakes in arguing that LLMs just
| store lossy copies of their training data. It was well-
| written, plausible, and so very incorrect.
| dr_dshiv wrote:
| I felt the same way. But I'd love to read a specific
| critique. Have you seen one?
| elefanten wrote:
| Here's one from a researcher (which also links to
| another), though I'm not qualified to assess it's content
| in depth.
|
| https://twitter.com/raphaelmilliere/status/16240731504754
| 319...
| supriyo-biswas wrote:
| Neural network based compression algorithms[1] are a thing,
| so I believe Ted Chiang's assessment is right. Memorization
| (albeit lossy) is also how the human brain works and
| develops reasoning[2].
|
| [1] https://bellard.org/nncp/
|
| [2] https://www.pearlleff.com/in-praise-of-memorization
| jejeyyy77 wrote:
| I mean, humans can't distinguish AI written text either -
| which is why this tool was built?
|
| I don't see how it will be possible to build such a tool
| either as the combination of words that can come after
| another is finite.
| jldugger wrote:
| >...to AI.
|
| Perhaps we can call it the "Synthromorphic principle," the
| bias of AI agents to project AI traits onto conversants that
| are not in fact AI.
| Maken wrote:
| I do agree that the most likely reason is that scientific
| papers tend to be highly formulaic and follow strict
| structures, so a LLM is be able to generate something much
| more alike to human writing than if it tries to generate
| narrative.
|
| But it's still fun to deduce that the reason is that the
| quality of technical writing has sunk so low, that is even
| below the standards for AI generated text.
| bentcorner wrote:
| Random thought: In the future the simplest method to reduce your
| chance of GPT usage being detected is to pretend your paper was
| written in 2022 or earlier.
| nobu-mori wrote:
| That's great! All we need to do is negate the output and it will
| be more accurate.
| netfortius wrote:
| FoxNews is accurate 10% of the time, at best, and it's allegedly
| produced by humans, so I'm not really seeing the problem here...
| ClumsyPilot wrote:
| you do it all wrong, it's 90% accurate and 90% sucessfull - its
| goal to be as inaccurate as possible without its audience
| realising.
|
| If fox news deletws the last 10% of reality from it's
| broadcasting, the people might start catching on. Although
| these days i am not sure
| mt_ wrote:
| Isn't the GPTZero based on model detector by OpenAI for GPT-2? In
| the initial preview it was exaclty like the demo, a text box
| styled diferently.
|
| [1] - https://github.com/openai/gpt-2-output-
| dataset/blob/master/d...
| explaininjs wrote:
| Non-editorialized title: GPTZero Case Study (Exploring False
| Positives)
|
| @dang
| [deleted]
| gtsnexp wrote:
| Detecting text generated by large language models like ChatGPT is
| a challenging task. One of the main difficulties is that the
| generated text can be highly variable and can cover a wide range
| of topics and styles. These models have learned to mimic human
| writing patterns and can produce text that is grammatically
| correct, semantically coherent, and even persuasive, making it
| difficult for humans to distinguish between the text generated by
| machines and the ones written by humans.
|
| Another challenge is that large language models are highly
| complex and constantly evolving. GPT-3, for example, was trained
| on a massive dataset of text and can generate text in over 40
| languages. With this level of complexity, it can be challenging
| to develop detection systems that can keep up with the ever-
| changing text generated by these models.
|
| To implement a reliable detection system like GPTZero, which is
| designed to detect text generated by GPT-3, several challenges
| need to be addressed. First, the system needs to be highly
| accurate and efficient in identifying text generated by GPT-3.
| This requires a deep understanding of the underlying language
| model and the ability to analyze the text at a granular level.
|
| Second, the system needs to be scalable to handle the vast
| amounts of data generated by GPT-3. The detection system should
| be able to analyze a large volume of text in real-time to
| identify any instances of generated text.
|
| Finally, the system needs to be adaptable to the evolving nature
| of large language models. As these models continue to improve and
| evolve, the detection system needs to keep up and adapt to the
| changing landscape.
| ericmcer wrote:
| It was weird how easy this was to identify if you have read any
| amount of ChatGPT content. It has a particular writing style
| that is pretty obvious.
|
| I am not sure how you would code something to detect an author
| based on writing style. It feels like something people would
| have tried to do before. Probably using a similar approach that
| LLMs use but with a separate predictor for specific authors.
| mds wrote:
| ChatGPT in particular writes in middle school essay format:
| introduction, point 1, point 2, point n, conclusion.
| [deleted]
| sdflhasjd wrote:
| Ironic, because to me this was so obviously ChatGPT to me from
| literally the first sentence.
|
| I think this is probably because it doesn't match the
| conversational style of a forum discussion.
| k__ wrote:
| Yes, it's really bad.
|
| False positives and negatives all over the place.
|
| I really wish it worked, but generally, it doesn't.
| icapybara wrote:
| Right, I mean, how could it know?
|
| I could write in the tone of ChatGPT if I tried hard enough. It's
| an intractable problem and a tool like this probably does more
| harm than good.
| JoeAltmaier wrote:
| If it's like many of these 'AI' engines, it's a statistical map
| of text. I'd expect it to produce no better than what it's
| trained on, then muddy that by randomly combining different
| versions of similar statements.
|
| I'm impressed it is ever accurate.
| brookst wrote:
| I've got a fun little side project that uses GPT. I tested
| gptzero against 10 of my projects' writings and 10 of my own. It
| detected 6 out of 10 correctly in both cases (4 gpt-written bits
| were declared human, 4 human-written were declared gpt).
|
| Which is better than 50% but not nearly good enough to base any
| kind of decision on.
| lwhi wrote:
| Perhaps AI generated text should be created with a specific
| signature in mind _specifically_ to be identifiable?
| mattnewton wrote:
| isn't this essentially asking anyone who runs a model to flip
| the evil bit[0]? People who want to misrepresent model output
| as human written output will trivially be able to beat this
| protection by removing the signature or using a version of the
| model that simply doesn't add it.
|
| [0] https://en.m.wikipedia.org/wiki/Evil_bit
| wongarsu wrote:
| There's a large body of research into invisible text
| watermarking, so this would certainly be possible. Maybe the
| simplest to implement in LLMs would be to bias the token
| generation slightly, for example by making tokens that include
| the letter i slightly more likely. In a long enough text you
| could then see the deviation from normal human text
| characteristics.
___________________________________________________________________
(page generated 2023-02-19 23:00 UTC)