[HN Gopher] DetectGPT: Zero-Shot Machine-Generated Text Detection
       ___________________________________________________________________
        
       DetectGPT: Zero-Shot Machine-Generated Text Detection
        
       Author : O__________O
       Score  : 66 points
       Date   : 2023-01-28 12:55 UTC (10 hours ago)
        
 (HTM) web link (ericmitchell.ai)
 (TXT) w3m dump (ericmitchell.ai)
        
       | mdorazio wrote:
       | I might not be understanding what AUROC really means, but how is
       | this performing in terms of false positives/negatives in
       | comparison to human readers?
        
         | O__________O wrote:
         | Here's a post on understanding AUC (Area Under The Curve) and
         | ROC (Receiver Operating Characteristics):
         | 
         | https://towardsdatascience.com/understanding-auc-roc-curve-6...
        
       | O__________O wrote:
       | Research paper:
       | 
       | https://arxiv.org/abs/2301.11305
       | 
       | Additional explanation:
       | 
       | https://twitter.com/_eric_mitchell_/status/16188203614199152...
        
         | m00viin_pics wrote:
         | Awesome thread by one of the authors of paper, concise and
         | insightful, thanks for sharing !
         | 
         | Research was aiming to bring a tool/approach on distinguishing
         | text from LLM and other sources but in the end of the day it
         | will only benefit those with non-open-source LLMs adjust to
         | such technique and "fool" better everyone else (cause we need
         | log-probs out of a model on each of the sample text).
         | 
         | It seems kinda ironic for me, maybe i missed some crucial point
         | here.
        
           | O__________O wrote:
           | Highly unlikely any popular model would not provide log-
           | probs, since they're potentially useful. For example:
           | 
           | https://aidungeon.medium.com/controlling-gpt-3-with-logit-
           | bi...
        
       | eclipsetheworld wrote:
       | This approach seems to require knowledge of which LLM was used to
       | generate the given text. I wonder if e.g. model fine-tuning - as
       | already provided by Open AI [0] - could evade this detection
       | approach.
       | 
       | [0] https://beta.openai.com/docs/guides/fine-tuning
        
       | gault8121 wrote:
       | How does this compare to GPT2 output detector?
       | 
       | I created AIwritingcheck.org to provide teachers with a user
       | friendly interface for this model.
        
         | andai wrote:
         | Wouldn't GPT-2 be detectable by virtue of being bad? Or is it
         | bad in a way that is more or less indistinguishable from
         | student writing?
         | 
         | It's been a while since I used it but I very rarely got
         | plausible output from it.
        
         | eh9 wrote:
         | Stop doing this. Have teachers figure out how to move forward
         | with this and students, but the false-positives are not worth
         | it
        
         | SachinDSI wrote:
         | Consider trying out GPTKit https://gptkit.ai it has higher
         | accuracy than GPTZero and uses 6 different methods
        
       ___________________________________________________________________
       (page generated 2023-01-28 23:01 UTC)