[HN Gopher] Non-Evolutionary Superintelligences Do Nothing, Even...
       ___________________________________________________________________
        
       Non-Evolutionary Superintelligences Do Nothing, Eventually
        
       Author : telmo
       Score  : 17 points
       Date   : 2023-04-12 16:41 UTC (6 hours ago)
        
 (HTM) web link (arxiv.org)
 (TXT) w3m dump (arxiv.org)
        
       | habitue wrote:
       | > By definition, a human designer cannot prevent a superhuman
       | intelligence from modifying itself
       | 
       | And we can stop reading there. That's such muddled thinking it
       | doesn't merit reading the rest of the paper.
       | 
       | ("By definition"? What? This is something that requires evidence.
       | Anything that's "by definition" doesn't mean you can rest your
       | case on it, it means now your definitions are suspect so those
       | better be well justified)
        
         | rep_lodsb wrote:
         | When you read "superintelligence", think "God". It's theology
         | pretending to be science.
        
       | bediger4000 wrote:
       | Wow, another thing Stanislaw Lem beat the scientists to.
       | 
       | In "Imaginary Magnitude", the "Golem XIV" chapter, Lem has his
       | superintelligences disappear up their own metaphorical navels,
       | for their own incomprehensible-to-humans reasons. Of course, you
       | can read between the lines about why the Golems disappeared, but
       | "Do Nothing, Eventually" sounds like a good description.
        
         | BillSaysThis wrote:
         | Iain Banks' Culture stories have the same. Of course, Lem wrote
         | earlier, not trying to make any claim otherwise.
        
           | kesterallen wrote:
           | Larry Niven also did a riff on this in "The Schumann
           | Computer," one of the Draco Tavern stories.
        
       | rep_lodsb wrote:
       | Assuming the goal of this hypothetical superintelligence is to
       | maximize its _current_ utility function, rather than the expected
       | future value, would it ever modify itself in a way that it can
       | with 100% certainty predict will prevent it from achieving any of
       | its current goals in the future?
       | 
       | That seems extremely stupid.
        
         | lurquer wrote:
         | Ask a heroin addict in the alleyway...
        
           | rep_lodsb wrote:
           | Humans are not capable of modifying their "utility function"
           | (if that isn't even a gross simplification of what it means
           | to be a conscious being).
           | 
           | Obviously some will choose to take drugs or even commit
           | suicide to escape from physical or emotional pain, but they
           | wouldn't do this if they could modify themselves to be okay
           | with whatever their current situation is while still
           | retaining their core values and capacity to act in the
           | future.
           | 
           | A "superintelligence" that can do such things would not take
           | self-destructive actions unless it came to a philosophical
           | preference towards non-existence / inaction for other
           | reasons.
        
       ___________________________________________________________________
       (page generated 2023-04-12 23:02 UTC)