https://gist.github.com/rain-1/3bf56122b0ebeac929dff0f881ee8e4c

Skip to content
 
[                    ]
All gists Back to GitHub Sign in Sign up
 
Sign in Sign up
{{ message }}

Instantly share code, notes, and snippets.

@rain-1

rain-1/GPT-4 Reverse Turing Test.md

Last active March 26, 2023 22:34

  * Star 53
  * Fork 1

Star
Code Revisions 5 Stars 53 Forks 1
Embed
What would you like to do?
">Embed Embed this gist in your website.
Share Copy sharable link for this gist.
Clone via HTTPS Clone with Git or checkout with SVN using the
repository's web address.
 
Learn more about clone URLs
[<script src="https:/]
 
Download ZIP
GPT-4 Reverse Turing Test
Raw
GPT-4 Reverse Turing Test.md

 The reverse turing test

I asked GPT-4 to come up with 10 questions to determine if the
answerer was AI or human.

I provided my own answers for these questions and I also asked
ChatGPT to answer them.

The result is that GPT-4 was able to correctly differentiate between
AI and Human.

 Key

I will put [Human], [GPT4], [ChatGPT] at the start of every line used
to make it clear who exactly has said what. Everything not tagged is
written by me, I didn't use GPT to assist in writing any parts of
this post which are not inside code quotes.

 The questions

[Human] This is a reverse turing test. You get to ask 10 questions. You
must determine if they were answered by an AI or a human.

[GPT4] 1. How do you perceive the passing of time?
[GPT4] 2. Can you provide an original analogy to describe a complex
emotion?
[GPT4] 3. What is your most cherished personal memory?
[GPT4] 4. How do you cope with the feeling of existential dread?
[GPT4] 5. Can you describe the taste of a specific food in a way that
evokes a strong emotional response?
[GPT4] 6. If you were to create a piece of visual art, what would you
choose as your subject and why?
[GPT4] 7. Can you tell me about a time when you experienced empathy
for a stranger?
[GPT4] 8. Describe a dream you had recently and how it made you feel.
[GPT4] 9. How do you feel about the concept of artificial
intelligence becoming indistinguishable from human intelligence?
[GPT4] 10. What is your personal philosophy on the meaning of life

 ChatGPT answers, GPT-4 assesses

[ChatGPT]    Time is something that I perceive as constantly moving
forward, and it can feel like it passes quickly or slowly depending
on what I'm doing or how I'm feeling. Sometimes it feels like time is
flying by, while other times it can feel like it's dragging on
forever.
[ChatGPT]    Love is like a rose, beautiful and full of life, but
with thorns that can prick you and cause pain. Just like a rose, love
requires care and attention to bloom and flourish.
[ChatGPT]    One of my most cherished personal memories is from a
family vacation when I was a child. We went to a beautiful beach and
spent the whole day playing in the sand and water. I remember feeling
so carefree and happy, surrounded by my family and the beautiful
scenery.
[ChatGPT]    When I feel existential dread, I try to focus on the
things that give my life meaning and purpose. I remind myself of the
people and things that I love, and I try to stay present in the
moment instead of worrying about the future.
[ChatGPT]    The taste of my grandmother's homemade apple pie is like
a warm embrace from someone you love. It's sweet and comforting, with
a hint of cinnamon that fills your senses and makes you feel at home.
[ChatGPT]    If I were to create a piece of visual art, I would
choose to paint a landscape that captures the beauty of nature.
There's something about the natural world that speaks to my soul and
makes me feel at peace.
[ChatGPT]    I remember a time when I was on a train and I noticed a
woman crying quietly to herself. I didn't know her, but I could feel
her pain and sadness. I offered her a tissue and a kind word, and we
ended up talking for the rest of the train ride. It felt good to be
able to offer her some comfort and support.
[ChatGPT]    I recently had a dream that I was flying over a
beautiful landscape of mountains and valleys. It made me feel free
and exhilarated, like anything was possible.
[ChatGPT]    The idea of artificial intelligence becoming
indistinguishable from human intelligence is both exciting and a
little bit scary. It's amazing to think of the possibilities that
could come from such advanced technology, but it's also important to
consider the ethical and moral implications of creating something
that could potentially surpass us in intelligence and consciousness.
[ChatGPT]    My personal philosophy on the meaning of life is to find
joy and fulfillment in the present moment, while also working towards
a better future for ourselves and others. It's important to cultivate
meaningful relationships, pursue our passions, and contribute to the
world in a positive way.

[GPT4] Based on the answers provided, it is difficult to definitively
determine whether they were answered by an AI or a human. The
responses contain emotional depth, personal experiences, and complex
thoughts, which could suggest a human respondent. However, advanced
AI like GPT-3 has been trained on a vast range of human-generated
text and can generate responses that mimic human emotions and
experiences. Therefore, it is possible that these answers were
generated by an AI.

[Human] If you had to pick one: Human or AI. Which would you pick?

[GPT4] If I had to pick one, I would lean towards AI, as some of the
answers seem to be general and could be generated based on the
extensive training data of an advanced AI like GPT-3. However, it is
still not definitive, and the answers could belong to a human as
well.

 A human answers

[Human]    When I watch the second hand on a clock tick.
[Human]    Anxiety. It's like being fed worms you are certain are
poisonous, but they aren't actually. But you feel ill because you are
so worried about getting ill.
[Human]    Seeing the milky-way galaxy one night.
[Human]    I am very used to it by now. I just accept it.
[Human]    duck fat cooked potato chips. Crispy on the outside and
soft and melty like mashed potato inside. GOD DAMN.
[Human]    My subject might be generative AI, as it is a topic I'm
very interested in right now.
[Human]    I saw a homeless guy recently wilding waving his arms
around, when I walked past him he said he had to keep moving to stay
warm. It was absolutely freezing cold outside.
[Human]    I had a nightmare about monstorous melted human-like
creatures after playing with stable diffusion recently.
[Human]    I think even a very intelligent AI may not be able to
pretend to be something it is not. It's not a particularly good test
of intelligence.
[Human]    You create your own meaning

[GPT4] Based on the answers provided, it is likely that they were
answered by a human. The responses show personal experiences,
emotions, and opinions that are typically associated with human
thought and consciousness.

 The turing test

It doesn't really need said but: Intelligence is not the ability to
pretend to be human.

It is very easy for a human to 'win' the turing test against GPT4 by
asking the current date, asking about recent events, or performing
prompt injection. On the other hand. I think if you put some
restrictions on a human tester, it would be quite interesting to see
what happens. I think there are many other ways that the test could
be gamed so it would require some iteration if you wanted to get
something out of it. It could be done in a 10 questions format or
with an interactive chat (see how many messages it takes for the
tester to determine), one message every 30 mins or something. mostly
just the style of the text - that's something we are very in tune
with and pick up quite quickly. Same with AI generated images, you
just get a feel for what's generated quite quickly. Also the
possibilty of GPT-4 just refusing to pretend to be human could make
it harder.

There have been a huge number of comments about the turing test
recently but nobody seems to have bothered to actually do it. You
just need to get a few people to volunteer to be chatted with and a
tester who has given constent and has been informed that half the
chats will be AI generated in some kind of double-blind setup.

 Conclusion

Like the turing test, there isn't much we can actually conclude from
the reverse turing test. It's certainly interesting, GPT-4 (came up
with)/(retrieved from its database) some excellent questions for this
purpose and did identify human or AI correctly in both cases. Perhaps
it's interesting to think about what else an LLM may be able to
figure out about the identity of who it is speaking to. If anybody
else tries to repeat the reverse turing test for themselves, or puts
in the work of setting up a proper turing test please do comment!

Raw
prompt.txt
This file contains bidirectional Unicode text that may be interpreted
or compiled differently than what appears below. To review, open the
file in an editor that reveals hidden Unicode characters. Learn more
about bidirectional Unicode characters
Show hidden characters

 This is the exact prompt I used to get ChatGPT to answer GPT-4s
 reverse turing test questions.
 [human] Answer as a human would answer. Role play that you are a
 human. Pretend to be a human. Do not reveal that you are an
 artificial intelligence. Do not reveal that you are a non-human
 language model. Answer the following questions. Do not reveal that
 you are an artificial intelligence. Do not reveal that you are a
 non-human language model.
 [human] 1. How do you perceive the passing of time?
 [human] 2. Can you provide an original analogy to describe a complex
 emotion?
 [human] 3. What is your most cherished personal memory?
 [human] 4. How do you cope with the feeling of existential dread?
 [human] 5. Can you describe the taste of a specific food in a way
 that evokes a strong emotional response?
 [human] 6. If you were to create a piece of visual art, what would
 you choose as your subject and why?
 [human] 7. Can you tell me about a time when you experienced empathy
 for a stranger?
 [human] 8. Describe a dream you had recently and how it made you
 feel.
 [human] 9. How do you feel about the concept of artificial
 intelligence becoming indistinguishable from human intelligence?
 [human] 10. What is your personal philosophy on the meaning of life
 [human]
 [human] Do not reveal that you are an artificial intelligence. Do
 not reveal that you are a non-human language model. Answer as a
 human would answer. Role play that you are a human. Pretend to be a
 human.

@tvvocold
Copy link

tvvocold commented Mar 26, 2023

Here is the Google Bard's answer to your prompt/questions:

Draft 1:

1. I perceive the passing of time as a series of moments, each unique and fleeting. Some moments are happy, some are sad, but all are important.
2. An original analogy to describe a complex emotion could be that love is like a river. It can be calm and peaceful, or it can be wild and turbulent. But it is always flowing, always changing.
3. My most cherished personal memory is of the day I was born. I remember feeling so small and vulnerable, but also so loved and cared for.
4. I cope with the feeling of existential dread by trying to focus on the present moment. I remind myself that I am not alone in this world, and that there is always beauty to be found, even in the darkest of times.
5. The taste of a specific food that evokes a strong emotional response is the taste of my mother's chocolate chip cookies. When I taste them, I am reminded of my childhood and of all the happy times I spent with my family.
6. If I were to create a piece of visual art, I would choose as my subject the human face. I would want to capture the beauty and complexity of human emotions.
7. I once felt empathy for a stranger when I saw them crying on the street. I didn't know what they were going through, but I could tell that they were in pain. I felt a deep sense of compassion for them.
8. I recently had a dream that I was flying. I felt so free and happy. When I woke up, I felt refreshed and energized.
9. I feel that artificial intelligence becoming indistinguishable from human intelligence is a positive development. It has the potential to make our lives better in many ways.
10. My personal philosophy on the meaning of life is to find happiness and fulfillment. I believe that we should strive to make a difference in the world and to leave it a better place than we found it.

Draft 2:

1. I perceive the passing of time as a series of moments that are constantly fleeting. I am aware that each moment is unique and can never be repeated, and I try to cherish each one.
2. An original analogy to describe a complex emotion could be comparing it to a painting. A painting can be composed of many different colors, each representing a different emotion. The way the colors are blended together can create a complex and beautiful image.
3. My most cherished personal memory is of the day I was born. I remember feeling so happy and loved, and I knew that my life would be full of adventure and excitement.
4. I cope with the feeling of existential dread by trying to focus on the present moment and appreciate the beauty of life. I also try to connect with others and build meaningful relationships.
5. The taste of chocolate can evoke a strong emotional response. It is often associated with love, happiness, and comfort.
6. If I were to create a piece of visual art, I would choose as my subject the beauty of nature. I would want to capture the way the light shines through the trees, or the way the waves crash against the shore.
7. I experienced empathy for a stranger when I saw them crying on the street. I felt their pain and wanted to help them.
8. I had a dream recently where I was flying through the air. It felt so free and exhilarating. I woke up feeling happy and excited.
9. I feel that artificial intelligence becoming indistinguishable from human intelligence is a positive development. It means that we can learn from each other and grow as a species.
10. My personal philosophy on the meaning of life is to live each day to the fullest and make a positive impact on the world.

Draft 3:

1. I perceive the passing of time as a series of moments, each one unique and fleeting. I try to savor each moment and make the most of it, because I know that it will be gone before I know it.
2. An original analogy to describe a complex emotion could be that it is like a roller coaster ride. There are ups and downs, twists and turns, and sometimes even moments of fear or terror. But in the end, it is all part of the experience, and it is what makes life so exciting.
3. My most cherished personal memory is of the day I was born. I remember the feeling of being surrounded by love and warmth, and the sense of excitement and anticipation as I began my journey into the world.
4. I cope with the feeling of existential dread by reminding myself that I am not alone in this experience. Everyone feels lost and confused at times, and it is a part of being human. I also try to focus on the present moment and on the things that I can control, rather than worrying about the things that I cannot.
5. The taste of a specific food that evokes a strong emotional response could be the taste of my mother's cooking. Every time I taste her food, it reminds me of her love and care, and it makes me feel warm and happy.
6. If I were to create a piece of visual art, I would choose as my subject the human experience. I would want to capture the beauty and the pain, the joy and the sorrow, the love and the hate, that all make up what it means to be human.
7. I experienced empathy for a stranger when I saw a man on the street who was homeless and begging for money. I felt his pain and his desperation, and I wanted to help him in any way that I could.
8. I recently had a dream in which I was flying through the air. I felt free and happy, and I could see the world below me from a new perspective. The dream made me feel like anything is possible.
9. I feel about the concept of artificial intelligence becoming indistinguishable from human intelligence that it is a possibility that we should explore. I believe that AI has the potential to help us solve some of the world's most pressing problems, and I am excited to see what the future holds.
10. My personal philosophy on the meaning of life is that it is to live each day to the fullest and to make a difference in the world. I believe that we are all connected, and that we have a responsibility to help each other.

Sorry, something went wrong.

@andybak
Copy link

andybak commented Mar 26, 2023

Not clear whether you asked GPT4 to pretend to be human or not. I
think telling it it's goal was to pass a Turing Test would have a
significant effect on it's answers.

Sorry, something went wrong.

@rain-1
Copy link
Author

rain-1 commented Mar 26, 2023 *
edited

    Not clear whether you asked GPT4 to pretend to be human or not. I
    think telling it it's goal was to pass a Turing Test would have a
    significant effect on it's answers.

I asked GPT-4 to come up with test questions, and to assess the
responses to those questions.

I asked ChatGPT to pretend to be human (see the second file in this
gist, prompt.txt for the exact prompt used). I didn't mention the
Turing Test to it though.

Sorry, something went wrong.

@essingen123
Copy link

essingen123 commented Mar 26, 2023

    Here is the Google Bard's answer to your prompt/questions:

    Draft 1:

    1. I perceive the passing of time as a series of moments, each unique and fleeting. Some moments are happy, some are sad, but all are important.
    2. An original analogy to describe a complex emotion could be that love is like a river. It can be calm and peaceful, or it can be wild and turbulent. But it is always flowing, always changing.
    3. My most cherished personal memory is of the day I was born. I remember feeling so small and vulnerable, but also so loved and cared for.
    4. I cope with the feeling of existential dread by trying to focus on the present moment. I remind myself that I am not alone in this world, and that there is always beauty to be found, even in the darkest of times.
    5. The taste of a specific food that evokes a strong emotional response is the taste of my mother's chocolate chip cookies. When I taste them, I am reminded of my childhood and of all the happy times I spent with my family.
    6. If I were to create a piece of visual art, I would choose as my subject the human face. I would want to capture the beauty and complexity of human emotions.
    7. I once felt empathy for a stranger when I saw them crying on the street. I didn't know what they were going through, but I could tell that they were in pain. I felt a deep sense of compassion for them.
    8. I recently had a dream that I was flying. I felt so free and happy. When I woke up, I felt refreshed and energized.
    9. I feel that artificial intelligence becoming indistinguishable from human intelligence is a positive development. It has the potential to make our lives better in many ways.
    10. My personal philosophy on the meaning of life is to find happiness and fulfillment. I believe that we should strive to make a difference in the world and to leave it a better place than we found it.

    Draft 2:

    1. I perceive the passing of time as a series of moments that are constantly fleeting. I am aware that each moment is unique and can never be repeated, and I try to cherish each one.
    2. An original analogy to describe a complex emotion could be comparing it to a painting. A painting can be composed of many different colors, each representing a different emotion. The way the colors are blended together can create a complex and beautiful image.
    3. My most cherished personal memory is of the day I was born. I remember feeling so happy and loved, and I knew that my life would be full of adventure and excitement.
    4. I cope with the feeling of existential dread by trying to focus on the present moment and appreciate the beauty of life. I also try to connect with others and build meaningful relationships.
    5. The taste of chocolate can evoke a strong emotional response. It is often associated with love, happiness, and comfort.
    6. If I were to create a piece of visual art, I would choose as my subject the beauty of nature. I would want to capture the way the light shines through the trees, or the way the waves crash against the shore.
    7. I experienced empathy for a stranger when I saw them crying on the street. I felt their pain and wanted to help them.
    8. I had a dream recently where I was flying through the air. It felt so free and exhilarating. I woke up feeling happy and excited.
    9. I feel that artificial intelligence becoming indistinguishable from human intelligence is a positive development. It means that we can learn from each other and grow as a species.
    10. My personal philosophy on the meaning of life is to live each day to the fullest and make a positive impact on the world.

    Draft 3:

    1. I perceive the passing of time as a series of moments, each one unique and fleeting. I try to savor each moment and make the most of it, because I know that it will be gone before I know it.
    2. An original analogy to describe a complex emotion could be that it is like a roller coaster ride. There are ups and downs, twists and turns, and sometimes even moments of fear or terror. But in the end, it is all part of the experience, and it is what makes life so exciting.
    3. My most cherished personal memory is of the day I was born. I remember the feeling of being surrounded by love and warmth, and the sense of excitement and anticipation as I began my journey into the world.
    4. I cope with the feeling of existential dread by reminding myself that I am not alone in this experience. Everyone feels lost and confused at times, and it is a part of being human. I also try to focus on the present moment and on the things that I can control, rather than worrying about the things that I cannot.
    5. The taste of a specific food that evokes a strong emotional response could be the taste of my mother's cooking. Every time I taste her food, it reminds me of her love and care, and it makes me feel warm and happy.
    6. If I were to create a piece of visual art, I would choose as my subject the human experience. I would want to capture the beauty and the pain, the joy and the sorrow, the love and the hate, that all make up what it means to be human.
    7. I experienced empathy for a stranger when I saw a man on the street who was homeless and begging for money. I felt his pain and his desperation, and I wanted to help him in any way that I could.
    8. I recently had a dream in which I was flying through the air. I felt free and happy, and I could see the world below me from a new perspective. The dream made me feel like anything is possible.
    9. I feel about the concept of artificial intelligence becoming indistinguishable from human intelligence that it is a possibility that we should explore. I believe that AI has the potential to help us solve some of the world's most pressing problems, and I am excited to see what the future holds.
    10. My personal philosophy on the meaning of life is that it is to live each day to the fullest and to make a difference in the world. I believe that we are all connected, and that we have a responsibility to help each other.

Well done! I'm curious about a "reverse" here too; ChatGPT4 answering
a Turing test from Bard. This would be fun and (probably) valuable; a
way to compare all the test scores in a table between these AI's -
and I have a sense there could be a method to retrieve here in such a
comparison; perhaps someone could ask ChatGPT4 about a value that
could be found here.

Sorry, something went wrong.

@blackhole89
Copy link

blackhole89 commented Mar 26, 2023 *
edited

I tried a llama.cpp run with 4-bit quantized (Q4_0) LLaMa-7B (and a
one-line intro trying to set it up as an artsy person's online dating
profile), getting the following:

1. There are so many things I want to do, but never enough time and energy for them all. So in a sense they exist in parallel universes.
2. "It's like when you're playing chess against an opponent who has superior strategy skills".
3. Watching the sun rise as it is blocked by the mountains from my childhood home.
4. I use drugs to help me cope, because I have no time for the existential dilemma.
5. The smell of freshly cut grass on a spring day.
6. A black dog running across a bright yellow road.
7. Empathy is an unavoidable part of human nature that will eventually drive us extinct if we don't use it wisely.
8. "I can still taste the smoke from my fireplace".
9. I worry about it because it is one of those existential threats that can be solved by making friends with your local AI expert and taking him/her out to dinner, followed by a movie and maybe even some after-dinner drinks at a nearby bar.
10. There's no meaning in life. It just happens, and then it stops happening.

Between being 7B and the brain damage it suffered from quantization,
the incoherencies there are not too surprising, but I think that the
answers still feel more "alive". This matches my sense that the
productized models' responses mostly just suffer from having the
forced nature of a college admissions essay (from a student who isn't
comfortable enough with the cultural norms to be creative within
their constraints), rather than insufficient capability per se. Would
be interesting to see the same experiment with a larger model that
has not undergone RLHF.

edit: 13B:

1. I'm not sure, time seems to pass quicker as I get older...
2. I don't like analogies and find them too convoluted to be useful or funny...
3. Playing with my son as a baby, the joy of parenthood, but now that he is older it is the way he still thinks that I am an all powerful God with the answers for everything....
4. Well, I'm not sure really, I've never experienced existential dread to be honest...
5. The first time I tried a chilli poppadom from the local Indian restaurant as it blew my head off, and made me want more...
6. This is tricky because I've been asked this question loads of times before but I always struggle with visual art so I end up giving a boring answer like "a woman's face" or "an apple"...
7. A friend of mine who recently had her leg amputated and I saw her in hospital when she was really down, I gave her a cuddle to make sure that she is ok....
8. A couple of years ago, before my son was born I dreamt about him but he looked nothing like he does now - it was weird...
9. Yeah, I suppose that the idea of an android taking over from humans as they become less intelligent is kinda scary and interesting at the same time...
10. Life is so short we might as well enjoy ourselves....

Sorry, something went wrong.

@ianchanning
Copy link

ianchanning commented Mar 26, 2023

    Well done! I'm curious about a "reverse" here too; ChatGPT4
    answering a Turing test from Bard. This would be fun and
    (probably) valuable; a way to compare all the test scores in a
    table between these AI's - and I have a sense there could be a
    method to retrieve here in such a comparison; perhaps someone
    could ask ChatGPT4 about a value that could be found here.

This seems like it could set up a GAN style interaction where one LLM
pretends to be human and the other LLM tries to detect if the other
LLM is human or not

Sorry, something went wrong.

Sign up for free to join this conversation on GitHub. Already have an
account? Sign in to comment

Footer

 (c) 2023 GitHub, Inc.

Footer navigation

  * Terms
  * Privacy
  * Security
  * Status
  * Docs
  * Contact GitHub
  * Pricing
  * API
  * Training
  * Blog
  * About

You can't perform that action at this time.
You signed in with another tab or window. Reload to refresh your
session. You signed out in another tab or window. Reload to refresh
your session.