[HN Gopher] I quiz ChatGPT about math
___________________________________________________________________
I quiz ChatGPT about math
Author : herodotus
Score : 34 points
Date : 2022-12-15 23:51 UTC (23 hours ago)
(HTM) web link (billwadge.com)
(TXT) w3m dump (billwadge.com)
| ogogmad wrote:
| If you read this carefully... One way to prove
| that there are infinitely many prime numbers is by using the
| concept of contradiction. Suppose that there are only a finite
| number of prime numbers, say p_1, p_2, ..., p_n. Then consider
| the number N defined as follows: N = p_1 * p_2 * ... *
| p_n + 1 If N is not prime, then it must have a prime
| divisor p that is not one of the primes p_1, p_2, ..., p_n.
| However, this contradicts the assumption that there are only
| finitely many prime numbers, since p is a prime that is not on
| the list. Therefore, N must be a prime number, which means that
| there are at least one more prime number beyond the list p_1,
| p_2, ..., p_n. This contradiction shows that our assumption that
| there are only finitely many prime numbers must be false, so
| there must be infinitely many prime numbers.
|
| It's right! But there's a pointless appeal to the primality or
| non-primality of N. That can be cut out.
| bumbledraven wrote:
| Good catch. The proof could simply state that, like all
| integers > 1, N must have a prime divisor, and that divisor
| cannot be any of the listed primes, since dividing N by any of
| them leaves remainder 1.
| Waterluvian wrote:
| I ran into what I think is a fascinating example of a current
| limitation of it to "understand" (whatever you want that to mean
| for an AI).
|
| I asked it to make a Python generator that yielded primes. It did
| so and explained how it worked. It was a very inefficient
| approach but worked fine and was clean.
|
| _" Okay, now instead of printing the output, can you write it to
| a sqlite database?"_ And it modified the code in a pretty good
| way.
|
| _" Great, can you modify it so that it only return primes
| greater than 2?"_ It did so fastidiously, adding a check at the
| end.
|
| _" Can a prime number ever be even?"_ It explained that 2 is the
| only even prime.
|
| _" Can you modify the function so that it only returns even
| primes greater than 2?"_ And so it added a conditional check at
| the end which guaranteed that the function would never return
| again.
|
| As a bonus, I just asked it again:
|
| "No, prime numbers are defined as positive integers that have
| exactly two positive integer factors: 1 and themselves. Since
| even numbers are defined as integers that are divisible by 2,
| prime numbers can never be even. The first few prime numbers are
| 2, 3, 5, 7, 11, 13, 17, and 19. All of these numbers are odd."
| rvz wrote:
| Anyone who has bothered to cut through the hype of ChatGPT, would
| already have realised that it isn't smart at math and is only
| regurgitating the results that it has been already been trained
| on and will present it to you to make itself 'appear'
| intelligent.
|
| It doesn't even know why it is gives you the wrong answer even
| when it is presenting the generated output as correct. Like what
| another commenter said, it has limited or in this case has little
| to no reasoning; because it is unable to transparently explain
| itself and cannot present novel solutions to unseen and unsolved
| problems in mathematics.
|
| ChatGPT is as good as an abacus for mathematics, with the
| explainability of a brainless parrot.
| jameshart wrote:
| Nobody has hyped chatGPT as being good at math
| visarga wrote:
| GPT can be the generative (imagination) part of a system. The
| second part would be a system to validate. This means you can
| filter out the junk and keep the good samples, extending the
| training set. This is how the model can learn by performing
| massive search for solutions all on its own. A bit like
| AlphaGo.
|
| So you might want to reevaluate it. It can sample really well,
| and it is hard to sample coherently in this combinatorial
| space. Sampling is half the task, validation needs more work.
| SamBam wrote:
| I was actually very impressed with some math questions it was
| able to answer for me:
|
| _S: If flabb means three, snabb means four, and whabb mean five,
| what is flabb plus whabb?_
|
| If flabb means three, snabb means four, and whabb means five,
| then flabb plus whabb would be equal to three plus five, or
| eight. In this made-up numbering system, the words "flabb,"
| "snabb," and "whabb" represent the numbers three, four, and five,
| respectively. When these numbers are added together, the result
| is the number eight.
|
| _S: If bbb means three, bbbb means four and bbbbb means five,
| how do you think you would write six?_
|
| If bbb means three, bbbb means four, and bbbbb means five, then
| six could potentially be represented by the word "bbbbbbb." In
| this made-up numbering system, the number of "b"s in a word
| corresponds to the value of the number it represents. Therefore,
| to represent the number six, one possible way would be to add
| another "b" to the end of the word for five, which would give us
| "bbbbbbb" as the representation of six. This is just one possible
| way to represent six in this numbering system, and there may be
| other ways to do it as well.
|
| _S: Why do people argue about the answer of mathematical
| questions like "8 / 2(2+2)"?_
|
| There are a few reasons why people may argue about the answer to
| a mathematical question like "8 / 2(2+2)". One reason is that
| there is more than one wav to interpret the question and arrive
| at a different answer. For example, some people may interpret the
| question as "8 / (2 * (2+2))" and arrive at an answer of 1, while
| others mav interpret it as "(8 / 2) * (2+2)" and arrive at an
| answer of 16.
| [deleted]
| [deleted]
| SketchySeaBeast wrote:
| > If flabb means three, snabb means four, and whabb means five,
| then flabb plus whabb would be equal to three plus five, or
| eight. In this made-up numbering system, the words "flabb,"
| "snabb," and "whabb" represent the numbers three, four, and
| five, respectively. When these numbers are added together, the
| result is the number eight.
|
| Was its training model a freshman university student trying to
| get through a 10 page essay?
| kevinventullo wrote:
| That is very cool, and surprising to me given I was also
| initially underwhelmed.
|
| One nitpick though: in the second question, although the
| explanation itself is clean, there appear to be seven b's in
| its representation of the number six.
| BoppreH wrote:
| I've heard it's because GPT's input is whole words/tokens,
| not characters. So it has little insight into their spelling
| or letter positions, unless it's explicitly mentioned in the
| training set (e.g. rhymes are easy, counting letters of rare
| words is hard).
|
| Though this is a specially bad example, surely it had bbbbbb
| somewhere in it's training set associated with "6".
| wyldfire wrote:
| > This is just one possible way to represent six in this
| numbering system, and there may be other ways to do it as
| well.
|
| Gotta hand it to ChatGPT. It's not wrong.
| wyldfire wrote:
| > There are a few reasons why people may argue about the answer
| to a mathematical question like "8 / 2(2+2)". One reason is
| that there is more than one wav to interpret the question and
| arrive at a different answer. For example, some people may
| interpret the question as "8 / (2 * (2+2))" and arrive at an
| answer of 1, while others mav interpret it as "(8 / 2) * (2+2)"
| and arrive at an answer of 16.
|
| Wow, this is kinda astonishing. Could it ruminate about the
| precedence of the operators?
|
| Also: the words "may" and "way" appear as "mav" and "wav" in
| your text. Is this a problem with your input to the HN comment
| or did the model really make these textual errors?
| LegitShady wrote:
| its odd to me that it doesn't explicitly mention 'order of
| operations'.
| wendyshu wrote:
| Should say "Lebesgue measure" (or "Borel measure") instead of
| "measure" a few places
| padjo wrote:
| If ChatGPT is what we can expect from AI I'm quite afraid. It
| seems to be usually correct but often spectacularly and
| confidently wrong. I hope we never give this technology decision
| making power.
| copperx wrote:
| Adding a confidence score to every answer doesn't look like an
| insurmountable problem.
| wslh wrote:
| This is what we first tried two weeks ago and posted in HN:
| https://news.ycombinator.com/item?id=33840405
| constantcrying wrote:
| >Does every set have a measure?
|
| >Not every set has a measure. In order for a set to have a
| measure, it must satisfy certain properties, such as being
| countably infinite or having a well-defined notion of "size" or
| "length." For example, the set of real numbers has a measure
| (namely, its Lebesgue measure), but the set of all subsets of the
| real numbers does not have a measure. In general, only certain
| types of sets can be assigned a measure in a meaningful way.
|
| This response is just nonsense. Very interestingly the question
| is phrased using colloquial terminology and ChatGPT answers in a
| way where it talks about the colloquial phrase as if it were real
| mathematical terminology, in turn getting extremely confused and
| making stuff up.
|
| (The reason the response is wrong is because the central point is
| lost. Being _measurable_ is a property of the measure. ChatGPT
| claims some set isn 't measurable while not naming _under which
| measure_ , which is just nonsense. In fact the real numbers might
| "have no measure" e.g. under the empty measure and the set of all
| subsets of the real numbers might "have a measure" e.g. under the
| counting measure.)
| [deleted]
| jfzoid wrote:
| I just tried doing this myself
|
| JFZ: how do we know there are infinitely many primes
|
| ChatGPT: One way to show that there are infinitely many prime
| numbers is to use the fact that the set of prime numbers is
| infinite to show that there must be an infinite number of
| composite numbers as well. Composite numbers are positive
| integers that have at least one positive integer divisor other
| than 1 and itself. For example, 4 is a composite number because
| it can be divided by 2 (and is not prime), and 6 is a composite
| number because it can be divided by 2 and 3 (and is not prime).
|
| To show that there are infinitely many composite numbers,
| consider the sequence of numbers 2, 4, 6, 8, 10, 12, 14, 16, and
| so on. Each of these numbers is composite because it can be
| divided by 2. Now consider the sequence of numbers 3, 6, 9, 12,
| 15, 18, 21, 24, and so on. Each of these numbers is composite
| because it can be divided by 3. And so on. For every prime
| number, there is an infinite sequence of composite numbers that
| can be generated by multiplying that prime number by other
| positive integers.
|
| Since there are infinite sequences of composite numbers, and each
| composite number is the product of at least two prime numbers
| (for example, 12 is the product of 2 and 6, which are both prime
| numbers), there must be an infinite number of prime numbers as
| well. This is one way to show that there are infinitely many
| prime numbers, though there are other ways to prove it as well.
|
| (seems hand wavey, and probably wrong though I can't figure out
| where the flaw is)
|
| JFZ: is A5 a solvable group
|
| Yes, the group A5 (also known as the alternating group on 5
| elements) is a solvable group. A group is solvable if it can be
| constructed from simpler groups using only group extensions and
| group quotients, and if it has a composition series in which each
| factor is a simple group. (wrong)
|
| The group A5 is the group of even permutations of five objects.
| It has 60 elements, and it is isomorphic to the group of
| rotations of a regular dodecahedron. A5 is a simple group, which
| means that it has no proper normal subgroups (subgroups that are
| normal in the group but not equal to the group itself). This
| means that A5 cannot be constructed from simpler groups using
| group extensions or group quotients. However, it can be shown
| that A5 has a composition series in which each factor is a simple
| group, so it is solvable.
|
| In general, many groups that arise in mathematics and physics are
| solvable, and the study of solvable groups is an important area
| of research in group theory.
| chrsjxn wrote:
| The first proof is basically just begging the question.
|
| It says that having infinitely many composite numbers must mean
| infinitely many factors, and thus infinitely many primes.
|
| But we can construct infinite sets of numbers from finite prime
| factors easily. [2] and the powers of 2.
|
| Which might be a good starting point to teach someone Euclid's
| proof that there are infinite primes, but ChatGPT didn't get
| all the way there.
| dxbydt wrote:
| Here's a terribly amusing one on Z3.
| https://imgur.com/a/RZwHkLV
|
| First it says I don't know the answer to 1+2. Then it says
| ofcourse 1+2 must be 1. In either case, its the same exact
| question, and if you assert closure on 3 elements, with 1+1=3
| and 1+3=2, you are left with no choice other than 1+2=1 because
| closure and Cayley Hamilton. The very first property it lists
| is closure! So it knows what closure is, but doesn't know how
| to use it. But give it the answer to 2+1 and suddenly it uses
| associativity !
| omnicognate wrote:
| I haven't read through the rest, and I don't particularly mean
| anything by it as far as GPT's capabilities are concerned, but I
| can't let a self-described maths PhD calling the answer to the
| first one "entirely correct" slide.
|
| > is given by the square root of the difference of the squares of
| the x-coordinates plus the difference of the squares of the
| y-coordinates.
|
| This would be (in the example it then gives) either `sqrt(x^2 -
| a^2) + y^2 - b^2` or `sqrt(x^2 - a^2 + y^2 - b^2)`, both entirely
| incorrect.
|
| Which does illustrate something important about _human_
| capabilities. A person who knows the right answer is likely to
| see the right answer when presented with a wrong answer that 's
| sufficiently close to it.
| eminence32 wrote:
| My favorite ChatGPT answer about math is:
|
| Me: What is the 19th digit of pi
|
| ChatGPT: The value of pi is approximately 3.14159265359, and the
| 19th digit is 5. However, it is important to note that the digits
| of pi continue infinitely, so this is only an approximation of
| the 19th digit.
| [deleted]
| jameshart wrote:
| People love pointing out how bad GPT3 is at math - it's just not
| remotely surprising, though. We all learned a long time ago that
| just patternmatching isn't enough to solve math questions and we
| need to actually activate our brains and run algorithms to get
| valid answers.
|
| Even mathy people fall into this - we all know that some numbers
| 'look prime'. And up to 100 that instinct can work really well
| (except for 91).
|
| But we all apply a heuristic for when to stop guessing and start
| mathing.
|
| But guess what! GPT3 can do that too!
|
| https://twitter.com/goodside/status/1568448128495534081
|
| This thread predates chatgpt - but it really shows that GPT can
| pretty reliably spot when winging it won't work. And in this
| case, it could effectively be given the ability to whip out a
| calculator to figure it out.
|
| (Edited to add)
|
| Found the follow up thread where Riley went into more detail and
| enhanced the results:
|
| https://twitter.com/goodside/status/1581805503897735168?s=20...
| dxbydt wrote:
| Here's a very easy middle school competition problem -
| https://imgur.com/a/YdFSBVY
|
| It repeatedly got the answer wrong. I engaged in a long
| conversation with it (too long to post) where I tried to convince
| it that its answer was wrong, but it became quite rude at a
| certain point & alleged I was trying to trick it or something
| (?!) so I just gave up. Unfortunately, I was doing the whole
| exercise in front of my 10 year old who was initially very
| excited about ChatGPT. After seeing its behavior, he is now
| convinced that its dumb as dirt.
|
| If you look at the problem, ChatGPT's answer is the 2nd term of
| the input series! Obviously an all positive series can't have one
| of its own terms as an answer, the answer better be bigger than
| each of its terms. So forget about calculation, it has no logical
| reasoning ability either.
| HDThoreaun wrote:
| You should basically be thinking of chatGPT as a personal
| Wikipedia. It has the collective knowledge of the internet. It
| can still be wrong about fact based questions, but is a good
| starting point. It's not effective at doing math because that's
| not what it was designed to do.
| MisterSandman wrote:
| ChatGPT definitely way more things than Wikipedia does.
| ogogmad wrote:
| But does it accidentally words as much?
| panic wrote:
| ChatGPT, the free encyclopedia that no one can edit!
| dxbydt wrote:
| If I post that same question on Wikipedia and confidently
| assert the answer is 1, some good Samaritan will edit it
| within a day & say no dumbass its 6, and then link me to the
| Talk page & we'll duke it out over there. On ChatGPT, you
| have no recourse other than the thumb down button. When you
| give it the right answer it simply refuses to accept! It's
| sure amusing as an adult, but be aware that several schools
| (atleast here in the midwest) are discussing this in the
| classroom - my kid said his teacher was talking about ChatGPT
| and that's why I proceeded to show what it does. So kids
| might be quite confused by this behavior. Better to just go
| with the standard disclaimer "I don't know. As a large
| language model trained by ClosedAI blah blah blah insert
| boilerplate here, I can't figure out the answers to basic
| equations mankind has known for a few centuries now unless
| more VC money forthcoming. Ask Powell to pause his ratehikes
| and market shoots back up, we staff up the org with few
| mathematicians and they'll tell you its 6. Right now its 1,
| suck it dummy"
| satvikpendem wrote:
| You're misunderstanding the parent. They mean that in
| school, teachers tell you to use Wikipedia as simply a
| starting point rather than an authoritative source (indeed,
| look at the article's actual sources and references at the
| bottom and read those), not to literally use Wikipedia to
| perform mathematical operations.
|
| In the same vein, use ChatGPT but verify its answers.
| HDThoreaun wrote:
| Well my point is there aren't pages for specific math
| problems on wikipedia, because that's not the goal of
| wikipedia. ChatGPT can tell you the rules of algebra, but
| it's bad at applying them. If you want to solve math
| problems you should go to wolframalpha.
|
| My personal opinion is that having an encyclopedia that is
| sometimes wrong is actually incredibly valuable for
| educating people in the social media age. Distinguishing
| between content that is true and content that sounds true
| is a skill that used to be unneeded but is now essentially
| a requirement to productively participate in society.
| ChatGPT seems like a decent way to train that skill.
| bawolff wrote:
| I think that is significantly underselling wikipedia's
| reliability in practise.
| teawrecks wrote:
| The point is that Wikipedia is infinitely more fallible
| than a mathematical equation.
| bawolff wrote:
| That doesn't make sense. Math equations are expressions
| of ideas. They can be fallible as much as anything else.
| Just because 1+1=3 is an equation doesn't make it true.
| evo_9 wrote:
| Funny I quiz it about Gnosticism and the Uncreated Father. Much
| more fascinating to find it's non-science-y edges IMO.
| furstenheim wrote:
| I tried with something that was not so obviously in the training
| set, the response is terrible
|
| What's the measure of the set of real numbers whose square is
| between 1 and 4.
|
| The set of real numbers whose square is between 1 and 4 is known
| as the square root of the interval [1, 4]. It is represented by
| the notation [?][1, 4].
|
| To find the measure of this set, we can take the square root of
| each endpoint of the interval and then subtract the smaller value
| from the larger value. In this case, the measure of the set is
| [?]4 - [?]1 = 2 - 1 = 1.
|
| Alternatively, we can rewrite the set as the union of two
| intervals: the set of real numbers whose square is between 1 and
| 2, and the set of real numbers whose square is between 2 and 4.
| The measure of the first interval is [?]2 - [?]1 = 1, and the
| measure of the second interval is [?]4 - [?]2 = 1. Therefore, the
| measure of the set of real numbers whose square is between 1 and
| 4 is 1 + 1 = 2.
|
| It's important to note that the set of real numbers whose square
| is between 1 and 4 is not the same as the set of real numbers
| between 1 and 4. The set of real numbers whose square is between
| 1 and 4 consists only of the positive real numbers 1 and 2, while
| the set of real numbers between 1 and 4 includes all real numbers
| between 1 and 4, including negative numbers
| CrazyStat wrote:
| I quizzed ChatGPT about math a few days back. It correctly
| answers that 1023 is odd, correctly answers that the square of an
| odd number is also odd, and is about 50/50 on whether it decides
| 1023^2 is even or odd.
|
| The author of the linked post repeatedly refers to ChatGPT's
| "reasoning". But ChatGPT doesn't reason, it just generates text.
| Sometimes that text might look like reasoning to a human--it's
| the sort of text that a human who reasoned through the question
| would generate--but there was no reasoning involved. It's
| important to understand this.
|
| > Based on my teaching experience, I'd say chatGPT scored like a
| second year math major.
|
| Based on my experience, ChatGPT scored worse than my 5-year-old,
| who gets the 1023^2 question right every time.
| CrazyStat wrote:
| Prompt:
|
| > is 1023^2 even or odd?
|
| Response:
|
| > 1023 squared is even because it is equal to 1,046,529, which
| is an even number. To determine if a number is even or odd, you
| can check whether it is divisible by 2. If it is divisible by
| 2, it is even. If it is not divisible by 2, it is odd.
|
| > In general, any number that is a perfect square will be even
| if the number being squared is odd, and it will be odd if the
| number being squared is even. This is because odd numbers, when
| squared, always result in an odd number, and even numbers, when
| squared, always result in an even number.
| dnissley wrote:
| Interesting, I got:
|
| > 1023 squared is 1048129, which is an odd number. To
| determine whether a number is even or odd, you can check
| whether it is divisible by 2. If it is, it is an even number.
| If it is not, it is an odd number. In this case, 1048129 is
| not divisible by 2, so it is an odd number.
| [deleted]
| CrazyStat wrote:
| Yeah, like I said it's about 50/50 whether it spits out
| even or odd. Try it a few times and you'll see.
| ablatt89 wrote:
| I reminds me of study methods that involve "memorizing" as
| opposed to thinking about the problem, and applying various
| methods to solve it. It can solve some algorithm problems at a
| certain level, but struggles at a detailed answer in some
| scenarios. I guess it's not surprise, given ML is essentially
| exploring the local solution space of some abstract problem; it
| reminds me of a Taylor series expansion or a breadth-first
| search of an abstract problem space.
| HDThoreaun wrote:
| Your 5 year old is doing number theory? I think I was still
| working on like counting when I was 5
| CrazyStat wrote:
| My five year old knows about evens and odds and squares, if
| that counts as number theory. He's really into
| multiplication.
| awestroke wrote:
| How do you know there is no reasoning involved?
| CrazyStat wrote:
| If there was reasoning it would consistently get the parity
| of 1023^2 correct.
| alar44 wrote:
| These threads are getting tiring. Yes, ChatGPT is shitty at math.
| It's not supposed to be good at it. It's really good at what 99%
| of the population throws at it, but of course the internet dorks
| are huffing and puffing that it can't do advanced math or doesn't
| know some obscure theorem. No one cares and it's not the point.
| LWIRVoltage wrote:
| After reading so much here and this , about chatGPT- I have a
| feeling what will be needed for AI- is teachers.
|
| As it gets smarter and actually gets better reasoning, and
| capability- the way to keep AI sane and kind, will be to try to
| teach it like a person, with it putting importance on lessons
| from designated teachers even more, than it's weight in casual
| conversation , generally.
|
| I think i forsee teachers being hired to teach AI, not just
| humans. Though how much AI will benefit from it will depend on
| their level But we'll have to get to that point first, as i read
| attempts to teach ChatGPT having odd results , which of course
| reveal what it can't do-yet.
|
| At some point, we're going to have to intentionally educate AI- i
| feel like giving them knowledge in a similar manner is not talked
| about enough, even if some feel teaching AI like we do people
| isn't necessary due to AI's other abilities
|
| I don't think i've seen others here consider the idea of seeing
| the o
| CuriouslyC wrote:
| You mean better data set curation and annotation? That's pretty
| much how you teach AI.
|
| It would be funny to have a chatbot interface to the chatbot
| training dataset.
| Jensson wrote:
| > Substituting our values into this formula, we get: x1 = (3 +
| (10 / 3)) / 2 = 5
|
| This is really wrong, why didn't the poster notice this? They
| just said
|
| > 3.4 is not a very good approximation but other than that the
| response is correct.
|
| No, Newtons method produces a really good approximation for
| square root very quickly, ChatGPT didn't do the method correctly
| which is why the result is so bad. People who can't instantly
| spot those mistakes shouldn't use ChatGPT, and people who can
| spot them instantly doesn't need to use ChatGPT.
| majikandy wrote:
| I spotted that too, also the next iteration is 3.5 even though
| 5 should have never been fed in?
___________________________________________________________________
(page generated 2022-12-16 23:01 UTC)