[HN Gopher] LLaMA2 Chat 70B outperformed ChatGPT
___________________________________________________________________
LLaMA2 Chat 70B outperformed ChatGPT
Author : georgehill
Score : 286 points
Date : 2023-07-27 15:44 UTC (7 hours ago)
(HTM) web link (tatsu-lab.github.io)
(TXT) w3m dump (tatsu-lab.github.io)
| 0xbadc0de5 wrote:
| And Vicuna-33B is not far behind - which you can actually run on
| a 24GB 3090/4090 GPU unlike LLaMAv2-70B. Although for a lot of
| tasks, Guanaco outperforms Vicuna.
| heliophobicdude wrote:
| The benchmark I care about the most for my development workflow
| is on structured output.
|
| Paul Gauthier made this benchmark [1] to measure correct git
| diffs. If you ask GPT-4 for help with your code, it can output a
| change in a git diff more reliably than 3.5.
|
| My hope is that we can do that with Llama 2.
|
| 1:https://aider.chat/docs/benchmarks.html
| coldblues wrote:
| Awesome. What's great is that it can be unfiltered as well. So
| we'll be able to have models without all that incessant
| apologizing.
| accrual wrote:
| Does this mean it may be possible to self-host a ChatGPT clone
| assuming you have a 70B model? I've used a 13B model with LLaMA1
| and it's surprisingly good, but still nowhere near ChatGPT for
| coding questions.
| ramesh31 wrote:
| >Does this mean it may be possible to self-host a ChatGPT clone
| assuming you have a 70B model?
|
| Not only possible but quite easy. Inference for 70B can be done
| with llama.cpp using CPU only, on any commodity hardware with
| >64GB of RAM
| beefield wrote:
| I have 64gb on my 5 year old thinkpad. What kind of
| performance (tokens per sec) I could expect on that nowadays
| for a 70B model?
| jerrygenser wrote:
| Llama cpp speed is dramatically improved by avx
| instructions. If your CPU has those it would be much faster
| than not.
|
| And if it doesn't you need to do some workarounds with
| compiling and it gets a bit harder to run.
| Zambyte wrote:
| When you say "coding questions" do you mean questions that
| should be answered by producing code, or questions about code
| ("explain this")? Or both?
| lhl wrote:
| You will want to look at HumanEval
| (https://github.com/abacaj/code-eval) and Eval+
| (https://github.com/my-other-github-account/llm-humaneval-
| ben...) results for coding.
|
| While Llama2 is an improvement over LLaMA v1, it's still
| nowhere near even the best open models (currently, sans test
| contamination, WizardCoder-15B, a StarCoder fine tune is at
| top). It's really not a competition atm though, ChatGPT-4 wipes
| the floor for coding atm.
| riku_iki wrote:
| this all numbers can be missleading, and simply indicate that
| gpt have these tasks in training data, and another model
| doesn't.
| lhl wrote:
| While there's some contamination, it's not like the
| community isn't aware of it. For example, here's this
| discussion: https://huggingface.co/sahil2801/replit-code-
| instruct-glaive...
|
| This was the HumanEval contamination one dev measured: ```
| replit_glaive: 56.71% replit: 7.32% wizard: 4.88% ```
|
| From the WizardCoder paper
| https://arxiv.org/pdf/2306.08568.pdf you can see that it
| hits SOTA (for open models) in not just HumanEval and
| HumanEval+, but also MBPP and DS-1000 as well, so it's not
| a one off.
|
| For those interested in reading more about various
| considerations for coding models, I highly recommend
| reading the MSR phi-1 paper:
| https://arxiv.org/pdf/2306.11644.pdf
|
| Looking forward to if they ever publish code/model/dataset
| since it has extremely strong performance trained on a very
| small number of tokens very manageable 1.3B and 350M
| parameter models.
| riku_iki wrote:
| > in not just HumanEval and HumanEval+, but also MBPP and
| DS-1000 as well, so it's not a one off.
|
| and how do you know all these benchmarks not leaked? I
| think they all scrapped from web sites, the same as
| training data for LLM, so risk of contamination is
| extremely high.
|
| The best way to measure this is through synthetic
| datasets, which generate new tasks every time and model
| can't memorize them during training. One example is
| BigBench has multiple such tasks, but researchers
| usually(always) not regenerating those datasets.
| marcosdumay wrote:
| I imagine that if you take the time to specialize it, you
| suddenly have a model that is better than anything from the
| large players on all the cases that you care about.
|
| But, well, I am currently not hyped enough about it to actually
| try.
| rvz wrote:
| Possibly. Might need to be further optimized in size and 4-bit
| quantisation, perhaps and then you have a scaleable and fast
| self-hosted AI model.
|
| Lets just hope that there won't be any embarrassing
| vulnerabilities coming out of this when someone could prompt
| the model to reveal its own environment variables or API keys
| or the internal prompt that it is using.
|
| But it seems the $0 free AI models are eating OpenAI's lunch
| and Meta so far is winning the race to zero.
| aantix wrote:
| What's the most straight forward way of downloading LLaMA2, and
| training it with additional documents?
|
| I have a whole host of personal pdf's and documentation that I
| would love to be able to ask questions about.
| ubj wrote:
| This may be relevant:
|
| https://www.sematic.dev/blog/tuning-and-testing-llama-2-flan...
|
| It's the most straightforward explanation I've found so far.
| I'd love to hear if anyone's found something better though.
| mk_stjames wrote:
| Has anyone published a similar run of benchmarks with llama2 70B
| but at different quantization levels? I assume this benchmark is
| evaluated on the base model run at FP16. How much does it lose
| quantizing to INT8?
| golergka wrote:
| But unlike ChatGPT, it's still exclusively English, right?
| seydor wrote:
| this can apparently run on 48GB
| treprinum wrote:
| 2xA6000 NVLinked Ampere can run 70B 8-bit which is almost as
| good as fp16. I bought another A6000 just for that.
| lolinder wrote:
| When quantized to 4 bits, yes. You lose some quality by doing
| that, though, as compared to the full f16.
| evilduck wrote:
| From what I've gathered when reading up on this topic, if RAM
| is your constraint the common thought has been that higher
| parameter models quantized down to smaller sizes will
| outperform lower parameter models running at higher
| quantization, i.e. it may still be preferable to use the 70B
| Llama model quantized to 4-bits than something like an
| unquantized f16 Falcon 40B or the "coming soon" f16 33B
| Llama2.
| lolinder wrote:
| Yes, that is true! But you lose enough performance that
| comparisons to GPT-3.5 stop working.
| alecco wrote:
| Better evaluation paints a bit different picture:
|
| https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
|
| *FreeWilly2 is a Llama2 70B model finetuned on an Orca style
| Dataset
|
| EDIT: actually, impressive:
| FreeWilly2 GPT-3.5 GPT-4 ARC 71.1
| 85.2 96.3 HellaSwag 86.4 85.5 95.3
| MMLU 68.8 70.0 86.4 TruthfulQA
| 59.4 47.0 59.0
|
| So reasoning (ARC) is lagging behind, but the other evaluations
| are at GPT-3.5 level and closing the gap with 4.
|
| Source for GPT-3.5 and GPT-4.0 values (but mind it might not be
| the same # of shots)
|
| https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
| lhl wrote:
| It depends on the eval, but I think it's fair to say that it's
| close. Here is the AGI Eval results organized into a table w/
| averages (also I put in the new Hermes LLama2 13B model as
| well: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-
| W_jMwYp...
|
| It beats out ChatGPT in every category except SAT-Math. We
| definitely need harder benchmarks.
|
| So far, there's BIG-Bench Hard
| https://github.com/suzgunmirac/BIG-Bench-Hard and just
| published, Advanced Reasoning Benchmark https://arb.duckai.org/
| Tostino wrote:
| That seems more in-line with my experience. I have been using
| GPT-3.5 and GPT-4 for data cleaning pipelines, and have tried
| to swap out LLaMA2 70B in a few of the "easier" tasks, and it
| hasn't performed well enough yet for any of my tasks done by
| GPT-3.5.
| dr_kiszonka wrote:
| Hi! Could you please share a few words on what type of data
| you are cleaning using GPT? It is an intriguing idea and I
| would love to learn more to see if I could use a similar
| approach.
| Tostino wrote:
| Youtube transcripts. It only works with single-person
| channels at the moment, as I haven't worked on
| disambiguating multiple speakers. They are very messy if
| they are just an auto transcription from Google.
| Practically unusable in most cases.
|
| So first step in the pipeline is cleaning up the
| transcripts for incorrectly transcribed words or sentences.
| Using the context of the rest of the transcript, it is able
| to fix the vast majority of them. Then we add punctuation
| and format it with paragraphs. Then I have another check
| over the whole transcript for any remaining issues.
|
| After all of that, I have a relatively clean transcript
| that represents the original audio very closely. From
| there, I am doing things like: 1. creating question/answer
| pairs from the transcript 2. creating a document of
| additional context that fills in details about what the
| speaker is talking about but may not have explicitly said
| 3. creating a summary of the transcript identifying the
| main purpose 4. creating a knowledge graph from the
| transcript with nodes and edges 5. creating an annotated
| version of the transcript using that knowledge graph
|
| I plan on putting some of this data into a vector database,
| and some of it will be used for fine tuning LLaMA2 models
| on specific tasks (like knowledge graph creation,
| annotation using a knowledge graph, and writing using a
| knowledge graph to keep track of events)
| agravier wrote:
| How do you represent knowledge in your knowledge graph?
| Do you use an existing open source ontology?
| dr_kiszonka wrote:
| That's very informative - thanks so much for the
| explanation!
|
| My only experience with transcripts is in the context of
| transcribing short interviews. I used Whisper and it was
| pretty good. I mostly work with quantitative data,
| though.
|
| In terms of the disambiguation of speakers, I haven't
| done it, but I remember blind signal separation discussed
| in a signal processing seminar I attended. There is also
| this paper, in case you haven't seen it already:
| https://enk100.github.io/speaker_separation/
|
| Thanks again!
| Tostino wrote:
| I have not read that yet, thank you!
|
| Also, I haven't tried using Whisper for getting a
| transcription from the audio. I went the route of
| downloading the automatically generated transcripts from
| Youtube for a set of videos. An audio processing pipeline
| is definitely something I could add later though as an
| additional input channel for the overall pipeline.
| popinman322 wrote:
| Was this with or without fine-tuning?
| Tostino wrote:
| That is with fine-tuning:
| https://stability.ai/blog/freewilly-large-instruction-
| fine-t...
| sytelus wrote:
| LLaMA2 is far and away from GPT 3.5. Just look at HumanEval and
| other code generation metrics. All these GPT-4 based "chat
| evals" are extremely misleading and people should take it with
| a bag of salt.
| alecco wrote:
| That's what I say in my comment, OP's ranking is quite
| misleading.
|
| The ranking I linked and quoted in my comment's is much
| better. See the About tab. It has 4 evaluations and it
| doesn't use GPT-4 to evaluate.
|
| Also the top one is a tuned Llama 2. Also clarified in my
| original comment.
| luckystarr wrote:
| The value of GPT-4 also lies in its stored knowledge. A 70B model
| can't store that much.
| lolinder wrote:
| The advantage of LLaMA 2 is that a company can fine tune it on
| the knowledge that they actually care about and then run it on
| their own hardware without paying API fees or relying on an
| unstable dependency that's constantly being tweaked.
| mensetmanusman wrote:
| That will be possible with cloud AI in the future. On prem
| will always be less compute capable unless you have your own
| GPU cluster to rival the FAANGs, that is why meta is
| releasing this for free.
| lolinder wrote:
| This is the key part of what I said:
|
| > without paying API fees or relying on an unstable
| dependency that's constantly being tweaked
|
| I see no evidence that this part will be possible with
| OpenAI. Usage fees will always be a thing because that's
| how they make money, and based on what I've heard from
| people who have actually tried to build on their APIs, I
| would not trust them to keep the model stable. There are
| always new safety features they need to add, and those
| changes break things.
| meepmorp wrote:
| > without paying API fees or relying on an unstable
| dependency that's constantly being tweaked.
|
| and without handing a whole bunch of data to a 3rd party and
| hope they're securing it properly
| lhl wrote:
| Note, that just because you're hosting itself won't mean
| you're securing it properly... Here's a just published
| injection attack that only works on open source models
| (public model weights): https://twitter.com/random_walker/s
| tatus/1683833600196714497
| Jackson__ wrote:
| *When asked by GPT4 to compare the outputs.
|
| I'm a staunch believer that it would be foolish to rely on GPT4
| for quality comparisons, and it has been mind boggling to see so
| many people do it and treat it as perfect proof of anything.
|
| It would be slightly more understandable if there was a study to
| see how human and gpt4 preferences compare, but I'm unaware of
| any such thing.
| letmevoteplease wrote:
| There is one: "The agreement between GPT-4 and humans reaches
| 85%, which is even higher than the agreement among humans
| (81%). This means GPT-4's judgments closely align with the
| majority of humans. We also show that GPT-4's judgments may
| help humans make better judgments. During our data collection,
| when a human's choice deviated from GPT-4, we presented GPT-4's
| judgments to humans and ask if they are reasonable. Despite
| different views, humans deemed GPT-4's judgments reasonable in
| 75% of cases and are even willing to change their choices in
| 34% of cases."[1]
|
| [1] https://arxiv.org/abs/2306.05685
| bottlepalm wrote:
| It's funny how ChatGPT really does give you the most
| balanced, middle of the road answers. It feels like a
| distillation of all human knowledge and sentiments. I use it
| constantly to get advice on plans, architectures, thoughts,
| etc.. to get an idea of pretty much what the average person
| would think. It often points out things I've overlooked which
| I'll improve my design with and go back and forth with
| ChatGPT until we're both in agreement.
|
| I even read a classic book the other day and had a great
| discussion with ChatGPT about moral relativism, the different
| schools of thought and how it fit into philosophy as a whole.
| For students this technology is incredible, I wish I had it
| for all my classes.
|
| Even sometimes comments I'll make on here or Reddit I'll pass
| through ChatGPT first to see if I made any mistakes in my
| logic.
| beepbooptheory wrote:
| Would put slight caution around asking it anything more
| than "what should I read next about this?"
|
| For whatever reason*, it is particularly bad at discussing
| philosophy I find. When I was grading philosophy 101, I
| would have probably given it a passing grade against the
| overall curve, but that's about it. Philosophy is a
| discipline of careful, sometimes jargoney, and always very
| couched assertions that can be easily misunderstood and
| appropriated. This is probably its greatest weakness, and
| in many ways this weakness is the progenitor of philosophy
| itself in the Western world, with Plato at the start (i.e.
| with the figure of the sophist, the paradox of a false
| wisdom).
|
| - Maybe one reason: there is a huge amount of, lets say,
| "armchair philosophy" on the internet, compared to other
| disciplines. Many blogposts and tiny manifestos of people
| really excited by some out of context quote from Spinoza or
| whatever. And you start to really feel this part of the
| dataset when you ask about philosophy. Many strange takes
| and misunderstandings.
| dgroshev wrote:
| The reason you feel that way might be that you are
| familiar enough with philosophy.
|
| After all, LLMs and ChatGPT in particular are are
| indistinguishable from productised Gell-Mann Amnesia.
|
| Edit: rewrote to be more neutral, sorry
| bottlepalm wrote:
| I asked it where moral relativism fit in with philosophy
| and it came back with this Philosophy
| Ethics/Moral Philosophy Meta-Ethics: The
| study of moral thought, language, and properties
| Moral Realism: Belief that there are objective moral
| facts Moral Anti-Realism: Denial of
| the existence of objective moral facts
| Moral Relativism: The belief that moral judgments are
| true or false only relative to some particular standpoint
|
| I thought that was pretty good, what do you think?
| beepbooptheory wrote:
| Haha I think it's fine. I think its kinda cheeky
| answering you so literally, giving it an actual place to
| fit into :).
|
| I don't doubt it can do, like, Wikipedia type
| classification ok, but that's not like really getting to
| the substance of anything! And, either way, its not like
| there is one decided-upon hierarchy of concepts like this
| people consciously work within. This is a fine picture to
| some, but others might contest, perhaps, that Meta-Ethics
| is the "study of moral thought, language, and
| properties." What is "moral language" anyway? Why is it
| meta relative to Moral Philosophy writ-large? Or perhaps
| one might argue that we need to think of meta-ethics as a
| sibling rather than child. The whole discipline is a mess
| of different thoughts and possible rebuttals and grand
| intellectual overturnings that will not be captured here.
| Maybe just try pasting that back into the prompt and
| asking "what's wrong with this picture?".
|
| But like I said, its fine in that its fairly comparable
| to Wikipedia for utility, (with IMO a worse interface,
| but I get why people like it more).
| sundarurfriend wrote:
| > what the average person would think
|
| Keep in mind that due to the nature of the data and the
| RLHF training, it's more like a weighted average, something
| like 0.5*(average American view) +
| 0.4*(average WEIRD view) + 0.1*(average human view).
|
| (where WEIRD = Western, Educated, Industrialized, Rich, and
| Democratic, standard terminology in psychological
| research.)
|
| This may or may not matter depending on the questions
| you're asking, just something to keep in mind.
| TowerTall wrote:
| > to get an idea of pretty much what the average person
| would think
|
| There is no such thing as an average person.
|
| https://www.thestar.com/news/insight/when-u-s-air-force-
| disc...
| Folcon wrote:
| Funnily enough I think you both might be right here,
| there isn't such a thing as an average person, but
| ChatGPT may be the synthesis of the average opinion.
| wahnfrieden wrote:
| what is an average opinion? it is the sum of opinions
| which disagree with the result
| JieJie wrote:
| Maybe "balanced" rather than "average" is a better way of
| putting it?
| wahnfrieden wrote:
| What's the balance between, say, opinions that trans
| people should be exterminated vs not? What's the balance
| between Ukraine sovereignty vs Russian occupation? Etc
| ben_w wrote:
| My general experience (not that topic) is that for
| anything even slightly approaching the sides of the
| Silicon Valley Overton Window, ChatGPT creates micro-
| essay saying "On the one hand, $foo, on the other hand,
| $bar. It's important to remember that $topic is
| controversial, and that many people disagree."
| oceanplexian wrote:
| It gives you the average opinion of someone on the
| Internet, particularly places like Reddit, which is FAR
| from the "average opinion" of most people. This has been
| a problem for a while, people keep assuming the Internet
| represents some kind of moral or ethical consensus on so
| many issues when it's not even close.
| mycall wrote:
| On the other hand, since AIs are taught using user
| content from Reddit, once other cultures start using
| these AIs, they will begin to conform more towards those
| norms (or will actively go against it and The Great Sort
| will continue forward).
| kelseyfrog wrote:
| The other way to phrase this is high-dimensional spheres
| are "spikey".
| lolinder wrote:
| 80% agreement is high, but the margins between models at the
| top are so low that even that remaining 20% could be enough
| to alter the final rankings, depending on which direction it
| errs.
| bostonsre wrote:
| Is perfect agreement possible? And what is the definition
| of agreement? Humans don't agree about much.. are we saying
| agreement means it matches the truth after intensive
| investigation by humans?
| shanusmagnus wrote:
| Ugh, when I was doing my PhD work we were studying
| creativity in an experiment, and we needed an assessment
| for how creative different solutions were, and trying to
| get inter-rater reliability on this quite simple thing
| was just agonizing.
|
| I wound up abandoning the experiment because getting
| enough reliability would have required screwing down the
| standards so tightly that it would have ruined the
| underlying point of the thing.
| bagels wrote:
| Not just that, but are the 80/20 randomly distributed?
| Probably not. These comparisons might have more in common
| with the 20%.
| s17n wrote:
| But 80% agreement is the same as between humans
| lolinder wrote:
| True, but given that you're using one of the models to
| judge the others, it's likely that the cases of
| disagreement will tend to favor GPT-4. You would never
| use one of the competitors as a judge among humans.
| sdenton4 wrote:
| If only there were some way to produce some kind of
| "interval" of scores where you were confident that the
| actual score sat, and then had some way of comparing these
| intervals between the different models...
| sdenton4 wrote:
| That's helpful!
|
| I've done a lot of work in audio synthesis, which is
| notoriously difficult measure. The gold-standard is human
| ratings of audio quality, but it is tough to design good
| tests (easy to fatigue raters) and the iteration time waiting
| for results is quite long.
|
| Instead, there's now some projects which use neural networks
| trained on human ratings to predict audio quality, such as
| ViSQoL: https://github.com/google/visqol
|
| This opens up fast iteration - scores going up generally
| corresponds to higher quality - followed by human testing at
| major milestones (eg, releasing a paper/model). VISQOL has a
| harder time comparing 'unrelated' models, IMO - ends up being
| not so great for comparison of different techniques, but
| excellent for measuring incremental improvement or catching
| regressions.
|
| But, in the end, yes - you can use NN's to measure the
| quality of other NN's, so long as you're careful about it and
| make use of human raters from time to time as well.
|
| The problem of test data getting into the training data seems
| to be an especially pernicious issue with LLM's, which isn't
| really arising in the audio synthesis space.
| 2c2c2c wrote:
| just curious, are there any open models doing the opposite
| of audio synthesis? As in able to generate the stems for a
| song?
| edude03 wrote:
| https://github.com/facebookresearch/demucs
| blackkettle wrote:
| I've started doing this with ASR hypotheses from colloquial
| spontaneous speech. It tends to have similar issues. Lots
| of shady human ground truth especially where addresses,
| alphanumeric sequences, repairs and repetitions and other
| essentially non read speech are concerned. The very large
| Whisper models are consistent in their transcription style
| and highly reliable as long as you pick strongly
| represented languages. And ChatGPT can do a very good job
| at comparing the linguistic coherence of hypotheses from
| multiple recognizers. Together these models can annotate,
| analyze and ingest far more data more consistently than
| human annotators at this point (at least in the best
| covered languages). We haven't quite realized this as a
| community yet though, because the standard datasets we use
| for evaluation contain all these human inconsistencies.
| Wild times.
| koalacola wrote:
| If it's trained by humans is it safe to assume that we'll get
| it so something crazy like 99% agreeable?
| reaperducer wrote:
| _it has been mind boggling to see so many people do it and
| treat it as perfect proof of anything._
|
| The world has long been divided into two camps: People who
| think computers can make mistakes; and people who think
| computers never make mistakes, and blame the humans that
| program them.
|
| Well, now the computers are programming themselves. And clearly
| they're making mistakes.
| [deleted]
| RcouF1uZ4gsC wrote:
| Those MacBook Pros with 96 GB of unified GPU/CPU memory are
| looking pretty good right now.
|
| It would be awesome to have all this running on a laptop in a
| completely offline mode.
| TillE wrote:
| I think it'd be more fun to spend an extra $700 and get an M2
| Ultra Mac Studio with way more GPU cores and 128GB of RAM, and
| set up a private server.
|
| But if you really want a portable offline thing, sure.
| shanusmagnus wrote:
| I get more excited at the prospect of popping into some
| random cafe, SSHing from my iPad into some vast.ai server and
| setting loose this giant AI brain on whatever stuff. Feels
| badass.
| lhl wrote:
| This was just posted a few hours ago and when I tried it they
| were neck and neck (for me, LLama 2 won by a 1 question, but it
| was close): https://llmboxing.com/
|
| It looks like the eval is open sourced so you could easily build
| a version w/ your own questions for blind testing...
| charcircuit wrote:
| At least when I tried Llama 2's response was always the longer
| one so it was hard to remain unbiased.
| paxys wrote:
| There are more ways of evaluating LLMs than there are LLMs. All
| of the "X is better than Y" statements are pointless unless there
| is a very clear consensus.
| generalizations wrote:
| * ChatGPT 3.5. But it's also within spitting distance of GPT4,
| which is very exciting.
| make3 wrote:
| performance is logarithmic as a function of money invested in
| compute, so maybe it's close but it's also far away
| valine wrote:
| GPT4 is more difficult to measure I think. The value I get from
| GPT4 is in the details it gets right on very obscure, complex
| questions. I'm not sure benchmarks are capturing how far GPT4
| is ahead of other models. For simple stuff it's not that much
| better than 3.5.
| nmfisher wrote:
| I haven't had a chance to use the GPT-4 API yet - is it that much
| better than the GPT-4 available via ChatGPT? Or am I
| misunderstanding?
| freedomben wrote:
| ChatGPT uses the GPT-4 API, so it's the same. With the API
| directly though you can change the system prompt, which can
| enable better results if you know what you're doing.
| bazmattaz wrote:
| ChatGPT uses the GPT-4, but there are conspiracy theories
| circling that ChatGPT is neutered and thus not as good as GPT-4
| through the API. The theory being that OpenAI are thottling the
| free version of GPT-4 (ChatGPT)
| cosmojg wrote:
| That's, uh, not a conspiracy theory. The free version of
| ChatGPT uses an entirely different model on the backend.
| bestcoder69 wrote:
| Free ChatGPT runs 3.5. You have to upgrade to plus to use
| GPT-4. The APIs seem close to ChatGPT, but it's a little
| opaque what they're actually doing. If you inspect network
| requests the models are named something like "chat-
| render-3.5" instead of the API model names.
|
| I'd imagine OpenAI might run experiments on ChatGPT that they
| wouldn't on the API, to avoid breaking 3rd party applications
| unannounced.
| zo1 wrote:
| I use the GPT4 API and still think it got neutered since I
| first started using it.
| bkanber wrote:
| ChatGPT is a wrapper to the GPT Completion API with some sane
| defaults. With a new beta feature you can edit the system
| prompt via ChatGPT, but you still can't adjust the other
| parameters you can reach with the API.
| Fergusonb wrote:
| Anecdotal evidence here - I find that the API is less likely to
| ask questions about what you are doing and get straight to the
| answer.
|
| For example, if I were to ask how to do something with burp it
| will just answer instead of going into the "as an AI"
| monologue.
| fzzzy wrote:
| Do something with burp?
| NegativeK wrote:
| Burpsuite.
| [deleted]
| iandanforth wrote:
| Isn't the chat version of llama 2 trained on gpt-4 output, hence
| it's non-commercial license (as opposed to the base model) or am
| I just making things up?
| knodi123 wrote:
| Yeah, my experience has been that every one of these freely
| downloadable models can be measured as "percent of chatgpt
| quality". and getting up to 85% is shockingly good.
|
| *edit: oops, my brain inserted "by" in the middle of
| "outperformed chatgpt". I'll leave my wrong comment up as a
| testament to shame.
| fullshark wrote:
| The reality is probably some queries ChatGPT outperforms and
| vice versa. Regardless the premise that ChatGPT's secret sauce
| could be hidden forever is very dead.
| exo-pla-net wrote:
| I wish it were convention to specify the model, such as
| "gpt-3.5", rather than "ChatGPT", which is a service that hosts
| multiple models. Talking about ChatGPT creates pointless
| ambiguity.
|
| (But maybe it's a good filter: if someone is talking about
| "ChatGPT's" performance, they probably don't have anything useful
| to say.)
| kuprel wrote:
| Was LLaMA2 potentially trained on the raw text from this page?
| https://huggingface.co/datasets/tatsu-lab/alpaca_eval/viewer...
| cj wrote:
| It looks like ChatGPT length is 827 while LLaMA2 length is more
| than double at 1790.
|
| Disclaimer from the site:
|
| > Caution: GPT-4 may favor models with longer outputs and/or
| those that were fine-tuned on GPT-4 outputs.
|
| > While AlpacaEval provides a useful comparison of model
| capabilities in following instructions, it is not a comprehensive
| or gold-standard evaluation of model abilities. For one, as
| detailed in the AlpacaFarm paper, the auto annotator winrates are
| correlated with length.
| cs702 wrote:
| Also, Llama 2 is still a few percentage points below GPT-4.
|
| Which is _not_ close, because performance is logarithmic in
| training compute. Each additional percentage point of
| performance requires exponentially greater investment in
| compute during pretraining. Llama 2 was pretrained on 2
| trillion tokens -- a significant investment in compute, for
| sure, but still not enough to get close to GPT-4.
| weare138 wrote:
| Whats up with WizardLM-13B-V1.2? I don't know anything about it
| but the description says it's based on Llama-2 with only 13B
| parameters and it's holding it's own in the top 5 with a fraction
| of the model size.
| treprinum wrote:
| Isn't Wizard one of the uncensored versions like Luna etc.?
| europeanNyan wrote:
| There is a cool website where you can blind judge the outputs
| from LLaMa 2 vs ChatGPT-3.5: https://llmboxing.com/
|
| Surprisingly, LLaMa 2 won 5-0 for me.
| Tommstein wrote:
| Pretty cool. ChatGPT won the first one for me, then Llama 2 won
| the next five.
| speedgoose wrote:
| It was much closer to me. But llama 2 did surprisingly good.
| It's looks like it's a great alternative of chatGPT 3.5.
| andrei512 wrote:
| all the shorter answers were from GPT-3 - if you like long
| answers you pick llama 2...
| thorum wrote:
| In a response about the Turing test on this site, LLaMa 2 used
| the phrase "to exhibit intelligent behavior equivalent to, or
| indistinguishable from, that of a human" which appears to be
| copied verbatim from the first sentence of the Wikipedia
| article on the subject (as well as quite a few other pages in
| Google). Makes me wonder how many of the responses are just
| repeating and rephrasing memorized content written by humans,
| which will of course appear better, while ChatGPT makes more
| effort to avoid this (and might be able to generalize better to
| things it hasn't memorized?).
| user_7832 wrote:
| Thanks for the link!
|
| At least in my examples, the llama output was more
| verbose/comprehensive. Sometimes ChatGPT didn't expand enough,
| sometimes Llama missed the mark entirely (eg explaining the
| Eiffel's architecture.)
| SV_BubbleTime wrote:
| Interesting exercise, and llama won for me with 1 GPT answer...
| but it would be VERY easy to cherry pick these results and
| select a winner for most people.
| drew-y wrote:
| I got the opposite result. ChatGPT-3.5 won 5-0 for me. For me,
| LLaMa 2 gave longer answers that sometimes strayed away from
| the original question.
|
| They both gave great answers overall though.
| thewataccount wrote:
| The fact Guanaco 33B is at 65% with Vicuna 13B at 70.43%
| immediately makes these results non-sensical from my own
| experience with them. Heck from my experience Guanaco 33B is
| better then Vicuna 33B!
|
| Not to mention GPT4 at 95% and ChatGPT at 89% - I use
| chatgpt(3.5turbo)/gpt4 daily for work, and I rarely ever bother
| with 3.5turbo because of how unreliable it's answers are compared
| to gpt4.
|
| So whatever this is effectively measuring is useless for
| comparing these models, especially across work types.
| jacknews wrote:
| on their own metric?
| freedomben wrote:
| Llama2 beat ChatGPT 3.5 with a 92.66% win rate to 89.37%, but
| lost to GPT-4 which got 95.28%. Still pretty amazing though!
| cs702 wrote:
| Not really close, because performance is logarithmic in
| training compute.
|
| That is, each additional percentage point of performance
| requires exponentially greater investment in compute during
| pretraining.
|
| Llama 2 was pretrained on 2 trillion tokens -- a significant
| investment in compute, for sure, but still not enough to get
| close to GPT-4.
|
| And this is only one benchmark.
| krisknez wrote:
| I've played a little bit around llama2 and gpt3.5 is still better
| but llama2 is not far behind.
| api wrote:
| LLaMA2 seems to compete with ChatGPT 3.5, which is great. It's
| nowhere near as large as GPT-4 so I would not expect it to be
| competitive with that.
|
| GPT-4 level models that regular people can run with a reasonable
| hardware budget are going to require innovations in optimization
| and model efficiency beyond just quantizing weights. Rumor has it
| that GPT-4 is a "committee" of ~220G models, which would require
| ~128GiB VRAM at 4-bit quantization to run each model.
| kosolam wrote:
| Is there some free service that allows chatting with the 70b
| llama2?
| whinvik wrote:
| llama2.ai
| Oranguru wrote:
| Yes, check out: https://huggingface.co/chat/
|
| You can easily opt out of the data sharing.
___________________________________________________________________
(page generated 2023-07-27 23:02 UTC)