[HN Gopher] Brave Leo now uses Mixtral 8x7B as default
___________________________________________________________________
Brave Leo now uses Mixtral 8x7B as default
Author : b_mc2
Score : 209 points
Date : 2024-01-27 05:04 UTC (17 hours ago)
(HTM) web link (brave.com)
(TXT) w3m dump (brave.com)
| rhdunn wrote:
| If you want to run Mixtral 8x7B locally you can use llama.cpp
| (including with any of the supporting libraries/interfaces such
| as text-generation-webui) with
| https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-S....
|
| The smallest quantized version (2bit) needs 20GB of RAM (which
| can be offloaded onto the VRAM of a decent 4090 GPU). The 4bit
| quantized versions are the largest models that can just about fit
| onto a 32GB system (29GB-31B). The 6bit (41GB) and 8bit (52GB)
| models need a 64GB system. You would need multiple GPUs with
| shared memory if you wanted to offload the higher precision
| models to VRAM.
|
| I've experimented with the 7B and 13B models, but haven't
| experimented with these models yet, nor other larger models.
| jodleif wrote:
| I prefer koboldcpp over llama.cpp. It's easy to spilt between
| gpu/cpu on models larger than VRAM
| DrSiemer wrote:
| Runs in Oobabooga textUi as well, if you add the llama.cpp
| extension. Easier interface imo, plus fun stuff like coqui
| and whisper integration.
| rhdunn wrote:
| That's interesting. It also looks like koboldcpp works better
| with long interactions, as it only processes changed tokens.
| I'm using llama.cpp with text-generation-webui and its OpenAI
| compatible API. I'll have to look to see if I can use
| koboldcpp with it.
| sp332 wrote:
| Llama.cpp has an interactive mode, but I don't think text-
| generation-webui uses it. https://github.com/ggerganov/llam
| a.cpp/blob/master/examples/...
| jodleif wrote:
| Indeed. Koboldcpp works fine with other UIs than the
| bundled one.
| magicalhippo wrote:
| I've got an aging 2080Ti and Ryzen 3800X with 96GB RAM, any
| point in trying to mess with the GPU or?
|
| Haven't really been able to justify upgrading to a 4090 or
| similar given I play so few new games these days.
| htsh wrote:
| Yes, offloading some layers to the GPU and VRAM should
| still help. And 11gb isn't bad.
|
| If you're on linux or wsl2, I would run oobabooga with
| --verbose. Load a GGUF, start with a small number of GPU
| layers and creep up, keeping an eye on VRAM usage.
|
| If you're on windows, you can try out LM Studio and fiddle
| with layers while you monitor VRAM usage, though windows
| may be doing some weird stuff sharing ram.
|
| Would be curious to see the diffs. Specifically if there's
| a complexity tax in offloading that makes the CPU-alone
| faster but in my experience with a 3060 and a mobile 3080,
| offloading what I can makes a big diff.
| macNchz wrote:
| > Specifically if there's a complexity tax in offloading
| that makes the CPU-alone faster
|
| Anecdotal, but I played with a bunch of models recently
| on a machine with a 16GB AMD GPU and 64GB of system
| memory/12 core CPU. I found offloading to significantly
| speed things up when dealing with large models, but there
| was seemingly an inflection point as I tested models that
| approached the limits of the system, where offloading did
| seem to significantly slow things down vs just running on
| the CPU.
| baq wrote:
| I had only cuda installed and it took 2 ollama shell
| commands in WSL2 from quite literally 0 local LLM
| experience to running mixtral fast enough on a 1070 and
| 12700k. Go for it.
| sp332 wrote:
| Llama.cpp has --n-gpu-layers that lets you set how much of
| the model to put on the GPU.
| attentive wrote:
| kobold bundles and runs llama.cpp. So it should be fairly the
| same with convenient defaults.
| DreamGen wrote:
| When talking about memory requirements one also needs to
| mention the sequence length. In case of Mixtral, which supports
| 32000 tokens, this can be a significant chunk of the memory
| used.
| viraptor wrote:
| And if you want better performance when talking about code, you
| can try the dolphin-mixtral fine tuning
| https://huggingface.co/TheBloke/dolphin-2.7-mixtral-8x7b-GGU...
| thriw63748 wrote:
| Why not normal RAM? Ryzen 5600 with 128GB DDR4 is perfectly
| fine to run mixtral 8bit, and costs less than $1000.
|
| GPUs are only needed if you can not wait 5 minutes for an
| answer, or for training.
| snowfield wrote:
| Or if you want multiple sessions at the same time. Or if you
| want to do anything else with your machine while it's
| running.
|
| But realistically, 5 minutes is too long. It should be
| conversational, and for that you need at least 5 tokens per
| second. Which your Ryzen just can't do.
| MPSimmons wrote:
| >It should be conversational, and for that you need at
| least 5 tokens per second.
|
| To be fair, a lot of people are using this for non-
| interactive work, like batching document analysis or
| offline processing of user generated content.
| Gracana wrote:
| In my experience, it takes some experimentation to figure
| out a good prompt. I don't think I would have gotten very
| far off I had to wait that long for each result.
| Diti wrote:
| This particular thread we are commenting on is about
| Dolphin Mixtral, which is mostly used for offline code
| completion (a la Microsoft GitHub Copilot). You don't
| want to have to wait 5 minutes at every keystroke to get
| code suggestions.
| rhdunn wrote:
| That was what I was referring to with the 32/64 GB systems.
| irusensei wrote:
| Why not both? Llama.cpp allows layering GGUF models between
| GPU and CPU memory.
| SkyMarshal wrote:
| What's the bandwidth between the Ryzen and that DDR4?
| brucethemoose2 wrote:
| > perfectly fine
|
| Only for very short context and responses.
|
| Beyond that, the performance is painful.
| juliangoldsmith wrote:
| Not sure why you're getting downvoted. It performs decent
| enough on my Ryzen 3600X with 64GB of RAM. It definitely
| wouldn't be usable for production or fine-tuning, but it's
| fine for experimenting.
| dragonwriter wrote:
| > GPUs are only needed if you can not wait 5 minutes for an
| answer
|
| Yeah, but that's generally true (or at least, "5 minutes for
| an answer is very suboptimal", even if "can't" isn't quite
| true) for interactive use cases, which are... a lot of LLM
| use cases.
| tarruda wrote:
| > You would need multiple GPUs with shared memory if you wanted
| to offload the higher precision models to VRAM.
|
| Or just a powerful apple silicon machine? I've tried dolphin
| mixtral 4bit on a 36gb ram MacBook m3, and inference is super
| fast.
| pennaMan wrote:
| I can run 4bit on a beat up 1070 ti. GP talks about higher
| precision models
| sp332 wrote:
| You wouldn't be able to fit the whole model into 8GB VRAM.
| It's faster than not using a GPU at all, but most of it
| would still be computed on the CPU.
| baq wrote:
| IME ollama ran mixtral on a 1070 fast enough.
| mkesper wrote:
| Or a Linux machine with a Ryzen using the internal GPU and
| the unified RAM (scroll down at llama.cpp and look for ROCm).
| stavros wrote:
| What? I can do this? _Runs to the PC_
|
| EDIT: I cannot, I need to install ROCm to compile with it,
| and then install something called hipBLAS, and who knows
| what else.
| assbuttbuttass wrote:
| OpenCL should also work on AMD cards, and is way easier
| to install
| stavros wrote:
| Does that let me use unified memory on the GPU, though?
| Or is it just so I can use my CPU memory?
|
| EDIT: Oh, no, I have an nVidia GPU, AMD CPU.
| brucethemoose2 wrote:
| It is dead slow on integrated graphics, unfortunately.
| jacooper wrote:
| Wait ROCm support Ryzen APUs and still doesn't support
| dedicatedly GPUs like the 6700XT?!
| jacooper wrote:
| Supports* dedicated*
| KeplerBoy wrote:
| Or a jetson orin agx (~2k$). Probably the cheapest way to
| get an Nvidia GPU with 64 GB of RAM.
| robterrell wrote:
| Have you tried this yourself? Curious to know how well
| this works for an LLM home lab.
| kkielhofner wrote:
| I've worked with Jetson going back to the TK1 and I
| highly recommend you do not do this.
|
| Nvidia has significant dominance in the AI space because
| of their work on software and the overall platform.
|
| With the Jetson line being the sole exception. Use it for
| what it's for - a targeted build for an embedded/specific
| application requiring small size and low power.
|
| The software is a mess. Support for Jetson (generally) is
| a far afterthought or not considered at all around
| projects at Nvidia and the broader ecosystem. When it is
| supported at all it lags behind significantly, using
| ancient distros (Jetpack), etc. To make matters worse the
| user base is so (relatively) tiny there are bugs and
| strange behavior everywhere.
|
| Just don't do it.
| qrios wrote:
| According to this article [1] it looks like there is no
| complex preparation needs to run the inference on a
| Jetson system. Should work with Mixtral too.
|
| [1] https://www.hackster.io/pjdecarlo/llama-2-llms-w-
| nvidia-jets...
| KeplerBoy wrote:
| I haven't tried it for LLMs yet, i use it for real time
| RF processing, but I actually have one of them on my desk
| and they are fun little devices.
|
| Maybe I will try to get a 32 GB+ LLM running one of those
| days.
| fsiefken wrote:
| I wonder what would be the cheapest way to run an LLM,
| with the latest Ryzen integrated graphics and 64G Ram or
| the Jetson AGX Orin 64. https://www.nvidia.com/en-
| us/autonomous-machines/embedded-sy...
| KeplerBoy wrote:
| The Ryzen is a lot cheaper, but most likely also a fair
| bit slower. You'd be looking at a 200$ CPU, 200$
| Motherboard + 200$ of ddr5 ram. Throw in a case, nvme
| drive and power supply and you're still below $1k and
| those numbers are quite generous estimates, you could do
| it a lot cheaper by going AM4 with DDR4 ram.
| asimpleusecase wrote:
| Did you do anything special to make that work? Is it useful?
| Or just a toy?
| EarthLaunch wrote:
| I've been using it for 'easy' queries like syntax/parameter
| questions, in place of ChatGPT 4. It's great for that. I am
| using a ~48GB version.
| windexh8er wrote:
| I have a 14" MBP with an M1 Max and 64GB. The M3 won't
| really make a difference, but the RAM, since unified, is
| huge. I can run most models on this machine with realtime
| performance compared to a Ryzen 7735HS and 64GB (DDR5). Now
| I'm not saying the Ryzen setup _should_ be good, but the M1
| architecture just makes it a much better option. I could
| add an eGPU to the Ryzen system and it could likely do
| better, but would also exceed the price point and
| portability.
| brucethemoose2 wrote:
| I've had less luck with Mixtral, but I run Yi 34B finetunes
| for general personal use, including quick queries for work.
|
| Its kinda like GPT 3.5, with no internet access and
| slightly less reliable responses, but unrestrained, _much_
| faster and with a huge (up to 75K on my Nvidia 3090) usable
| context.
|
| Mixtral is extremely fast though, at least at a batch size
| of 1.
| simonw wrote:
| Which Yi 34B finetunes are you using that have a 75,000
| token length?
| brucethemoose2 wrote:
| All of the Yi 200K finetunes should support it, but you
| have to be careful because some degrade the base model's
| quite excellent long context performance more than
| others. The very strong Bagel 34B DPO model, for
| instance, basically doesn't work at long context.
|
| Nous Capybara is a popular one. I personally use my own
| merge of many models, and you can look through the
| constituent models to see if any interest you:
| https://huggingface.co/brucethemoose/Yi-34B-200K-DARE-
| megame...
|
| You can't really use llama.cpp for super long context
| btw, its just too slow and vram inefficient at the
| moment.
| chpatrick wrote:
| `ollama run mixtral:8x7b-instruct-v0.1-q3_K_L` works fast on my
| 3090 locally
| beck5 wrote:
| Dumb question, but how can a 32 bit number be converted to 2
| bits and still be useful? It seems like magic.
| ttoinou wrote:
| All the 32 bits weren't necessarily used, and it's the whole
| network itself that has to be useful. It's a tradeoff. We
| started with very good precision to test the new method, now
| we can optimize some parts of it
| sp332 wrote:
| The extra precision is more useful for training. Once the
| network is optimized, it's a statistical model and only needs
| enough precision to make good guesses. In fact, one of the
| big papers on this also pointed out that you can drop about
| 40% of the weights completely. I think people generally skip
| that part because sparse matrix operations are slower, so it
| doesn't help here.
| viraptor wrote:
| For models with dropped weights, the keyword is
| "distilled". For example ssd-1b is a 50% size version of
| Stable Diffusion XL (https://huggingface.co/segmind/SSD-1B)
| sp332 wrote:
| That's crazy, I've never seen one that dropped whole
| layers from a pre-trained model. I guess that avoids the
| sparse matrix math.
| beefield wrote:
| I think of it with this kind of analogy: the original image
| is stored with 32 bit color scheme. You can reduce the color
| scheme to 16 bit accuracy and still figure out pretty well
| what the image is about. 2 bit is stretching this to a bit
| far, basically either pixel is white or it is black, but even
| if you lose lots of nuances in the image, in many images even
| that gives you some idea whats going on in the image.
| DougBTX wrote:
| That's an interesting question, I wonder if there is an
| analogy in quantisation to image dithering?
| DougBTX wrote:
| Nice graphs here:
| https://github.com/ggerganov/llama.cpp/pull/1684
|
| So for example, 2 bit version of the 30B is much worse than
| the original, but still better than the 13B model.
|
| Also, there are lots of extra details, eg, not all of the
| weights are 2 bit, and even the 2 bit weights are higher than
| that overall as groups of quantised weights share scale
| factors stored elsewhere.
| regularfry wrote:
| Mixtral and others are often distributed as 16-bit floats, so
| that chops the problem in half immediately, but then it turns
| out that LLMs only have about four bits per parameter of
| actual information stored. There's a lot of redundancy. The
| ideal quantisation scheme would only throw away useless data,
| but no quantisation scheme is perfect so they inevitably harm
| the model somehow.
|
| You've then got to remember that one thing neural networks
| are very, very good at is being noise tolerant. In some
| senses that's all they are - noise correction systems. The
| inaccuracies introduced by quantisation are "just" a sort of
| noise, so it's not surprising that they aren't fatal. It just
| raises the noise floor and gives the model more ways to be
| wrong.
|
| Finally the thing to know is that these quantisation schemes
| don't do a naive "chop each number down to two bits", not
| exactly. Simplifying a bit, for each parameter in this
| example they'd try to find a mapping from a two-bit index
| into a four element lookup table of higher-precision values
| such that the information destroyed by replacing the original
| parameter by the lookup value is minimised. That mapping is
| calculated across small blocks of parameters, rather than
| across the entire model, so it can preserve local detail. The
| lookup table gets stored per block, which throws the
| compression ratio off a little.
| brucethemoose2 wrote:
| Its not really 2 bits.
|
| Modern quantization schemes are almost like lossy compression
| algorithms, and llms in particular are very "sparse" and
| amenable to compression.
| hnfong wrote:
| This blog post might shed some light on the matter. If I'm
| understanding it correctly, it claims there are emergent
| features on the LLM weights that make it easier to "compress"
| the floats into smaller bits without losing much precision.
|
| https://timdettmers.com/2022/08/17/llm-int8-and-emergent-
| fea...
|
| Note that 2 bit quantization is generally regarded as too
| aggressive. Generally 4bits+ achieves a good tradeoff, see
| eg. https://arxiv.org/abs/2212.09720
| zaat wrote:
| Using TheBloke nous-hermes-2-mixtral-8x7b-dpo.Q5_K_M.gguf,
| llam.cpp with oobabooga, on Ryzen 9 7950x and RTX 2070 super,
| offloading 4 layers to the GPU.
|
| 27,631MiB system memory used.
|
| 7358/8192MiB usage on the GPU.
|
| I'm getting 6.32 tokens/second.
|
| You: You are Hunter S. Thompson, write a 450 words essay on the
| LHC project. The writing must be in Thompson's style
|
| AI: Well, buckle up, kiddos, because we're about to dive into
| the twisted world of particle physics and the Large Hadron
| Collider, or as I like to call it, the Doomsday Machine. This
| beast of a thing is buried deep beneath the Swiss countryside,
| a 17-mile-long ring of death and destruction that makes the
| atom bomb look like a goddamn firecracker.
|
| Now, let me tell you, when the Swiss government first proposed
| this monstrosity, I thought they'd gone completely batshit
| crazy. But then again, this is the land of Toblerone and Heidi,
| so maybe I shouldn't be too surprised. The LHC is essentially a
| giant, underground racetrack for subatomic particles, where
| they're accelerated to nearly the speed of light before being
| smashed into each other like a couple of drunken frat boys at a
| kegger.
|
| The goal of this madness? To recreate the conditions that
| existed just after the Big Bang, when the universe was nothing
| more than a seething cauldron of energy and chaos. And if that
| doesn't sound like a recipe for disaster, I don't know what
| does.
|
| But let's not forget the real reason we're all here: the Higgs
| boson, or as it's more commonly known, the God Particle. This
| little bugger is what gives other particles their mass, and
| without it, we'd all be floating around like a bunch of cosmic
| tumbleweeds. The LHC was designed to find this elusive
| particle, and in 2012, they finally did it. Sort of.
|
| You see, the Higgs boson they found was a bit of a
| disappointment. It was too light, too wimpy, and didn't behave
| quite the way they expected it to. So now, the physicists are
| back at the drawing board, trying to figure out what the hell
| went wrong. And in the meantime, the LHC continues to churn out
| more and more data, like some kind of unholy particle-smashing
| machine.
|
| But let's not forget the dark side of the LHC. There are those
| who believe that this thing could very well be the end of the
| world as we know it. They fear that the LHC could create a
| black hole, or even a miniature Big Bang, right here on Earth.
| And while the scientists assure us that these fears are
| unfounded, I can't help but wonder if they're just whistling
| past the graveyard.
|
| So, what have we learned from all this? Well, for one thing,
| we've discovered that the universe is a much stranger place
| than we ever imagined. We've also learned that even the most
| brilliant minds can be blinded by their own ambition, and that
| sometimes, the pursuit of knowledge can lead us down some very
| dark paths.
|
| In the end, the LHC is a testament to the insatiable curiosity
| of the human race, and our unquenchable thirst for knowledge.
| It's a monument to the power of science, and the potential
| consequences of playing God. And as we continue to push the
| boundaries of what's possible, we must always remember that
| with great power comes great responsibility. Or, as I like to
| say, "When the going gets weird, the weird turn pro."
| tacoman wrote:
| I am using the exact same model. Ryzen 5600G w/32GB and an
| Nvidia P40 w/24GB VRAM
|
| 20/33 layers offloaded to GPU, 4K context. Uses 25GB system
| RAM and all 24GB VRAM. 5-7 tokens per second.
| zaat wrote:
| Context is set to 32768, I didn't change it I guess that's
| the model's default.
|
| Thanks for making me feel better about investing in tht
| motherboard + CPU + RAM upgrade and deferring the GPU
| upgrade.
| bearjaws wrote:
| 2bit is pretty damn terrible, I don't recommend it for anything
| serious.
| ignoramous wrote:
| At that level of quantization / distillation, smaller models
| like _phi-2_ (q &a) and _wavecoder-6.7b_ (code-gen) might be
| preferable over QLoRAd ones:
| https://huggingface.co/microsoft/phi-2
|
| > _2bit is pretty damn terrible_
|
| Wait till you go hybrid [0] or even 1bit [1]
|
| [0] https://github.com/efeslab/Atom
|
| [1] https://github.com/IST-DASLab/qmoe
| EVa5I7bHFq9mnYK wrote:
| Faraday.dev has it in its selection of models now. Good for us
| clueless Windows folks. Runs decently fast with 16gb mobile
| 3080 gpu. Results seem better than any other free option.
| MuffinFlavored wrote:
| What differences would I measurably notice running the 2-bit
| version vs the 4-bit version vs the 6-bit vs the 8-bit?
| davikr wrote:
| It's nice using Brave because you have Chromium's better
| performance, without having to worry about Manifest V2 dying and
| taking adblocking down with it. I have uBlock Origin enabled, but
| it has barely caught anything that slipped past the browser
| filters.
| deely3 wrote:
| Is this submarine comment?
| thebruce87m wrote:
| What is the definition of a submarine comment? Google fails
| and ChatGPT says:
|
| > A "submarine comment" on social media refers to a comment
| that is made on an old post or thread, long after the
| conversation has died down. This term derives from the idea
| of a submarine which remains submerged and out of sight for
| long periods before suddenly surfacing. In the context of
| social media, it's when someone delves deep into someone
| else's posts or timeline, finds an old post, and leaves a
| comment, bringing the old post back to attention. This can
| sometimes surprise the original poster and other
| participants, as the conversation was thought to have been
| concluded.
|
| Which doesn't make sense in this context
| mcbuilder wrote:
| I think GP is trying to coin a term for stealth marketing
| Hacker News comments, except the analogy doesn't really
| make sense.
| freediver wrote:
| Seen on HN first!
| deely3 wrote:
| Definition: https://paulgraham.com/submarine.html
| deely3 wrote:
| Definition: https://paulgraham.com/submarine.html
| croes wrote:
| Brave"s support of Manifest V3 is totally dependent on Google
| and Chrome
|
| >Brave will support uBO and uMatrix so long as Google doesn't
| remove underlying V2 code paths (which seem to be needed for
| Chrome for enterprise support, so should stay in the Chromium
| open source)
|
| https://twitter.com/BrendanEich/status/1534893414579249152
| davikr wrote:
| Yeah, but the Brave adblocker is built-in, it's not an
| extension.
| croes wrote:
| There are more extensions than just ad blockers
| steve_rambo wrote:
| It doesn't really matter much in practice. The built-in
| adblocker (which does not rely on extension APIs) has
| excellent performance, about as effective as ublock origin,
| and supports the same filter lists.
| croes wrote:
| It does matter if you use other extensions that require
| Manifest V2
| guipsp wrote:
| The built-in adblocker still relies on a large amount of
| extension APIs.
| charcircuit wrote:
| MV3 doesn't prevent adblockers from existing.
| rpastuszak wrote:
| It makes them almost useless in practice.
| charcircuit wrote:
| That is a baseless statement. It doesn't make them useless
| as they can still block ads.
| HeatrayEnjoyer wrote:
| Because the filter list is capped, right? Is there a reason
| the Brave team cannot just remove or increase the cap?
| gkbrk wrote:
| Not just because of the filter list cap. It also reduces
| ad blockers to static filter lists instead of powerful
| dynamic filters.
|
| MV3 makes it impossible for ad-blockers to inspect
| requests with code and then allow/deny dynamically.
| charcircuit wrote:
| >It also reduces ad blockers to static filter lists
| instead of powerful dynamic filters.
|
| This is very outdated information and borderline
| misinformation by representing it as how it currently
| works. It allows for 30,000 dynamic rules and 5,000
| session rules (session rules only persist until the
| browser is closed).
|
| >MV3 makes it impossible for ad-blockers to inspect
| requests with code and then allow/deny dynamically.
|
| Giving this ability to extensions can slow down the
| browser for the user. These ads can still be blocked
| through other means.
| charcircuit wrote:
| >Because the filter list is capped, right?
|
| The limits are 300,000 static rules [1] + 30,000 dynamic
| rules [2] + 5,000 session rules [3]. For reference
| easylist is about 35k rules. The Chrome team has been
| constantly tweaking these limits themselves and Brave
| could set their own limits if they wish. The API is
| designed such that extensions can query to see how many
| rules they can use.
|
| [1] https://source.chromium.org/chromium/chromium/src/+/m
| ain:ext...
|
| [2] https://source.chromium.org/chromium/chromium/src/+/m
| ain:out...
|
| [3] https://source.chromium.org/chromium/chromium/src/+/m
| ain:out...
| srid wrote:
| > I have uBlock Origin enabled, but it has barely caught
| anything that slipped past the browser filters.
|
| I have been completely satisfied with Brave's builtin ad-
| blocker. Does uBlock Origin catch anything in particular that
| the builtin one does not?
| rozab wrote:
| I couldn't see Brave being able to counter, say, the recent
| YouTube anti-adblock push as quickly as the open source
| community did. I could see that kind of stunt becoming more
| common as Google tries to nail the coffin shut and deny their
| competitors this USP
| SparkyMcUnicorn wrote:
| Brave's ad blocking is compatible (and uses) the uBlock
| Origin lists, so you shouldn't really notice a difference
| between having uBO enabled or disabled in brave.
| Diti wrote:
| If by performance you mean browser performance, you have more
| performance with Firefox nowadays.
| https://news.ycombinator.com/item?id=36770883
| jacooper wrote:
| While still not being as secure as chromium and still not
| supporting many advanced features like WebGpu.
| FoodWThrow wrote:
| I use firefox because it has the most hassle-free hardware
| decoding in linux. However, everything basically feels better
| with Brave, even with the same amount of plug-ins.
| kristianp wrote:
| I found the snap update notifications too annoying on
| Ubuntu, so I tried the ppa. But it the video plugin would
| crash. So back to Chrome for me.
| firtoz wrote:
| What are good API providers that serve mixtral? I know only octo
| ai which seems decent but will be good to know alternatives too
| jug wrote:
| You also have Replicate:
| https://replicate.com/mistralai/mixtral-8x7b-instruct-v0.1
| htsh wrote:
| openrouter, fireworks, together.
|
| we use openrouter but have had some inconsistency with speed. i
| hear fireworks is faster, swapping it out soon.
| Tiberium wrote:
| OpenRouter is generally a good option (already mentioned), the
| best part is that you have a unified API for all LLMs, and the
| pricing is the same as with the providers themselves. Although
| for OpenAI/Anthropic models they were forced (by the respective
| companies) to enable filtering for inputs/outputs.
| firejake308 wrote:
| I personally like Anyscale Endpoints
| mark_l_watson wrote:
| I have used both Mistral's commercial APIs and also AnyScale's
| commercial APIs for mixtral-8-7b- both providers are easy to
| use.
|
| I also run a 3 bit quantization of mixtral-8-7b on my M2 Pro
| 32G memory system and it is fairly quick.
|
| It is great having multiple options.
| Philpax wrote:
| I've had good experiences with Together, and they have very
| competitive pricing.
| bearjaws wrote:
| Together.ai seems to be the best, incredibly fast.
| RupertEisenhart wrote:
| These guys are much faster than openrouter, and their llama2
| runs faster than 3.5-turbo. Amazing work.
| jeanloolz wrote:
| The creators of the model actually have their own platform
| where you can access this model and others via API:
| https://console.mistral.ai/
| petercooper wrote:
| Both already mentioned, but I am using Anyscale Endpoints with
| great success, very fast and will work on ten jobs at a go out
| of the box. Together.ai also seems to work fine in my initial
| tests, but haven't tried it at scale yet.
| fifteen1506 wrote:
| Just checking: PDF summarization is not yet implemented, right?
| 1equalsequals1 wrote:
| The Kagi browser extension can do that, if you're a subscriber
| charcircuit wrote:
| It's interesting that they made it so you can ask LLM queries
| right from the omnibar. I wonder if they eventually will come up
| with some heuristic to determine if thr query should be sent
| directly to an LLM or if the query should use the default search
| provider.
| finikytou wrote:
| quick question I have 24GB VRAM and I need to close everything to
| run MIXTRAL at 4 bit quant with bitsandbyte. there is no way to
| run it at 3,5 on windows?
| syntaxing wrote:
| Interesting, I must have missed the first Leo announcement. I
| really like how privacy conscious it is. They don't store any
| chat record which is what I want.
| Dwedit wrote:
| There is no way to confirm that claim, just like there is no
| way to confirm that a VPN service is "no log".
| Erratic6576 wrote:
| You gotta trust them by their word
| lolinder wrote:
| Yes, at some point if you're going to interface with other
| humans you will eventually just have to trust their word.
|
| For some people's threat models that isn't good enough, but
| for the vast majority of people--people who aren't being
| pursued by state intelligence agencies but who are squeamish
| about how much data a company like Google collects--a pinky
| promise from Brave or Mullvad is good enough.
| bcye wrote:
| I would like to think GDPR ensures this pinky promise is
| good enough
| wolverine876 wrote:
| > For some people's threat models that isn't good enough,
| but for the vast majority of people--people who aren't
| being pursued by state intelligence agencies but who are
| squeamish about how much data a company like Google
| collects--a pinky promise from Brave or Mullvad is good
| enough.
|
| Who are you to say it's good enough (and ridicule people
| who disagree)? We don't have too much evidence of it,
| because they have very few options and of course most
| people are not informed and lack the expertise to
| understand the issues (a good situation for regulation). At
| one point lots of people used lead paint and were fine with
| it; they would have told us.
|
| > Yes, at some point if you're going to interface with
| other humans you will eventually just have to trust their
| word.
|
| There's technology, such as the authorization tokens used
| by Brave, that reduces that risk. Of course, no risk can be
| complete eliminated but that doesn't mean we shouldn't
| reduce it.
| lolinder wrote:
| > say it's good enough (and ridicule people who
| disagree)?
|
| I'm not ridiculing anyone, I explicitly say that for some
| people's threat models it isn't good enough.
| wolverine876 wrote:
| You said that people who agreed with you are,
|
| > people who aren't being pursued by state intelligence
| agencies
|
| That implies that the only valid reason to disagree is if
| you are pursued by state intelligence agencies. Obviously
| it's ridiculous to think that you are.
| salad-tycoon wrote:
| Well, these days a lot of people are actually pursued.
| You know like parents, social activists, various
| religious movements. It's not just only the Dr. Evils of
| the world anymore.
| Tanath wrote:
| It's foolish to trust Brave. https://www.spacebar.news/p/stop-
| using-brave-browser
| kristianpaul wrote:
| I run Mixtral locally using ollama
| andai wrote:
| Asked Mistral 8x7B for an essay on ham. It started telling me
| about Hamlet.
| Erratic6576 wrote:
| It must start from the beginning. Pig > piglet. Ham > Hamlet
| m3kw9 wrote:
| If you have used gpt4 and then use mistral, it's like looking at
| a Retina display and then have to go back to a low res screen.
| You are always thinking "but GPT4 could do this though"
| mpalmer wrote:
| Have you used mixtral?
| wolverine876 wrote:
| Kudos to Brave (for this and other privacy features):
|
| _Unlinkable subscription: If you sign up for Leo Premium, you're
| issued unlinkable tokens that validate your subscription when
| using Leo. This means that Brave can never connect your purchase
| details with your usage of the product, an extra step that
| ensures your activity is private to you and only you. The email
| you used to create your account is unlinkable to your day-to-day
| use of Leo, making this a uniquely private credentialing
| experience._
| quinncom wrote:
| This is very cool, and something I'd like to integrate in my
| own apps. Does anybody know how this works exactly, not using
| foreign keys?
| luke-stanley wrote:
| I could guess, an "anonymous payment credential service"
| could do something like this: 1. User completes payment for
| the paid for service, 2. To track the payment entitlement, a
| random, unique ID is generated by the service for the user,
| that is not related to any of their data. 3. This ID is saved
| in a database as a valid payment key. 4. The database records
| IDs in shuffled batches, or with semi-random fuzzy / low
| resolution timestamps to prevent correlation between payment
| time and ID generation. 5. Each ID has an entitlement limit
| or usage stopping point, ensuring it's only valid for the
| subscribed period.
|
| Another way might be Zero-Knowledge Proofs (ZKPs), but that
| might be more complex. They might even use their BAT crypto
| stuff for this somehow, I suppose. Whatever solution, would
| need a fundamental solution for how to avoid correlation, I
| think.
___________________________________________________________________
(page generated 2024-01-27 23:01 UTC)