Post B6X0t9hF78gdQE9moC by AGARTHA_NOBLE@shortstacksran.ch
(DIR) More posts by AGARTHA_NOBLE@shortstacksran.ch
(DIR) Post #B6Wzk4w5LQzs64CubA by sapphire@shortstacksran.ch
1 likes, 1 repeats
@AGARTHA_NOBLE whats the state of AI on Mac? Got a 64GB M1 Max that I’ll likely get to keep and it’s useless for anything else (busted display)
(DIR) Post #B6WzlCj3rqAiXweV16 by AGARTHA_NOBLE@shortstacksran.ch
0 likes, 0 repeats
@sapphire Not so sure about M1, but with that VRAM you can probably load high-precision (8 bit quant) Qwen 3.6 MoE 35B and expect pretty snappy performance, even using Ollama. Lemme look into it.
(DIR) Post #B6WzlCvp6ONdBWcguO by AGARTHA_NOBLE@shortstacksran.ch
0 likes, 0 repeats
@sapphire Yea, looks like either 6 or 8 bit quant works fine, but caveat emptor on your tokens per second. I suggest Qwen 3.6 MoE for speed, 27B Dense if you want accuracy in coding or what have you.
(DIR) Post #B6WzlD6SSqt3iVbBU8 by AGARTHA_NOBLE@shortstacksran.ch
1 likes, 0 repeats
@sapphire If you only want to line up one model and make that shit run fast, tinker with a vLLM config, but honestly Ollama will probably suit you super well, and you can connect OpenWeb UI to it remotely over docker pod to chat with the little virtual idiot, if that's all you want.
(DIR) Post #B6WzmtbvuJrJqF7mz2 by mischievoustomato@tsundere.love
0 likes, 0 repeats
@AGARTHA_NOBLE @sapphire what's the delta between ollama and vllm?
(DIR) Post #B6X0ZFrsSVyKlrdr3w by AGARTHA_NOBLE@shortstacksran.ch
1 likes, 0 repeats
@mischievoustomato @sapphire Trade-off math: do you want to run a lot of models really easily and to be able to swap between them seamlessly(ollama), or do you want to tune one model to serve a lot of people really well(vllm)? There are other options that I haven't yet touched like lmstudio, and llamacpp, but those fall into the second bucket, not the first- ollama's a wrapper around llamacpp, meaning you don't have to care about implementation so long as you've got the hardware to support it.
(DIR) Post #B6X0hOqww8IAxE5kEC by mischievoustomato@tsundere.love
0 likes, 0 repeats
@AGARTHA_NOBLE @sapphire yeah. currently i use koboldcpp, lets me use text models, tts and image gen ones.
(DIR) Post #B6X0t9TPwXcyjLgkG8 by sapphire@shortstacksran.ch
0 likes, 0 repeats
@AGARTHA_NOBLE keep in mind that’s total memory, not just VRAM. I think the M1 can divert up to half
(DIR) Post #B6X0t9hF78gdQE9moC by AGARTHA_NOBLE@shortstacksran.ch
0 likes, 0 repeats
@sapphire Looks like it's up to about 48gb by default, which still puts you in the right range for a quantized Qwen, plus respectable KV cache. I'm using a bunch of tricks to get my cache window to about 110K tokens; you'll likely be in the same vicinity, and those models support up to 256k tokens natively.
(DIR) Post #B6X0t9u0LgtY3o7yhU by AGARTHA_NOBLE@shortstacksran.ch
0 likes, 0 repeats
@sapphire I THINK that's just soft-cap, too; there's not really anything preventing you from accessing all but 4GB with the GPU on those things except for sanity.
(DIR) Post #B6X0tA4HjT7OZgwBiy by sapphire@shortstacksran.ch
0 likes, 0 repeats
@AGARTHA_NOBLE macOS is really light especially when you trim it down
(DIR) Post #B6X0tAEZ7FLF5ZkOkS by mischievoustomato@tsundere.love
0 likes, 0 repeats
@sapphire @AGARTHA_NOBLE thank you, based apple
(DIR) Post #B6X19QiBcTyIpLMHrc by AGARTHA_NOBLE@shortstacksran.ch
1 likes, 0 repeats
@sapphire All the better- run the highest quantization you can make work. Definitely worth trimming KV-cache for higher quants, too; smaller models are super susceptible to quantization.
(DIR) Post #B6X1Amo6xEcde2mClE by AGARTHA_NOBLE@shortstacksran.ch
1 likes, 0 repeats
@mischievoustomato @sapphire Kobold's really solid for the smaller stuff, and the framework's good, but for creative work I'm starting to look towards Stabilitymatrix: https://lykos.ai/
(DIR) Post #B6X1CdlxW4EQCfebs8 by mischievoustomato@tsundere.love
0 likes, 0 repeats
@AGARTHA_NOBLE @sapphire I'm kind of limited to just vulkan stuff due to being on an igpu, but bookmarkd