Post B9Z4trIQ4bMakpyQT2 by nukosu@pao.moe
 (DIR) More posts by nukosu@pao.moe
 (DIR) Post #B9Z4trIQ4bMakpyQT2 by nukosu@pao.moe
       0 likes, 0 repeats
       
       Ornith-1.5-35B-A3B on GeForce RTX 3090 (24GB)COMMON='--no-mmproj -fa on -ngl all --fit off --temp 0.6 --top-p 0.95 --top-k 20'MODEL='ornith-ai/Ornith-1.5-35B-A3B-GGUF'## Q8_0: n-cpu-moe 1540 t/s> llama-server -hf $MODEL:Q8_0 -c 20480 --n-cpu-moe 15 $COMMON## Q6_K: n-cpu-moe 684 t/s> llama-server -hf $MODEL:Q6_K -c 20480 --n-cpu-moe 6 $COMMON## Q5_K_M150 t/s> llama-server -hf $MODEL:Q5_K_M -c 20480 $COMMON## Q4_K_M157 t/s> llama-server -hf $MODEL:Q4_K_M -c 180000 $COMMON