[HN Gopher] Ask HN: How to get started with local language models?
___________________________________________________________________
Ask HN: How to get started with local language models?
I remember using Talk to a Transformer in 2019 and making little
Markov chains for silly text generation. I've lurked in the g /lmg/
threads, installed the Umbrel LlamaGPT on a Raspberry Pi, and ran
webGPU WebGPT (GPT-2) locally. But I don't know how anything works
or how to do anything besides following installation instructions
on GitHub. HuggingFace still confuses me; there's so much stuff out
there to read through, and I have been lost since the release of
the llama models. I heard about Mistral. I tried using the Mozilla
Llamafiles since it was meant to be easy for the
Mixtral-8x7B-Instruct, but apparently, windows won't run an EXE
bigger than 4GB, so I have to do something about the weights, and I
freeze here. I know where to look and how to learn about any other
technology, but I'm completely lost for how to learn about local
models when everything is moving so fast. I'm missing something
fundamental. How can I understand these technologies?
Author : sandwichukulele
Score : 60 points
Date : 2024-03-17 04:04 UTC (18 hours ago)
| ShamelessC wrote:
| Sounds like you're on the right track. Persistence is key.
| boppo1 wrote:
| It might help you to dive into SD and comfyui. Loras, finetunes,
| embeddings, etc. Are easier to understand from a practical
| standpoint when you can visually compare their output. That will
| give you intuition for how similar layers work for LLLMs.
| vendiddy wrote:
| I'm an outsider to the ML space and recently started watching
| videos from Andrej Karpathy.
|
| His videos really helped me build an intuition of how LLMs work.
| What I like is that he builds very simple versions of things that
| are easier to wrap you rhead around.
|
| https://www.youtube.com/@AndrejKarpathy/videos
| PeterStuer wrote:
| You might want to start playing with with Ollama or LmStudio to
| get you started. The model's weights aren't normally inside your
| .exe . Typically you can address 128TB of virtual memory on
| Windows.
|
| https://ollama.com/
|
| https://lmstudio.ai/
| leopoldj wrote:
| I wrote a series of blog post precisely for someone in your
| situation. I hope you find them useful.
|
| Easiest way to run a local LLM these days is Ollama. You don't
| need PyTorch or even Python installed.
|
| https://mobiarch.wordpress.com/2024/02/19/run-rag-locally-us...
|
| Hugging Face can be confusing but in the end a very well designed
| framework.
|
| https://mobiarch.wordpress.com/2024/03/02/start-using-mistra...
| instagib wrote:
| You can use webui https://github.com/oobabooga/text-generation-
| webui
|
| Once you get a version up and running I make a copy before I
| update it as several times updates have broken my working version
| and caused headaches.
|
| a decent explanation of parameters outside of reading archive
| papers: https://github.com/oobabooga/text-generation-
| webui/wiki/03-%...
|
| a news ai website: https://www.emergentmind.com/
|
| Reddit locallamma and how to prompt an llm:
| https://old.reddit.com/r/LocalLLaMA/comments/1atyxqz/better_...
|
| Since you mention silly text generation there is also sillytavern
| which runs on top of another llm software such as webui.
| https://docs.sillytavern.app/
| xue160709 wrote:
| https://github.com/xue160709/Local-LLM-User-Guideline There is a
| cookbook
| exe34 wrote:
| Have you tried llama.cpp?
| fzzzy wrote:
| How much ram do you have? How much gpu ram do you have? What gpu
| do you have? What kind of performance are you hoping to get in
| tokens per second?
| Const-me wrote:
| If you just want to run Mistral on Windows, you could try my
| port: https://github.com/Const-
| me/Cgml/tree/master/Mistral/Mistral...
|
| The setup is relatively easy: install .NET runtime, download 4.5
| GB model file from BitTorrent, unpack a small ZIP file and run
| the EXE.
| mitjam wrote:
| If you are on Windows, Ollama and OpenWebUI now are as easy to
| get started as they have been on macOS and Linux, already. I have
| created a short video about Ollama on Windows and how to install
| and use it with OpenWebUI:
| https://m.youtube.com/watch?v=z8xi44O3hvY
___________________________________________________________________
(page generated 2024-03-17 23:02 UTC)