[HN Gopher] Ask HN: Is anyone doing anything cool with tiny lang...
___________________________________________________________________
Ask HN: Is anyone doing anything cool with tiny language models?
I mean anything in the 0.5B-3B range that's available on Ollama
(for example). Have you built any cool tooling that uses these
models as part of your work flow?
Author : prettyblocks
Score : 153 points
Date : 2025-01-21 19:39 UTC (3 hours ago)
| psyklic wrote:
| JetBrains' local single-line autocomplete model is 0.1B (w/
| 1536-token context, ~170 lines of code):
| https://blog.jetbrains.com/blog/2024/04/04/full-line-code-co...
|
| For context, GPT-2-small is 0.124B params (w/ 1024-token
| context).
| WithinReason wrote:
| That size is on the edge of something you can train at home
| vineyardmike wrote:
| If you have modern hardware, you can absolutely train that at
| home. Or _very_ affordable on a cloud service.
|
| I've seen a number of "DIY GPT-2" tutorials that target this
| sweet spot. You won't get amazing results unless you want to
| leave a personal computer running for a number of hours/days
| and you have solid data to train on locally, but fine-tuning
| should be in the realm of normal hobbyists patience.
| nottorp wrote:
| Hmm is there anything reasonably ready made* for this spot?
| Training and querying a llm locally on an existing
| codebase?
|
| * I don't mind compiling it myself but i'd rather not write
| it.
| smaddox wrote:
| You can train that size of a model on ~1 billion tokens in ~3
| minutes on a rented 8xH100 80GB node (~$9/hr on Lambda Labs,
| RunPod io, etc.) using the NanoGPT speed run repo:
| https://github.com/KellerJordan/modded-nanogpt
|
| For that short of a run, you'll spend more time waiting for the
| node to come up, downloading the dataset, and compiling the
| model, though.
| pseudosavant wrote:
| I wonder how big that model is in RAM/disk. I use LLMs for
| FFMPEG all the time, and I was thinking about training a model
| on just the FFMPEG CLI arguments. If it was small enough, it
| could be a package for FFMPEG. e.g. `ffmpeg llm "Convert this
| MP4 into the latest royalty-free codecs in an MKV."`
| jedbrooke wrote:
| the jetbrains models are about 70MB zipped on disk (one model
| per language)
| mettamage wrote:
| I simply use it to de-anonymize code that I typed in via Claude
|
| Maybe should write a plugin for it (open source):
|
| 1. Put in all your work related questions in the plugin, an LLM
| will make it as an abstract question for you to preview and send
| it
|
| 2. And then get the answer with all the data back
|
| E.g. df["cookie_company_name"] becomes df["a"] and back
| politelemon wrote:
| Could you recommend a tiny language model I could try out
| locally?
| mettamage wrote:
| Llama 3.2 has about 3.2b parameters. I have to admit, I use
| bigger ones like phi-4 (14.7b) and Llama 3.3 (70.6b) but I
| think Llama 3.2 could do de-anonimization and anonimization
| of code
| OxfordOutlander wrote:
| +1 this idea. I do the same. Just do it locally using
| ollama, also using 3.2 3b
| RicoElectrico wrote:
| Llama 3.2 punches way above its weight. For general
| "language manipulation" tasks it's good enough - and it can
| be used on a CPU with acceptable speed.
| seunosewa wrote:
| How many tokens/s?
| sitkack wrote:
| So you are using a local small model to remove identifying
| information and make the question generic, which is then sent
| to a larger model? Is that understanding correct?
|
| I think this would have some additional benefits of not
| confusing the larger model with facts it doesn't need to know
| about. My erasing information, you can allow its attention
| heads to focus on the pieces that matter.
|
| Requires further study.
| sauwan wrote:
| Are you using the model to create a key-value pair to
| find/replace and then reverse to reanonymize, or are you using
| its outputs directly? If the latter, is it fast enough and
| reliable enough?
| RhysU wrote:
| "Comedy Writing With Small Generative Models" by Jamie Brew
| (Strange Loop 2023)
|
| https://m.youtube.com/watch?v=M2o4f_2L0No
|
| Spend the 45 minutes watching this talk. It is a delight. If you
| are unsure, wait until the speaker picks up the guitar.
| 100k wrote:
| Seconded! This was my favorite talk at Strange Loop (including
| my own).
| iamnotagenius wrote:
| No, but I use llama 3.2 1b and qwen2.5 1.5 as bash oneliner
| generator, always runnimg in console.
| andai wrote:
| Could you elaborate?
| XMasterrrr wrote:
| I think I know what he means. I use AI Chat. I load
| Qwen2.5-1.5B-Instruct with llama.cpp server, fully offloaded
| to the CPU, and then I config AI Chat to connect to the
| llama.cpp endpoint.
|
| Checkout the demo they have below
|
| https://github.com/sigoden/aichat#shell-assistant
| XMasterrrr wrote:
| What's your workflow like? I use AI Chat. I load
| Qwen2.5-1.5B-Instruct with llama.cpp server, fully offloaded to
| the CPU, and then I config AI Chat to connect to the llama.cpp
| endpoint.
| Havoc wrote:
| Pretty sure they are mostly used as fine tuning targets, rather
| than as-is.
| dcl wrote:
| But for what purposes?
| arionhardison wrote:
| I am, in a way by using EHR/EMR data for fine tuning so agents
| can query each other for medical records in a HIPPA compliant
| manner.
| azhenley wrote:
| Microsoft published a paper on their FLAME model (60M parameters)
| for Excel formula repair/completion which outperformed much
| larger models (>100B parameters).
|
| https://arxiv.org/abs/2301.13779
| barrenko wrote:
| This is really cool. Is this already in Excel?
| andai wrote:
| This is wild. They claim it was trained exclusively on Excel
| formulas, but then they mention retrieval? Is it understanding
| the connection between English and formulas? Or am I
| misunderstanding retrieval in this context?
|
| Edit: No, the retrieval is Formula-Formula, the model (nor I
| believe tokenizer) does not handle English.
| 3abiton wrote:
| But I feel we're going back full circle. These small models are
| not generalist, thus not really LLMs at least in terms of
| objective. Recently there has been a rise of "specialized"
| models that provide lots of values, but that's not why we were
| sold on LLMs.
| colechristensen wrote:
| But that's the thing, I don't need my ML model to be able to
| write me a sonnet about the history of beets, especially if I
| want to run it at home for specific tasks like as a
| programming assistant.
|
| I'm fine with and prefer specialist models in most cases.
| behohippy wrote:
| I have a mini PC with an n100 CPU connected to a small 7" monitor
| sitting on my desk, under the regular PC. I have llama 3b (q4)
| generating endless stories in different genres and styles. It's
| fun to glance over at it and read whatever it's in the middle of
| making. I gave llama.cpp one CPU core and it generates slow
| enough to just read at a normal pace, and the CPU fans don't go
| nuts. Totally not productive or really useful but I like it.
| Dansvidania wrote:
| this sounds pretty cool, do you have any video/media of it?
| bithavoc wrote:
| this is so cool, any chance you post a video?
| Uehreka wrote:
| Do you find that it actually generates varied and diverse
| stories? Or does it just fall into the same 3 grooves?
|
| Last week I tried to get an LLM (one of the recent Llama models
| running through Groq, it was 70B I believe) to produce randomly
| generated prompts in a variety of styles and it kept producing
| cyberpunk scifi stuff. When I told it to stop doing cyberpunk
| scifi stuff it went completely to wild west.
| o11c wrote:
| You should not ever expect an LLM to actually do what you
| want without handholding, and randomness in particular is one
| of the places it fails badly. This is probably fundamental.
|
| That said, this is also not helped by the fact that all of
| the default interfaces lack many essential features, so you
| have to build the interface yourself. Neither "clear the
| context on every attempt" nor "reuse the context repeatedly"
| will give good results, but having one context producing just
| one-line summaries, then fresh contexts expanding each one
| will do slightly less badly.
|
| (If you actually want the LLM to do something useful, there
| are many more things that need to be added beyond this)
| dotancohen wrote:
| Sounds to me like you might want to reduce the Top P - that
| will prevent the really unlikely next tokens from ever
| being selected, while still providing nice randomness in
| the remaining next tokens so you continue to get diverse
| stories.
| janalsncm wrote:
| Generate a list of 5000 possible topics you'd like it to talk
| about. Randomly pick one and inject that into your prompt.
| keeganpoppen wrote:
| oh wow that is actually such a brilliant little use case--
| really cuts to the core of the real "magic" of ai: that it can
| just keep running continuously. it never gets tired, and never
| gets tired of thinking.
| ipython wrote:
| That's neat. I just tried something similar:
| FORTUNE=$(fortune) && echo $FORTUNE && echo "Convert the
| following output of the Unix `fortune` command into a small
| screenplay in the style of Shakespeare: \n\n $FORTUNE" | ollama
| run phi4
| eb0la wrote:
| We're using small language models to detect prompt injection. Not
| too cool, but at least we can publish some AI-related stuff on
| the internet without a huge bill.
| sitkack wrote:
| What kind of prompt injection attacks do you filter out? Have
| you tested with a prompt tuning framework?
| A4ET8a8uTh0_v2 wrote:
| Kinda? All local so very much personal, non-business use. I made
| Ollama talk in a specific persona styles with the idea of
| speaking like Spider Jerusalem, when I feel like retaining some
| level of privacy by avoiding phrases I would normally use.
| Uncensored llama just rewrites my post with a specific persona's
| 'voice'. Works amusingly well for that purpose.
| deet wrote:
| We (avy.ai) are using models in that range to analyze computer
| activity on-device, in a privacy sensitive way, to help knowledge
| workers as they go about their day.
|
| The local models do things ranging from cleaning up OCR, to
| summarizing meetings, to estimating the user's current goals and
| activity, to predicting search terms, to predicting queries and
| actions that, if run, would help the user accomplish their
| current task.
|
| The capabilities of these tiny models have really surged
| recently. Even small vision models are becoming useful,
| especially if fine tuned.
| ignoramous wrote:
| We're prototyping a text firewall (for Android) with Gemma2 2B
| (which limits us to English), though DeepSeek's R1 variants now
| look pretty promising [0]: Depending on the content, we rewrite
| the text or quarantine it from your view. Of course this is easy
| (for English) in the sense that the core logic is all LLMs [1],
| but the integration points (on Android) are not so straight
| forward for anything other than SMS. [2]
|
| A more difficult problem we forsee is to turn it into a real-time
| (online) firewall (for calls, for example).
|
| [1]
| https://chat.deepseek.com/a/chat/s/d5aeeda1-fefe-4fc6-8c90-2...
|
| [1] MediaPipe in particular makes it simple to prototype around
| Gemma2 on Android:
| https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inf...
|
| [2] Intend to open source it once we get it working for anything
| other than SMSes
| mritchie712 wrote:
| I used local LLMs via Ollama for generating H1's / marketing
| copy.
|
| 1. Create several different personas
|
| 2. Generate a ton of variation using a high temperature
|
| 3. Compare the variagtions head-to-head using the LLM to get a
| win / loss ratio
|
| The best ones can be quite good.
|
| 0 - https://www.definite.app/blog/overkillm
| flippyhead wrote:
| I have a tiny device that listens to conversations between two
| people or more and constantly tries to declare a "winner"
| pseudosavant wrote:
| I'd love to hear more about the hardware behind this project.
| I've had concepts for tech requiring a mic on me at all times
| for various reasons. Always tricky to have enough power in a
| reasonable DIY form factor.
| oa335 wrote:
| This made me actually laugh out loud. Can you share more
| details on hardware and models used?
| econ wrote:
| This is a product I want
| amelius wrote:
| You can use the model to generate winning speeches also.
| jjcm wrote:
| Are you raising a funding round? I'm bought in. This is
| hilarious.
| simonjgreen wrote:
| Micro Wake Word is a library and set of on device models for ESPs
| to wake on a spoken wake word.
| https://github.com/kahrendt/microWakeWord
|
| Recently deployed in Home Assistants fully local capable Alexa
| replacement. https://www.home-
| assistant.io/voice_control/about_wake_word/
| kristopolous wrote:
| I'm working on using them for agentic voice commands of a limited
| scope.
|
| My needs are narrow and limited but I want a bit of flexibility.
___________________________________________________________________
(page generated 2025-01-21 23:00 UTC)