[HN Gopher] Fine-tune Google's Gemma 3
___________________________________________________________________
Fine-tune Google's Gemma 3
Author : tomdekan
Score : 107 points
Date : 2025-03-19 16:34 UTC (6 hours ago)
(HTM) web link (unsloth.ai)
(TXT) w3m dump (unsloth.ai)
| rockwotj wrote:
| is anyone outside of the research labs fine tuning models for
| production use cases? I have been seeing more people just using
| foundational models off the shelf especially in light of a new
| advancement that seems to come every few months
| deepsquirrelnet wrote:
| I'm trying right now. The combination of small models, qlora
| and grpo has made it accessible to experimenters. I'm not using
| unsloth yet, but I will probably start checking it out pretty
| soon so that I can train larger models or increase the number
| of generations for grpo.
| simonw wrote:
| I've had trouble getting a great answer to this question - I
| ask it in various places every month or so, most recently here:
| https://nitter.net/simonw/status/1895301139819860202
|
| On paper fine tuning smaller models can greatly reduce the cost
| for a specific task, but I've not heard many real-world success
| stories around that.
|
| I think vision LLMs are one of the most interesting
| applications here - things like fine-tuning for better results
| extracting data from a specific paper form or report structure.
| Again, not many public examples of that.
| kiratp wrote:
| We use multiple post-trained models in production, at scale
| at https://osmos.io
| simonw wrote:
| Have you published details of how you're doing that
| anywhere?
|
| Could be a useful marketing strategy for you, given how
| _starved_ we all are of information about successful fine
| tuning stories.
| deet wrote:
| Vision LLMs are definitely an interesting application.
|
| At Avy.ai we're running small (2B-7B, quantized) vision
| models as part of a Mac desktop application for understanding
| what someone is working on in the moment, to offer them
| related information and actions.
|
| We found that the raw results in understanding the images
| with a light LORA fine tune are not substantially different
| -- but the ease of getting a small model to follow
| instructions in outputting structured data in response to the
| image and at the level of verbosity and detail we need is
| greatly enhanced with fine tuning. Without fine tuning the
| models on the smaller end of that scale would be much more
| difficult to use, not reliably producing output that matched
| what the consuming application expects
| msp26 wrote:
| Was constrained decoding not enough to force the output to
| be in a specific format?
| deet wrote:
| Using a grammar to force decoding say valid JSON would
| work, but that hasn't always been available in the
| implementations we've been using (like MLX). Solvable by
| software engineering and adding that to the decoders in
| those frameworks, but fine tuning has been effective
| without that work.
|
| The bigger thing though was getting the models to have
| the appropriate levels of verbosity and detail in their
| ouput which fine tuning made more consistent.
| jbentley1 wrote:
| I am. I have some use cases related to data extraction where
| using a fine tuned small model outperforms the best-in-class
| closed source models and at a fraction of the cost.
| icelancer wrote:
| We _were_ but with the models becoming so good, so large, and
| so cheap, we 've largely abandoned it in our long-term roadmap.
| minimaxir wrote:
| Finetuning is easy and worthwhile, especially with LoRAs as
| these Unsloth demos do. The bottleneck then becomes how to
| _self-host_ the finetuned model in a way that 's cost-effective
| and scalable.
|
| In practice prompt engineering and few-shot prompting with
| modern LLMs, due to their strong-and-only-getting-better-over-
| time prompt adherence, tends to be more pragmatic.
| slopeloaf wrote:
| Yeah this big time. I haven't found a solution that makes
| sense. Larger models are already good enough and so
| convenient.
|
| When it's more feasible to do inference on the client
| (browser or desktop) I can see SLMs popping up more common in
| production.
| naveen99 wrote:
| If you have the resources to fine tune, you have the
| resources to run inference on fine tuned model.
|
| If you want to scale up and down on demand, you can just fine
| tune on openai and google cloud as well.
| simonw wrote:
| > If you have the resources to fine tune, you have the
| resources to run inference on fine tuned model.
|
| I don't think that's true.
|
| I can fine tune a model by renting a few A100s for a few
| hours, total cost in the double digit dollars. It's a one-
| time cost.
|
| Running inference with the resulting model for a production
| application could cost single digit dollars per hour, which
| adds up to hundreds or even thousands of dollars a month on
| an ongoing basis.
| curious_cat_163 wrote:
| This assumes that inference is needed 24/7.
|
| That may or may not be true for use-cases that require
| asynchronous, bulk inference _and_ require some task-
| specific post-training.
|
| FWIW, my approach towards tasks like the above is to
|
| 1. start with using an off-the-shelf LM API until
|
| 2. one figures out (using evals that capture product
| intent) what the failure modes are (there always are
| some) and then
|
| 3. post-train against those (using the evals)
| anon373839 wrote:
| There are inference providers such as Together AI that will
| serve your LoRA adapters at no extra cost above the model
| price. Then, there's basically no difference between using
| your fine-tuned model or an API model off the shelf (except
| for the benefits you get from fine-tuning).
| qudat wrote:
| For self-hosting I've been using https://tuns.sh which is a
| tunneling solution using SSH. It works great for prototyping
| and I've been using it to host open-webui
| ekojs wrote:
| > The bottleneck then becomes how to self-host the finetuned
| model in a way that's cost-effective and scalable
|
| It's not actually that expensive and hard. For narrow
| usecases, you can produce 4-bit quantized fine-tunes that
| perform as well as the full model. Hosting the 4-bit
| quantized version can be done on relatively low cost. You can
| use A40 or RTX 3090 on Runpod for ~$300/month.
| refulgentis wrote:
| IMHO the biggest factor holding that back is how rushed and
| distanced these model releases are, still.
|
| Both Phi-4-mini and Gemma 3 were released recently. Phi-4's
| damn close to a good, real, model release. Microsoft's done a
| great job of iterating.
|
| Gemma 3's an excellent, intelligent, model, but it's got a
| gaping blind spot: tool-calling / JSON output. There was a
| vague quick handwave about it in some PR, a PM/eng on the Gemma
| team commented here in response to someone else that TL;DR
| "it's supported in Ollama!", which is Not Even Wrong, i.e. in
| the Pauli sense of the phrase.
|
| - Ollama uses a weak, out of date llama.cpp thing where the
| output tokens are constrained to match a JSON schema. This
| falls apart almost immediately, i.e. as soon as there is more
| than one tool.
|
| - The thing that matters isn't _whether we can constrain output
| tokens_ , any model can do that, I've had Llama 3 1B making
| tool calls that way. The thing that matters is A) did you train
| that in and B) if you did, tell us the format
|
| All that to say, IMHO we're still 6 months to a year out from
| BigCo understanding enough about their own stuff to even have a
| good base for it. Sure, tool calling and fine-tuning are
| orthogonal, in a sense, but in practice, if I'm interested in
| getting a specific type of output, odds are I wanted that
| formatted a specific way.
| tough wrote:
| could one train now a gemma 3 fine tune for tool use?
|
| found this on HF https://huggingface.co/ZySec-
| AI/gemma-3-27b-tools
| 317070 wrote:
| I've been finetuning these models since before chatGPT, and the
| one lesson I've learned is that by the time you have set up
| everything to fine-tune a model, you can expect a newer model
| to do as well with prompt-tuning.
|
| So, unless you hope to stay at the fore front (e.g. to be ahead
| of competitors), there has been no real reason to finetune for
| the last 4 years, at best you could hope to stay about 1-3
| months ahead, depending on how fast you were at setting up your
| training. And if that is what you did hope to achieve, you
| needed to automate on a higher level, i.e. automate data
| collection and the collection of eval cases.
| nwienert wrote:
| It feels like there should be a service where I just drag
| drop a folder of examples and it fine tunes the latest
| DeepSeek or whatever for me and even can host it for me at
| some cost. I'd pay for that immediately, but last I checked
| there was nothing that really did that well (would love to be
| wrong).
| arkmm wrote:
| There are some options out there, depending on what type of
| task you're trying to fine tune. I think RL finetuning for
| DeepSeek e.g. isn't well developed yet, but you can
| finetune a small LLama model (~3B params) for
| classification or extraction tasks and it works really
| well. What sort of tasks were you looking at finetuning
| for?
| nwienert wrote:
| Code generation or question answering. But ideally 70+B
| m101 wrote:
| I feel like this is true but would be great if you could
| provide examples so we could get a better idea of why you
| think/know this.
| netdur wrote:
| I have documents from the last 50 years that I need to
| digitalize, millions of them written in old Arabic. The OCR is
| not accurate due to handwritten documents, so I need to fine-
| tune a model on around 300k pairs of texts (OCR output and
| manually corrected versions)
| Diederich wrote:
| This sounds very interesting; can you share more? Thanks!
| netdur wrote:
| I followed this guide for fine-tuning:
| https://ai.google.dev/gemini-api/docs/model-tuning
|
| Arabic OCR is a mess with historical texts. Take the word
| lf (alf/thousand) in dates like 1950 - in old documents,
| the f (fa) had a dot below it, but modern OCR doesn't get
| this and outputs ld (alad), which is just gibberish in
| Arabic
|
| Same problem with q (qaf) written as f (fa) in old Arabic
|
| And don't get me started on merged letters! In mHmd
| (Muhammad), sometimes the m (meem) sits right on top of the
| H (haa), or appears as a little circle below the line.
| Modern OCR has no clue what to do with these
|
| My solution? Run OCR first, then use LLMs to fix the mess
| based on context. The surprising part? In my tinkering,
| smaller fine-tuned models actually do BETTER at this
| specific task than the big general-purpose ones. They seem
| to learn the patterns of historical Arabic quirks more
| effectively. Pretty neat tradeoff of specialized knowledge
| vs. general intelligence
| siliconc0w wrote:
| It likely makes sense to use more expensive frontier models as
| teachers or architects for smaller fine-tuned ones that generate
| the majority of tokens (though possibly against the ToS).
| yieldcrv wrote:
| Instead of versions, these things should be labeled by their
| release date, since this kind of training is based on started at
| a dataset snapshot in time, colloquially called knowledge-cutoff
| date which isnt really accurate
|
| we are optimizing these on different dimensions at once, and
| multiple branches of evolution from each model
|
| so a successor version name doesn't really convey that
| bryan0 wrote:
| Are people fine-tuning LLMs on their local machines with a single
| GPU? What are people using to scale their training to multiple
| nodes / gpus? I've been playing around with Hugging Face
| Estimators in sagemaker.huggingface but not sure if there are
| better options for this?
| samspenc wrote:
| It takes a significant amount of time (few hours) on a single
| consumer GPU, even 4090 / 5090, on personal machines. I think
| most people use online services like runpod, vast ai, etc to
| rent out high-powered H100 and similar GPUs for a few cents per
| hour, run the fine-tuning / training there, and just use local
| GPUs for inference on those fine-tuned models generated on
| cloud-rented instances.
| jsight wrote:
| For experimentation? Absolutely. It can often be done overnight
| for smaller models and reasonably sized GPUs (24GB+).
|
| It'd become a lot less practical with huge datasets, but I'd
| guess that a lot of fine tuning tasks aren't really that large.
| michaelt wrote:
| Take a look at the hardware requirements at
| https://github.com/hiyouga/LLaMA-Factory?tab=readme-ov-file#...
|
| A 'LoRA' is a memory-efficient type of fine tuning that only
| tunes a small fraction of the LLM's parameters. And
| 'quantisation' reduces an LLM to, say, 4 bits per parameter. So
| it's feasible to fine-tune a 7B parameter model at home.
|
| Anything bigger than 7B parameters and you'll want to look at
| renting GPUs on a platform like Runpod. In the current market,
| there are _used_ 4090s selling on ebay right now for $2100
| while runpod will rent you a 4090 for $0.34 /hr - you do the
| math.
|
| It's certainly possible to scale model training to span
| multiple nodes, but generally scaling through bigger GPUs and
| more GPUs per machine is easier.
| deet wrote:
| Google Colab is quite easy to use and has the benefit of not
| making your local computer feel sluggish while you run the
| training. The linked Unsloth post provides a notebook that can
| be launched there and I've had pretty good luck adapting their
| other notebooks with different foundational models. As a
| sibling noted, if you're using LORA instead of a full fine-
| tune, you can create adapters for fairly large models with the
| VRAM available in Colab, especially the paid plans.
|
| If you have a Mac, you can also do pretty well training LORA
| adapters using something like Llama-Factory, and allowing it to
| run overnight. It's slower than an NVIDIA GPU but the increased
| effective memory size (if you say have 128GB) can allow you
| more flexibility.
| smokel wrote:
| I'm interested to know if anyone is using fine-tuning to train a
| model on proprietary or in-house codebases and documentation.
|
| RAG solutions seem to have their limitations, and fine-tuning
| might be a more effective approach.
|
| How much effort is required to turn code into something one can
| use for fine-tuning?
| t1amat wrote:
| I would like to see more knowledgeable people with experience
| talk about this.
|
| Is it just a matter of assembling Q/A pairs like: "What's class
| X?", "class X { ... }"
|
| Do you really need to do this training on the base model
| instead, which means you have to fine tune chat on it
| afterward?
|
| How does this work?
| Tostino wrote:
| I've not done fine tuning on code bases but I have done other
| fine tuning.
|
| You will generally get better results when you fine-tune the
| base model on your data.
|
| Since you still want to use it with the chat template in the
| end, you fine-tune the base model with the chat template with
| your specific data.
|
| From there you'll have a lora that knows your data alright,
| but still doesn't really work for chatting.
|
| You take that lora, merge it with the base model. Let's call
| this the stage model.
|
| Then you use mergekit to merge the base model with both the
| stage model and the chat model. I used the TIES merge method
| in the past. Now you have your final model.
|
| I use vLLM for inference, and needed access to multiple fine
| tunes on only a single set of hardware. So from that point I
| go and take the base model and my final model and extract a
| new lora. I also take the base model and chat model and
| extract another lora for that. Then I load up vLLM with the
| base model and as many of the fine tune loras I need + the
| chat lora.
|
| The only time this hasn't worked is if the chat model adds a
| bunch of new tokens on top of the base model. If I remember
| right there was an issue with that
|
| This has worked well for me in the past.
| vineyardmike wrote:
| I've actually found the opposite. At work, we went from a fine-
| tuned model to a RAG system for internal and external
| documentation and a generic coding-focused model for code.
|
| Fine tuning against in-house code seems like a small gain over
| a base model and search. It's unlikely your code is unique and
| special and big enough that it's hard to get results from a
| base model. You'll be pinned to a certain version of a certain
| model, and you won't be able to upgrade to future models nearly
| as quickly. Of course, you're also fighting time again on each
| commit changing the code unless you continually fine tune it.
|
| A RAG model might still struggle with a super vague question
| like "where does the foo cal bar with bax set" but it's
| unlikely that this would work for fine tuning as well. This is
| where static code search by symbols really should be used.
| moffkalast wrote:
| Well, why not both? If you've already got a tuned model why
| not use RAG on that to get even better results? It already
| knows the big picture, it just needs the details so it
| doesn't have to hallucinate them.
| fine_tune wrote:
| > I'm interested to know if anyone is using fine-tuning to
| train a model on proprietary or in-house codebases and
| documentation.
|
| I've done it, 1/2 the team though it was great 20% of the time,
| 1/2 the team hated it from day 0. I used roughly 500K lines of
| code.
|
| > How much effort is required to turn code into something one
| can use for fine-tuning?
|
| Very little to moderate, less than 200 lines of python, QWEM
| FIM, HF, LLAMA.CPP, LLAMA.CPP code extension.
|
| > RAG solutions seem to have their limitations, and fine-tuning
| might be a more effective approach.
|
| The only problem either way is keeping the information up to
| date, RAG just adds more cost to the inference process (which
| at my dev speed is pretty important).
|
| > How much effort is required to turn code into something one
| can use for fine-tuning?
|
| Fine tuning "fill in the middle" process is the process of
| taking a file, cutting out a some text in the middle and asking
| AI to guess what was there - there is a hugging face example
| that will have you doing it in an hour or less - your OPs team
| saying "No you cant litreally copy all code to a single folder"
| is probably the biggest hurdle (advise them you'll do it in CI
| and then they can stand up a FIM training endpoint that accepts
| a csv, pretty easy)
| admiralrohan wrote:
| Have anyone used those small models in any production
| environment?
|
| If yes, what they are good and bad at?
___________________________________________________________________
(page generated 2025-03-19 23:00 UTC)