[HN Gopher] I have written gemma3 inference in pure C
___________________________________________________________________
I have written gemma3 inference in pure C
Author : robitec97
Score : 45 points
Date : 2026-01-26 14:05 UTC (2 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| w4yai wrote:
| > It proves that modern LLMs can run without Python, PyTorch, or
| GPUs.
|
| Did we need any proof of that ?
| skybrian wrote:
| Knowing the performance is interesting. Apparently it's 1-3
| tokens/second.
| kgeist wrote:
| ikllama.cpp is a fork of llama.cpp which specializes on CPU
| inference, some benchmarks from 1 year ago:
| https://github.com/ikawrakow/ik_llama.cpp/discussions/164
| jasonjmcghee wrote:
| I guess llama.cpp isn't quite as popular as I had assumed.
| tolerance wrote:
| I imagine so regarding GPUs, right? Is this is a legitimate
| project then doesn't it provide a proof of concept for
| performance constraints that relate to them? Couldn't the
| environmentally concerned take this as an indicator that the
| technology can progress without relying on as much energy is
| potentially spent now? Shouldn't researchers in the industry be
| thinking of ways to prevent the future capabilities of the
| technology from outrunning the capacity of the infrastructure?
|
| I know very little about AI but these are things that come to
| mind here for me.
| yorwba wrote:
| GPUs are more efficient than CPUs for LLM inference, using
| _less_ energy per token and being cheaper overall. Yes, a
| single data center GPU draws a lot of power and costs a
| fortune, but it can also serve a lot more people in the time
| your CPU or consumer GPU needs to respond to a single prompt.
| tolerance wrote:
| I got you, thanks!
| jdefr89 wrote:
| Python and PyTorch all call out to C libraries... I don't get
| what he means by "proving LLMs can run without Python and
| PyTorch" at all. Seems like they don't understand basic
| fundamentals about things here...
| christianqchung wrote:
| A bizarre claim like that would be what happens when you let an
| LLM write the README without reading it first.
| austinvhuang wrote:
| My first implementation of gemma.cpp was kind of like this.
|
| There's such a massive performance differential vs. SIMD though
| that I learned to appreciate SIMD (via highway) as one sweet spot
| of low-dependency portability that sits between C loops and the
| messy world of GPUs + their fat tree of dependencies.
|
| If anyone want to learn the basics - whip out your favorite LLM
| pair programmer and ask it to help you study the kernels in the
| ops/ library of gemma.cpp:
|
| https://github.com/google/gemma.cpp/tree/main/ops
| janwas wrote:
| :D Your code was nicely written and it was a pleasure to port
| to SIMD because it was already very data-parallel.
| behnamoh wrote:
| but why tho? next gemma is coming and no one uses gemma 3 in prod
| anyway.
| NitpickLawyer wrote:
| > no one uses gemma 3 in prod anyway.
|
| Umm, we do. It's still one of the best for eu countries support
| / help chatbot style. It's got good (best?) multilingual
| support ootb, it's very "safe" (won't swear, won't display
| chinese characters, etc) and it's pretty fast.
| behnamoh wrote:
| but it lacks system prompt support.
| gunalx wrote:
| Yep. Before gemma3 we where struggling with multilinguality
| on smaller European languages, and it is still one of the
| batter ones in that regard (even large open or closed models
| struggle with this to some extent). Gemma3 also is still
| pretty decent multi modal wise.
| uncognic wrote:
| I think /* */ single-line comments is a pretty good indication.
| data-ottawa wrote:
| Gemma3 is probably the best supported fine tunable model.
___________________________________________________________________
(page generated 2026-01-28 23:01 UTC)