[HN Gopher] Using mmap to make LLaMA load faster
___________________________________________________________________
Using mmap to make LLaMA load faster
Author : yawboakye
Score : 236 points
Date : 2023-04-05 15:36 UTC (7 hours ago)
(HTM) web link (justine.lol)
(TXT) w3m dump (justine.lol)
| kuroguro wrote:
| Lol, there's a "broken" image in the footer that claims your IP
| for ipv4.games http://ipv4.games/user.html?name=jart
| MrStonedOne wrote:
| [dead]
| ftxbro wrote:
| related hacker news discussion
| https://news.ycombinator.com/item?id=35411909
| ingenieroariel wrote:
| Should we be linking flagged items? That discussion was very
| ugly and unproductive.
| Technetium wrote:
| Usually you'd be very right, but it's contextually relevant
| here due to slaren and jart replying there.
| secondcoming wrote:
| Isn't `madvise` useful if you want to use Huge Pages?
| scottlamb wrote:
| If you're hoping file-backed mmaps from any filesystem other
| than tmpfs/shmem will use huge pages, you will be disappointed.
| :-( See the following note at
| https://www.kernel.org/doc/html/latest/admin-guide/mm/transh...
|
| > Currently THP only works for anonymous memory mappings and
| tmpfs/shmem.
| the8472 wrote:
| Not sure if that's true any longer on recent kernels. There
| have been some changes with folios that I think enable huge
| pages in the page cache. $ cat /proc/<firefox
| process>/smaps [...] 7efca62e3000-7efcaa13d000
| r-xp 00ae3000 00:18 75354786
| /usr/lib/libLLVM-15.so Size: 63848 kB
| KernelPageSize: 4 kB MMUPageSize: 4 kB
| Rss: 58420 kB Pss: 19213 kB
| Pss_Dirty: 0 kB Shared_Clean: 56372 kB
| Shared_Dirty: 0 kB Private_Clean: 2048 kB
| Private_Dirty: 0 kB Referenced: 58420 kB
| Anonymous: 0 kB LazyFree: 0 kB
| AnonHugePages: 0 kB ShmemPmdMapped: 0 kB
| FilePmdMapped: 57344 kB <==== huge pages
| Shared_Hugetlb: 0 kB Private_Hugetlb: 0 kB
| Swap: 0 kB SwapPss: 0 kB
| Locked: 0 kB THPeligible: 1
| VmFlags: rd ex mr mw me sd
| scottlamb wrote:
| Exciting if true!
|
| I see https://docs.kernel.org/filesystems/proc.html
| describes FilePmdMapped as "Page cache mapped into
| userspace with huge pages", consistent with what you are
| saying. I don't fully understand the distinction between
| that and FileHugePages: "Memory used for filesystem data
| (page cache) allocated with huge pages". I wouldn't think
| it'd be possible to map it into userspace as huge pages if
| the kernel hasn't allocated it as contiguous physical
| memory, so there's something I'm missing.
|
| What kernel version did that output come from? Do you
| happen to know if Firefox did anything special to set that
| up? What filesystem type is this?
| leni536 wrote:
| You don't actually need _transparent_ huge pages, but maybe
| you can work with explicit huge pages.
| scottlamb wrote:
| Is there a working way to request explicit huge pages for a
| file-backed mmap on ext4/xfs/btrfs? I'm not aware of it.
| manv1 wrote:
| There was a post a few months ago about a developer who's job it
| was to rework ML stuff into actual efficient code. This reminds
| me of that post because it seems that lots of ML stuff is just
| plain inefficient...in that they use way too many resources given
| the problem.
|
| And they're arguing about mmap, something that's been around
| forever.
|
| It reminds me of that speedup in a package manager because they
| were reading uncached byte-at-a-time off of disk. You need to
| explicitly turn buffered reads off...but why would you do that in
| the first place? Unbuffered reads are almost never a good idea,
| ever.
|
| It makes me wonder what other weird sub-optimal stuff is lying
| underneath the resource behemoth that is ML.
| bob1029 wrote:
| > it seems that lots of ML stuff is just plain inefficient
|
| This is just like any other technology. Use it wrong, you will
| get burned. Doesn't matter how shiny the container is.
|
| "ML stuff" also (mostly?) includes purely statistical methods
| from the 90s that are deterministic and arguably the best way
| to solve a large variety of non-generative problems.
|
| In fact, unless generation of arbitrary output is a major
| objective, it's likely you can solve whatever ML task on a
| workstation from 2010 that uses intel integrated graphics.
| candrewlee14 wrote:
| jart is a genius. What they've done with Blink, Cosmopolitan C,
| Redbean, and now llama.cpp is incredible. It gives me hope for
| the future of systems/low-level programming.
| oceanplexian wrote:
| I suspect you're trolling but Jart is not a genius for co-
| authoring a 300 line change to use mmap(), this is a pretty
| typical system engineering optimization that goes on in big
| companies, which is usually framed as a performance improvement
| for certain use cases with pros and cons, not "Hacker news
| front page 100X revolutionary performance improvement".
| dimator wrote:
| holy cow, her website is enough to give me imposter syndrome
| https://justine.lol/
| stephc_int13 wrote:
| She is very good indeed, but this mmap thing is not comparable
| to her previous work.
|
| Hopefully we should publicize this kind of achievement as a way
| to teach more devs about mmap... (this really should be common
| knowledge)
| candrewlee14 wrote:
| Of course its simpler, I just wanted it to make sure the
| comment still related to her work on this :) In summation,
| what she's done is awesome.
| astrange wrote:
| Using mmap is basic systems programming knowledge and not
| "incredible" even though it's a useful contribution.
| anshumankmr wrote:
| Systems Programmer is itself a niche field, when compares to
| other fields of interest in Software Developement. So let's
| not trivialize anything, cause it can be amazing to someone,
| but not to another. It depends from person to person.
| jesse__ wrote:
| I think reducing this to "using mmap is basic" is pretty
| unfair.
|
| The trick she did overriding malloc & friends to validate
| that the optimization would be worth doing is, in my mind,
| one of the high-points of the paper. It's a very clever way
| of making a meaningful measurement, which was the keystone of
| the entire change.
|
| I've never heard of, thought of, or used that trick, and the
| fact she had it in her arsenal to apply to this very specific
| situation is pretty impressive, to me at least.
| csmpltn wrote:
| Glad to see we're slowly coming to the realization that
| wrangling NumPy and Pandas doesn't actually count as Software
| Engineering.
| anshumankmr wrote:
| Let's not gatekeep what Software Engineering is or isn't.
| saagarjha wrote:
| ...I don't see how this is related?
| baq wrote:
| it's as much engineering as is gluing mmap, recv and
| send... there are people who can do that and can't piece
| together a dataframe pipeline, no need to be passive
| agressive here
| csmpltn wrote:
| > "it's as much engineering as is gluing mmap, recv and
| send"
|
| As you can see from comments up this thread (ref. various
| GitHub issues) - the people "gluing mmap" don't actually
| have a single clue. They can't properly measure memory
| consumption (they don't understand what the numbers
| they're seeing actually mean). They don't understand how
| paging, swapping or virtual memory work. They don't
| actually understand the concept of memory-mapped files,
| why they're there and how they work. They can't explain
| why their code behaves differently when using memory-
| mapped files.
|
| Moments like this are here to remind you that there's
| actual knowledge and skill to building scalable and
| efficient software, and that hustling and copy-pasting
| StackOverflow examples will only get you so-far, as will
| "piecing together dataframe pipelines" in Python.
| baffo32 wrote:
| justine worked hard in these changes in good faith and was
| attacked and experienced character assassination by orchestrated
| trolling, it's so sad
| astrange wrote:
| > I don't think I've ever seen a high-level library that's able
| to do what mmap() does, because it defies attempts at
| abstraction.
|
| I'm not sure what this means but I'm pretty sure I can name
| several "high level libraries" that mmap things. None of those
| are the STL, but it's not exactly perfect design.
| mhh__ wrote:
| I'm not sure what really needs abstracting about it other than
| tuning stuff for a given kernel/system.
|
| It's already _extremely_ high level from a certain perspective
| :)
| plorkyeran wrote:
| Boost.interprocess is an example of an abstraction over mmap
| which solves some of the things the blog post mentions. It
| abstracts away the difference between mmap() and
| CreateViewOfFile(), and gives you smart pointers and
| container types which are close to being drop-in replacements
| for std::vector and std::map that can be stored in a memory-
| mapped file.
| astrange wrote:
| Someone has to own the memory region and know it gets
| munmap()ed not free()d. (deleted?)
|
| If you send it over IPC it's nice to keep it mmapped instead
| of accidentally copying it too.
| pradn wrote:
| It might be the non-obvious problem with mmap : failures are
| surfaced through signals. So, a failure signal handler may run
| at any time. Your program needs to be resilient to this, and
| it's not trivial to do so. It's not a local change to one
| class.
| dundarious wrote:
| I read it to mean mmap is irreplaceable. There is no other
| sophisticated dance of system calls or userspace trickery that
| can achieve what mmap can achieve. She's saying that everything
| up and down the stack, including high level libraries, _do_
| just call mmap, because there would be no DIY alternative with
| similar cost-benefit.
| [deleted]
| 10000truths wrote:
| Except it's not irreplaceable, at least on Linux. userfaultfd
| allows you to define custom page fault handling. With it, you
| can even do crazy things like "mmap" a remote resource by
| making HTTP range requests on a read fault.
| lostmsu wrote:
| Is there anything like that on Windows?
| astrange wrote:
| Yes, it's how JITs like Java work. Though I don't
| specifically know what the Windows API is.
| dundarious wrote:
| You're right. Irreplaceable was a stronger way of putting
| it than the original, I think, so that's more my mistake
| than hers, and I think the contrast with userspace stands.
|
| mmap sits at this lovely intersection between virtual
| memory and the disk, and it's been around for a long time.
| By now there are other means of playing within that nice
| intersection, but mmap is the pop classic.
| stuaxo wrote:
| The original change made intuitive sense, some of the arguments
| against seem a bit weird - asserting that MMAPing the file could
| mean the memory sticks around after the program stops.. no.
|
| Suggesting that MMAP limits things to the size of the ram, well -
| no, as well - paging may happen, but then we are just back out to
| the file.
|
| Honestly, some of the weird assertions wouldn't take long for
| people to double check an verify (or falsify).
| up2isomorphism wrote:
| Smart people contribute to open source and fight with internet,
| wise people provide a service and profit from it.
|
| Maybe sarcastic, but it is how things look like today.
| valine wrote:
| It's hard to keep up with all developments around LLaMA. What's
| the best RLHF alpaca like model you can download right now?
| a5huynh wrote:
| I recently found this list of models that works with llama.cpp:
| https://rentry.org/nur779 (with dl links, albeit given llama's
| licensing gray area, use at your own risk)
|
| The latest so far would be Vicuna, whose weights were just
| recently release.
| tantony wrote:
| Vicune
| menzoic wrote:
| Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%*
| ChatGPT Quality by the Team with members from UC Berkeley, CMU,
| Stanford, and UC San Diego
|
| https://vicuna.lmsys.org/
| dang wrote:
| Related:
|
| _Vicuna: An open-source chatbot impressing GPT-4 with 90%
| ChatGPT quality_ -
| https://news.ycombinator.com/item?id=35378683 - March 2023
| (167 comments)
| isoprophlex wrote:
| Are the Vicuna weights available for download, and are they
| llama.cpp compatible? I can't grok that by skimming the
| page...
| danielbln wrote:
| The weight deltas are available: https://github.com/lm-
| sys/FastChat#vicuna-weights
| TehCorwiz wrote:
| The github page (https://github.com/lm-sys/FastChat#vicuna-
| weights) is better:
|
| > We release Vicuna weights as delta weights to comply with
| the LLaMA model license. You can add our delta to the
| original LLaMA weights to obtain the Vicuna weights.
| detrites wrote:
| Vicuna's fine-tune of the LLaMA weights are available for
| download, called "deltas".
|
| So you get the LLaMA weights (somewhere), then apply the
| Vicuna deltas to them to end up with the Vicuna model.
| patrakov wrote:
| Officially, only as deltas against LLaMa weights, and
| needing a complicated and resource-intensive conversion
| procedure. Unofficially, yes, a pre-converted llama.cpp
| compatible ggml file is available, but obviously I won't
| publish the link here to avoid violating the Y Combinator's
| terms of use.
| adeon wrote:
| I have got Vicuna-13B working on GTX 3090 Ti + OpenCL + CPU
| with 90% of weights on the GPU (otherwise running out of
| memory) at around 500ms per token.
|
| This model is really good for a (semi-)open source model. I
| think this may be the first locally runnable model that I
| will actually use for real stuff rather than just play around
| for fun.
|
| It's not ChatGPT level but it's not that far behind. It will
| draw ASCII art HUDs for a text adventure or analyze data or
| recognize languages or write stories. AFAIK it's been trained
| on ChatGPT discussions so makes sense.
|
| This AI still gets uppity sometimes about offensive content
| but unlike ChatGPT, you can edit the prompts to put words in
| its mouth to encourage it to answer properly.
| VadimPR wrote:
| Mind sharing how you got it to work in your setup?
| adeon wrote:
| I work on an independent LLM implementation here:
| https://github.com/Noeda/rllama/
|
| I only got it working at all yesterday and there's no
| nice UX at all. Not sure I recommend trying to use this
| as llama.cpp will probably have this in no time with a
| much better user experience, although I am also trying to
| make it more usable.
|
| If you follow the instructions on Vicuna page over how to
| apply the deltas, and you can compile the project, then
| you could run:
|
| cargo run --release --features opencl -- --model-path
| /models/vicuna13b --param-path
| /models/vicuna13b/config.json --tokenizer-path
| /models/vicuna13b/tokenizer.model --prompt-file prompt
| --top-p 1.0 --top-k 20 --repetition-penalty 1
| --temperature 0.9 --max-seq-len 2048 --f16 --percentage-
| to-gpu 0.9
|
| Where /models/vicuna13b is the HuggingFace-compatible
| model. This will put 90% of weights on GPU and remaining
| 10% non CPU which is just barely enough to not run out of
| GPU memory (on a 24 gig card)
|
| Create a text file 'prompt' with the prompt. I've been
| using this template:
|
| You are a helpful and precise assistant for checking the
| quality of the answer.###Human: Can you explain nuclear
| power to me?###Assistant:
|
| (the model seems to use ### as delimiters to distinguish
| Human and Assistant). The "system prompt" is whatever
| text is written at the beginning.
| wensheng wrote:
| How did you distribute the weights between CPU and GPU?
| Thanks
| adeon wrote:
| See my response on the sibling comment; I implemented it
| in a custom Rust implementation.
| mhh__ wrote:
| I've seen this on my HN feed about 5 times now, twitter twice.
| Why? (Author's blog is acceptable as they did the change, but the
| other 4?)
|
| Is this AI people learning that mmap exists? People simping cos'
| there a Justine involved?
| MuffinFlavored wrote:
| [flagged]
| HopenHeyHi wrote:
| No? I mean yes, probably will get downvotes from the usual
| crowd.
|
| There is as a matter of fact some of that _explicitly_ going
| on in the GitHub issue, which derailed the conversation.
|
| Such are the woes of being an internet micro celebrity I
| guess. Truly we live in the stupidest timeline.
| MuffinFlavored wrote:
| > There is as a matter of fact some of that explicitly
| going on in the GitHub issue
|
| Could you link an example so I can learn/retract my
| comment? I didn't think there was actually going on.
| HopenHeyHi wrote:
| You can just read it (at least ctrl-f jart) before
| telling people what is/isn't appropriate. As I recall in
| short order people started arguing about jealousy/credit,
| the file format initials being changed in jarts honor,
| "I'm not technical and don't know what's going on but..",
| 'etc. Goofy stuff.
| dekhn wrote:
| If I've learned two lessons in computing over 35+ years, it's:
| you can never have enough RAM, and IO is the hardest bottleneck
| to solve.
| radicality wrote:
| Since the post is from day, so the improvements were all 'real'?
| I didn't follow closely but I remember multiple points people
| brought up earlier like: is the memory counting correct, why
| aren't all the weights accessed for a query, whether quantisation
| is a problem etc. Were all these fixed?
| simion314 wrote:
| There are no memory improvements, people were not measuring
| correct. The giant improvement is the load times after the
| first run(if you do not invalidate your caches). Quantization
| to 4 bit is a big gain, the loss appears to be minimal from
| benchmarks. So with quantization you gain the ability to try a
| bigger model, if you have the hardware to fit the biggest model
| then you can skip it but for most people we need to try to fit
| the biggest model possible in our VRAM or RAM.
| scottlamb wrote:
| Unless the prior code was using O_DIRECT, the data was
| getting loaded into the kernel's page cache, and then the
| application was copying it into its own anonymous memory. Now
| the copy isn't happening. There are some subtleties involved
| [1] but it's not crazy to claim approximately half the RAM
| usage, even before bringing multiple processes into the
| picture.
|
| [1] The kernel doesn't necessarily load the whole thing into
| page cache at once and keep it around indefinitely. It might
| have been recognizing a sequential loading pattern before and
| basically discarding pages almost immediately, where as now
| it might be keeping them for much longer. Or it might now be
| essentially skipping loading the whole thing in at once and
| doing it page-by-page on demand, which could be more RAM-
| efficient but slower. To some extent, you can control these
| behaviors with madvise, mlock, MAP_LOCKED, MAP_POPULATE, as
| well as various sysctls. Also, if it had to page out before,
| the anonymous memory was "dirty" and thus had to be swapped
| (written out to disk) where as the mmap()ed bytes are "clean"
| and can simply be discarded and (if needed to be paged back
| in later) reread from the existing file unchanged.
| simion314 wrote:
| Thanks for the extra clarifications, but the claims were
| something impossible like a 23 Gb model only using 6Gb with
| this change. So maybe before this change it would have used
| a lot more of 23 Gb. I was referring to those miracle
| memory reductions, unfortunetly not possible, I would like
| to try 3 bit qunatizations when models and software will be
| ready(found none in my searches today)
| scottlamb wrote:
| Yes, those claims were a bit much, and in fairness jart
| chimed in to say so too. [1]
|
| fwiw, I'm not a ML person, but it doesn't seem entirely
| crazy to me to think that SSDs are becoming fast enough
| that you could avoid keeping a huge model in RAM in some
| cases. Especially if "computational SSDs" (SSDs that can
| do some basic first-stage computation without
| transferring the input data over PCIe) ever become
| common. (I think some of the ML accelerators for sale
| today might be approximately this.)
|
| [1] https://news.ycombinator.com/item?id=35393615
| dekhn wrote:
| much of performance in computing is about moving the
| memory hierarchy around in ways that are inconvenient to
| programmers.
|
| I made an SSD into a spare swap device, and basically
| treated my system as having RAM+SSD's worth of RAM. It
| allowed me to finish a few big jobs (~96GB RAM) overnight
| that wouldn't have otherwise.
| astrange wrote:
| > There are no memory improvements, people were not measuring
| correct.
|
| Using filebacked pages instead of anonymous memory is a real
| improvement because it doesn't have to get swapped out if
| there's memory pressure. And this program probably isn't the
| only thing running on the machine.
| simion314 wrote:
| I was referring that you would not gain any memory, there
| was no magic compression so you could use a bigger model on
| the same hardware. There were some wild claims made but it
| was some people meassuring memory usage wrong, but you are
| correct there might be some small memory improvements and
| soem speed improvements.
| astrange wrote:
| Well, you can use a bigger model now, it will "just" be
| really slow. This is different from GPUs, which would
| just fail to load larger models than VRAM because they
| don't support paging (unless you build that yourself.)
| ingenieroariel wrote:
| The post is a technical article that describes a very cool
| systems engineering approach to a problem that is usually coded
| away in proprietary code from nvidia.
|
| No wonder a lot of people see it as "mmap, nothing new". But this
| is not the case in a lot of libraries where the norm is to just
| budget for a lot of time moving things to/from gpu and just
| relying on someone else's code.
|
| Instead of accumulating technical debt the owner of the repo
| decided to merge this, have a breaking change and move on. When
| there was some community backlash to the breaking changes there
| was a pull request trying to revert all changes instead working
| through the issues (it was a net win for several users but not
| all, some configurations with slower drives were better served by
| the older approach). There was an ugly back and forth and the
| repo owner decided to ban both the person who did the pull
| request and the author of this post.
|
| This article brings the conversation back to the technical
| merits, the roadmap, credit the the original authors and tones
| down the ownership tone that may have pissed off some community
| members.
|
| That pull request has now been closed by the owner of the repo.
| They are trying to move on and be productive, let's do the same.
| crims0n wrote:
| Read that as nmap and got really confused.
| NickGerleman wrote:
| Worth pointing out, there has been quite a bit of contention
| around this change, both technical, and some accusations of
| plagiarism/miscrediting here.
| https://github.com/ggerganov/llama.cpp/pull/711
| bestcoder69 wrote:
| Also worth pointing out that you can follow the thread's link
| to Rentry, which links to a 4chan (?) archived thread, where
| you can see anons getting worked up over jart being a trans
| internet celebrity. And unless you're playing dumb, you have to
| admit they were looking for an excuse to troll jart. Unless you
| seriously want me to believe they were all _that_ mad about...
| mmap
| HopenHeyHi wrote:
| Unfortunately Justine has attracted a peculiar
| fanbase+haterbase. As their numbers swell the collective
| intelligence and technical understanding diminishes.
|
| So the discussions end up gravitating towards weird drama. I
| wish you wouldn't have linked this thread. Theres going to be a
| bunch of stupid comments here as well about how great/awful
| jart is.
| oceanplexian wrote:
| I'm not a fan or a hater, I didn't even know who this person
| was until this thread.
|
| Does the change deserve a blog post or wild claims like
| "llama.cpp is 100x faster and uses half the memory!"? No. The
| original PR looks like a decent addition but the blog posts
| reads as incredibly narcissistic (i.e. lots of language like
| "We spent several weeks volunteering" and "our project") uh
| whatever. It also breaks a backwards compatibility when
| there's no technical reason it couldn't have been optional or
| put behind a feature flag, plus a ton of condescending
| language in the PR. Not really the kind of work I'd be proud
| of or would be advertising in a blog post.
| whimsicalism wrote:
| > The original PR looks like a decent addition but the blog
| posts reads as incredibly narcissistic
|
| Better than most stuff I see in the corporate world.
| HopenHeyHi wrote:
| Yes, exactly.
|
| The claim that it uses half the memory was probably a
| honest mistake. The ensuing disappointment that it did not
| in fact halve memory usage and drama attracted trolls and
| white knights and is icky. The discussion around nmap I
| suppose is subtle and when emotion abounds can no longer be
| had. :/
| popip wrote:
| [flagged]
| djur wrote:
| Is this related to her advocacy for neoreactionary politics
| or is it just a transphobia thing?
| saagarjha wrote:
| I mean, there's also the part where she's wrong a lot.
| HopenHeyHi wrote:
| [flagged]
| pantalaimon wrote:
| Is mmap really that broken on Windows? Or is the poster just
| confused that the data stays in the page cache? But that's what
| the page cache does - that memory will be used for other things
| if needed, but if the memory is not needed it might as well
| keep the old data in cache.
| tedunangst wrote:
| There's a weird breed of programmer who only wants to see the
| free memory column in top be maximized. I bought all this RAM
| and I want to make sure none of it is used in case I want to
| use it later.
| Analemma_ wrote:
| No, mmap on Windows is fine. A generous, charitable statement
| would be that the OP on that thread is very confused, but
| based on some comments elsewhere on this thread about jart
| attracting a chorus of haters, it seems more likely that
| they're just trolling.
| chad1n wrote:
| [flagged]
| BeefWellington wrote:
| Greg's own comment on the changes:
|
| > Regarding the version comment - yes, the plan was to bump
| versions and no the magic. But I'm ok to change the magic to
| commemorate the significance of this update. In fact, maybe
| we can make this a thing and everybody who makes a
| significant contribution to the project will get their
| initials appended to the version. What do you think? smile
| vore wrote:
| jart is a woman, for what it's worth.
| chad1n wrote:
| I just presented the drama and I referred to him as `he` as
| a developer, my teammates are mostly male, so I usually say
| `he`.
| whimsicalism wrote:
| Right, so maybe you shouldn't do that?
| timpani wrote:
| [flagged]
| tedunangst wrote:
| Do you also refer to your female teammates as he?
| timpani wrote:
| [flagged]
| timpani wrote:
| [flagged]
| vore wrote:
| Did you make a new account just to do this?
| dekhn wrote:
| This sort of statement really don't contribute anythign
| to the discussion and in fact greatly distracts from the
| technical content. We don't need to hear your opinions
| about trans people in this thread.
| mew12623 wrote:
| [dead]
| 514723553 wrote:
| If people were mad at Greg for changing or not changing a
| magic number--then they can just make a PR to fix it! That'd
| be so easy!
| chad1n wrote:
| Greg didn't change it, it was changed in Jart's pull
| request. Also they can't just make a PR to fix it, because
| the models were already converted to that magic string that
| was changed for no reason.
| 514723553 wrote:
| Well if it wasn't Greg, then I feel like I need to ask--
| why the heck would you use he/him to refer to Justine?
| I'm glad you've changed your mind and referred to her
| with they/them above, though!
|
| > they can't just make a PR to fix it, because the models
| were already converted to that magic string that was
| changed for no reason.
|
| I _think_ the magic string was changed _because_ of
| versioning issues--it sounds like you're arguing that the
| magic version number _instead_ of the magic string
| should've been changed...but it sounds like Justine _was_
| concerned with versioning, even if the versioning wasn't
| done in what you're saying is the best possible way. I
| just don't think that "one magic number was changed
| instead of another magic number" really warrants this
| level of animosity.
| dekhn wrote:
| I feel significantly dumber for reading that merge request.
|
| The one thing to understand is that the performance
| implications of mmap are _subtle_ and only work when you have
| much more RAM than the files you 're mapping in.
| marginalia_nu wrote:
| > only work when you have much more RAM than the files you're
| mapping in.
|
| Really depends on what you're doing, like memory access
| patterns. I've definitely seen scenarios when mapping
| hundreds of gigabytes of data on dozens of gigabytes of ram
| where mmap has been an almost absurd performance boost over
| traditional I/O, both immediately but also asymptotically as
| all the most frequently accessed data ends up in cache and
| the least accessed data is paged out.
|
| I don't disagree with the subtlety part though. It's very
| difficult to reason about I/O performance in general. Modern
| systems are like an onion of hidden performance optimization
| tricks and caching layers (both in software and hardware).
| Karrot_Kream wrote:
| Yeah and on top of that, different systems (software and
| hardware combos) are different, so I can see the
| performance of this depending on the implementation of mmap
| on the system and the implementation of caches and virtual
| memory on the architecture. When I've debugged stuff like
| this, it's either been for myself in which case I know what
| combo I'm running on or it's been for work where we know
| which combinations we target and we run regression tests to
| observe perf implications.
| jjoonathan wrote:
| > least accessed data is paged out
|
| Aren't all the weights touched in every pass?
| marginalia_nu wrote:
| Speaking in general.
| londons_explore wrote:
| In this case, the main benefit is from multiple invocations
| of the same program.
|
| Using mmap, you avoid doing any work at all the 2nd time you
| load the file.
| bigodanktime wrote:
| What do you mean by work. The underlying page cache will
| keep much of the data actual cached if it's recent. Even
| databases like PostGreSQL use this to their advantage (http
| s://github.com/postgres/postgres/blob/master/src/backend...
| ).
| astrange wrote:
| Copying the file backed pages to heap memory and possibly
| having to swap them out.
| bigodanktime wrote:
| I may have parsed your statement incorrectly, but I'm
| assuming you are talking about the copy of data when
| using either mmap or File IO (memcpy versus write)
| Whether you do File IO versus mmap, there's going to be
| copy. With files, the copy occurs within kernel space
| with data being copied into the pages in the buffer
| cache, with mmap the copy occurs in userspace with data
| being copied into the address space. Swapping can occur
| in the buffer cache or mmap, this is why so many
| databases implement their own buffer cache to ensure
| specific data isn't flushed, leaving them in an
| inconsistent state.
|
| An advantage of copying in userspace is the ability to
| use more performant instructions to perform the memcopy,
| which the kernel does not typically have access to
| (https://www.mongodb.com/blog/post/getting-storage-
| engines-re...)
| astrange wrote:
| > With files, the copy occurs within kernel space with
| data being copied into the pages in the buffer cache,
| with mmap the copy occurs in userspace with data being
| copied into the address space.
|
| There is no copy with mmap, the page is either unwritable
| or CoW. There's always a copy with read(). (But read()
| can still be faster and more memory efficient
| nevertheless.)
|
| > An advantage of copying in userspace is the ability to
| use more performant instructions to perform the memcopy,
| which the kernel does not typically have access to
| (https://www.mongodb.com/blog/post/getting-storage-
| engines-re...)
|
| Darwin kernel does though.
|
| I believe Linux uses the builtin old memcpy instructions
| on Intel, just to force CPU vendors to keep them usable.
| bigodanktime wrote:
| > There is no copy with mmap
|
| You are right, if you are directly modifying the mmaped
| region. I always internally model my data as staging my
| changes to be synchronized to the mmaped region, so thats
| my mistake there.
|
| > the page is either unwritable or CoW.
|
| This is not universally true, or maybe I'm confused on
| this statement. MAP_SHARED exists, but maybe you are
| referencing a specific kernels' implementation on how
| they achieve coherence between file backed shared memory
| regions in two processes? Im not sure.
|
| > Darwin kernel does though.
|
| Sure we can always point to a kernel that has has
| implemented some feature or another, which is why I said
| typically you don't see it.
| saagarjha wrote:
| > Darwin kernel does though.
|
| It does not. Compare the implementation of _bcopyout
| against _platform_memmove, you'll see the difference :)
| astrange wrote:
| Huh, maybe I was thinking of "you can use floating point
| in the kernel".
|
| That doesn't work in every kernel because they don't want
| to bother saving/restoring the extra registers.
| dekhn wrote:
| Yes- I have 35 years experience with UNIX and used to use
| mmapping with BLAST, a sequence search tool, as well as my
| own codes.
|
| I'll repeat myself: mmap is subtle. If what you mmap is
| larger than your host RAM, only some of the pages will be
| loaded at any time, and depending on access patterns, can
| lead to significant paging.
| bigodanktime wrote:
| I can understand these folks struggling with what mmap is
| actually doing. But this isn't a new discussion about the
| qualities of MMAP versus file based IO etc. Although, many of
| the comments stated are quite wrong.
|
| Related Work on this problem: 1.
| https://www.mongodb.com/blog/post/getting-storage-engines-re...
| - talks about developments on MongoDB's backend to use mmap. 2.
| https://www.pdl.cmu.edu/PDL-FTP/Database/p13-crotty.pdf - Talks
| about some of the cons of mmap, some I think are not as
| prevalent due to the existence of low latency, high throughput
| storage devices. 3.
| https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up...
| - less relevant but related.
| yieldcrv wrote:
| only thing this discussion has showed me is that more people
| need Computer Science degrees again
|
| like, wow, mmap and paging. really guys?
| eternalban wrote:
| I actually don't understand this. I think a lot of regular hn
| commentators are just avoiding these threads [given the
| questionable circumstances surrounding the related PRs and
| the "drama"].
|
| We've had regular discussions on HN about various storage
| engines, how the latencies are cut down, etc. I share your
| surprise at hearing 'wow, mmap!' and all the debates in the
| issues as what it actually does.
|
| https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu.
| ..
| stephc_int13 wrote:
| I feel the same.
|
| I maybe should not be surprised, given that we live in the
| era of Unity and Electron, but using mmap() to load large
| files should be not be seen as rocket science.
|
| And this is basically available on almost any platform with a
| MMU and a kernel.
| yieldcrv wrote:
| I think there just hasn't been a consumer application that
| is really resource constrained, for a long time now. Only
| things for enthusiasts have been. LLMs have product market
| fit, but running a useful one client side is resource
| constrained, but instead of it truly being a consumer
| hardware limitation, it just turns out they were never
| optimized to begin with - coming from the perceived "top
| AI/ML minds" at FAANGs, while some of the most basic
| optimizations are seemingly a lost art.
|
| On the other hand, its only been a few weeks, so maybe I
| should ignore this absurdity and just wait.
| kristjansson wrote:
| > never optimized to begin with
|
| I think the better read is that they're being adapted to
| new applications, constraints, and environments, all at
| once.
| telotortium wrote:
| Probably a combination of (a) ML framework people not
| paying much attention to CPU inference due to already
| having GPUs/TPUs already lying around for training - CPU
| inference is just for very quick experiments (b) research
| code has never been the best optimized for performance
| (c) ML people are not generally systems programmers, and
| a lot of systems programmers are afraid to mess with the
| ML code outside of low-level computation kernels (doesn't
| help that ML code is notoriously unreproducible).
| chad1n wrote:
| Why would Facebook care about running LLAMA on a cpu with
| optimizing for 1-2% more latency when it has a lot of
| A100s laying around?
| astrange wrote:
| > while some of the most basic optimizations are
| seemingly a lost art
|
| mmap isn't relevant to anyone except CPU-using
| programmers because other hardware doesn't have virtual
| memory paging. Firmware programmers don't care, GPU
| programmers don't care.
| leni536 wrote:
| AFAIK CUDA offers unified memory which basically works
| with virtual address space and page faulting in data from
| main memory. There is also IOMMU in general.
| dekhn wrote:
| Many of us would like to get rid of the host CPU and have
| ML trainers that are just GPUs and drives and NICs all
| attached to a northbridge. The GPU has everything
| required to make disk requests over the bus, and ideally
| the drive can receive network messages that get plumbed
| straight to the drive (I'm only partially joking).
| danieldk wrote:
| Word embeddings were big for their time (especially with
| subword embeddings like fastText). We _mmap_ ed word
| embeddings for similar reasons. But yeah, I was kinda
| surprised that one post about LLaMa.cpp mmap support
| talked about a 'fairly new technique'. _mmap_ has been in
| a UNIX programmer 's tool belt for literally decades.
| Dwedit wrote:
| Using memory mapped files is not always the right answer.
|
| Memory mapped files have their disadvantages. The biggest
| disadvantage is that any disk read error (or yanking the
| USB drive) becomes an access violation exception (also
| known as a crash), just like you read from a bad pointer.
| You need to have robust exception handling, which is a
| taller order than just checking a return value.
|
| Another disadvantage is that even when you have your pages
| mapped into memory, calling the page fault handler and
| getting your page has a cost of ~1200 CPU cycles on Windows
| just to do the User<->Kernel mode transition, plus the cost
| of actually performing the IO. "Just reading the file"
| skips many User<->Kernel mode transitions, so it's one per
| read call rather than one per page fault.
| dekhn wrote:
| although it's true that many hardware problems exhibit as
| SIGBUS on memmapped memory, remember that this is an API
| and implementation written for high performance disk
| drives on important servers; for example, the ingres
| server on berkeley's research vax (IIRC mmap became used
| widely after one of the BSD 4.3 subreleases was
| released). IE, at the time, the idea of a drive that
| could be easily detached being used for production
| computing would have been crazy so I think crashing the
| app when a drive is removed is not completely insensible.
| marwis wrote:
| Wouldn't huge pages and readahead make number of page
| faults and context switches potentially smaller than with
| read()?
| [deleted]
| super256 wrote:
| > only thing this discussion has showed me is that more
| people need Computer Science degrees again
|
| You have too much faith in unis. Mine did not teach me about
| mmap at all.
| crysin wrote:
| I'm in a grad program for Software Engineering. At my
| university, the only difference between the Comp Sci and
| Software Engineering degree is that comp sci requires an
| advanced algorithm class whereas software engineering has a
| capstone class where you have to work with a team to build
| a MVP that is unit tested, uses CI/CD, and obviously works.
|
| I say this to highlight the parent comment. I'm essentially
| in a computer science program and we have learned
| absolutely 0 about paging or memory in any of my required
| courses. We practically don't touch OS anything in any of
| the classes. That's not to say the courses for that aren't
| offered but they aren't part of the core curriculum and
| over my time in my program, they've mostly not been offered
| due to lack of student interest.
|
| I did learn how to use linked lists like a champion though!
| 514723553 wrote:
| I don't get the "plagiarism/miscrediting" accusations. This was
| in the original PR
| (https://github.com/ggerganov/llama.cpp/pull/613):
|
| > This PR was written in collaboration with @slaren. This PR is
| also rebased on PR #586 so please do not squash merge! Use
| either merge or rebase.
|
| jart made sure to that the other user got credit, in addition
| to making sure that their name was properly attributed in the
| commit log. Given all this, it feels like the drama--shouldn't
| exist? Like, if there's an issue with attribution, it's not
| because of bad-faith, and I feel like a good-faith conversation
| could have just resolved this, instead of bringing in trolls.
| IAmNotACellist wrote:
| That's not the original PR. jart was working on a malloc()
| approach that didn't work and slaren wrote all the code
| actually doing mmap, which jart then rebased in a random new
| PR, changed to support an unnecessary version change, magic
| numbers, a conversion tool, and WIN32 support when that was
| already working in the draft PR. https://archive.ph/Uva8c
|
| This is the original PR:
| https://github.com/ggerganov/llama.cpp/pull/586.
|
| Jart's archived comments:
|
| "my changes"
|
| "Here's how folks in the community have been reacting to my
| work."
|
| "I just wrote a change that's going to let your LLaMA models
| load instantly..."
|
| https://archive.ph/PyPFZ
|
| "I'm the author"
|
| https://archive.ph/qFrcY
|
| "Author here..."
|
| "Tragedy of the commons...We're talking to a group of people
| who live inside scientific papers and jupyer notebooks."
|
| "My change helps inference go faster."
|
| "The point of my change..."
|
| "I stated my change offered a 2x improvement in memory
| usage."
|
| https://archive.ph/k34V2
|
| "I can only take credit for a 2x recrease in RAM usage."
|
| https://archive.ph/MBPN0
|
| "I just wrote a change that's going to let your LLaMA models
| load instantly, thanks to custom malloc() and the power of
| mmap()"
|
| https://archive.ph/yrMwh
|
| slaren replied to jart on HN asking her why she was doing and
| saying those things, and she didn't bother to reply to him,
| despite replying to others in that subthread within minutes.
| https://archive.ph/zCfiJ
| empathy_m wrote:
| Hmm, based on what you've quoted here and knowing nothing
| else but a few messages on AI Twitter I would invest in
| jart.
|
| This is BillG-style product skill -- there is a ton of work
| that goes into representing a piece of software as
| something important and valuable that people should buy
| into.
| smoldesu wrote:
| Jart is a pretty exceptional engineer, even if she wrote
| this patch single-handedly it would hardly be a footnote
| in her list of professional accomplishments. This is the
| author of Cosmopolitan libc, redbean and APE we're
| talking about, after all.
|
| That being said, it's important to attribute work
| properly. It can be easy to mix things up (eg. "my patch"
| is excusable) but repeatedly insisting authorship when
| you're not the author of the change just seems
| disingenuous. I'm sure it was in good faith, but since
| they didn't address the issue or clear anything up, it's
| come to this.
|
| Dramatic, and hardly the conclusion people wanted to the
| story of a free performance improvement. It's not
| entirely contrived though, and I think the maintainer
| handled this exceptionally well given the circumstances.
| eternalban wrote:
| > > This PR was written in collaboration with @slaren. This
| PR is also rebased on PR #586 so please do not squash merge!
| Use either merge or rebase.
|
| I read that PR (didn't click any links) and here on HN posted
| a "Great work" to jart. The reason I did that is precisely
| because those final lines in the PR came across as an upright
| acknowledgement that some people helped out. I also got the
| impression that jart was a co-owner of the project with all
| the "we"s that were thrown around.
|
| If I was writing that PR, it would be something like "this PR
| consolidates slaren's mmap approach with additional work done
| for ... by myself". _After_ hearing about the drama, actually
| reading slaren 's PR, and reviewing jart's comments in issues
| and the PR and the hn show and tell, I am now convinced this
| is someone who wants to steal other people's thunder. Heck,
| even this front page article is yet another PR stunt. I
| suspect "faster fork of llama.cpp" posts will follow.
|
| Giorgi Gerganov remains for me the hacker hero here as far as
| LLMs are concerned -- mmap is kiddie stuff to be frank, but
| anyone who gets whisper and llama to work on my laptop with a
| handful of files (many thanks to you sir) has my technical
| respect. And I think he has made the right call regarding the
| project.
| chad1n wrote:
| I think that Georgi regrets making the project so openly to
| PR, he was probably happier with running it on his own.
| gmork13 wrote:
| This is from today apr 5 saying the mmap change loads twice as
| big models with x100 speed up - is this not a blatant lie?
|
| Wasn't it discovered last week that loading larger models was an
| error in measurement and the speed up was from keeping things in
| memory after the first loading?
|
| Please do correct me if I'm wrong.
| saurik wrote:
| Justine knows this and it is stated right there on the page:
|
| > The first time you load a model after rebooting your
| computer, it's still going to go slow, because it has to load
| the weights from disk. However each time it's loaded
| afterwards, it should be fast (at least until memory pressure
| causes your file cache to be evicted).
| blitzkrieg3 wrote:
| mmap() will keep things in memory after first loading, but the
| page cache will _also_ keep things in memory after first
| loading. The difference is in order to re-use that you still
| need to read the file and store yourself (requiring 2x memory),
| instead of just doing a memory access. This has two
| consequences:
|
| * 2x memory. A 20G data set requires 40G (20 for page cache and
| 20 for LLaMA)
|
| * Things would be _even slower_ if they weren't in page cache
| after first loading. mmap is fast because it does not require a
| copy and reduces the working set size
| IshKebab wrote:
| Why would you need to keep the entire file in cache though?
| jerf wrote:
| "Blatant lie" seems a bit strong. Running a large model for a
| second time in a row is a pretty common use case and that
| speedup strikes me as real in that common case. Attribution may
| have been wrong but the time saved is real.
| gmork13 wrote:
| So it can load twice as large models somehow?
| mb_18850 wrote:
| [dead]
| rossgellerty wrote:
| [flagged]
| antimatter15 wrote:
| Note that as a result of some llama drama associated with this
| change, @jart (author of this post) and @anzz1 are apparently no
| longer welcome as collaborators on llama.cpp:
| https://github.com/ggerganov/llama.cpp/pull/711#issuecomment...
| version_five wrote:
| I can't really understand what's going on after a few minutes
| poking around (best I can tell an argument over attribution?),
| but I certainly sympathize with the project author having to
| deal with some stupid internet drama that takes away from a
| cool project. This seems to be one of the less-discussed
| challenges with popular open-source.
| bobbylarrybobby wrote:
| The issue is that mmap was unilaterally (or close to
| unilaterally) implemented and made the only way of loading
| files. Users do not have an option to continue to not use
| mmap.
| lostmsu wrote:
| Is there a reason not to?
| ncann wrote:
| I found this summary from that thread:
|
| https://rentry.org/Jarted
| pantalaimon wrote:
| > @slaren made 7 commits in his fork, which @jart then
| squashed down into one
|
| Good lord, it's terrible when the peanut gallery feels like
| they have to comment on development practice. Why would
| numbers of commits be a relevant metric in an Open Source
| project? Of course squashed commits are easier to handle
| during reabses and such, and when that work can be squashed
| to a single "initial mmap support" commit, then that's
| fine.
|
| > @jart rewrote @slaren's code, which slaren wrote first
|
| now this is just kindergarten level of arguments
| Operyl wrote:
| Which Jart contests to some degree:
| https://news.ycombinator.com/item?id=35413289
| bestcoder69 wrote:
| Click the 4chan links and you'll see in their own words
| what this was really about (trans maintainer, channers
| worried about their waifu bot getting cucked.) Most seem to
| not know what they're talking about- some admit to being
| retarded. Highly suspicious of this being tech related.
|
| Also note the stats on GH subscribers and stuff. This is a
| lolcow dossier...
|
| Events like this make me glad I don't contribute OSS. I'll
| keep my coombots proprietary.
| EGreg wrote:
| I didn't know Justine was trans. Her wikipedia article
| doesn't mention it, and has another female middle name
| which suggests that her parents gave the names to her.
|
| I am not much familiar with her work except the
| impressive Cosmopolitan / Redbean mentioned on HN in the
| past. But she seems to be quite a controversial figure
| that is for some weird technocracy and against democracy
| and leftists, despite being a leader in the zucotti park
| protests... in short, someone who is no stranger to drama
| and controversy, and actively courts it:
|
| https://www.thedailybeast.com/articles/2014/08/01/occupyi
| ng-...
| bestcoder69 wrote:
| All true, stuff I didn't like about her, but also ancient
| history (article is from 2014!)
|
| And like I said, you can follow the links and see the
| brigade discussing something else entirely.
|
| Personally I don't want an OSS ecosystem that banishes
| trans people _or_ people who had weird proto-alt-right
| politics pre-Trump. If you're gonna banish anyone, banish
| the ones posing existential risks to projects by their
| brigading against contributors they don't like.
| EGreg wrote:
| Yeah, I definitely prefer to be part of very inclusive
| and open OSS ecosystems, that do things in good faith.
|
| I am not part of the YCombinator or West Coast ecosystem,
| but didn't it banish gay people with weird pro-alt-right
| politics pre-Trump, and then supported Trump? Like, for
| some reason there was a movement to banish Peter Thiel:
| https://mashable.com/article/peter-thiel-y-combinator
|
| How do you feel about banishing people who simply
| admitted to voting for Proposition 8:
| https://www.latimes.com/business/technology/la-fi-tn-
| mozilla...
|
| If you take a look at a larger problem, you'll see that
| there is a lot of inconsistency with human welfare on a
| far larger scale.
|
| Because this same system takes Saudi money a lot. The
| only moment of self-reflection came after one guy,
| Kashoggi, was killed:
| https://www.barrons.com/articles/saudi-arabia-tech-
| fundraisi...
|
| But not the situation of millions of people in Yemen:
| https://news.un.org/en/story/2022/03/1113852 ...
| https://techcrunch.com/2023/04/01/andreessen-horowitz-is-
| now...
|
| The US military industrial complex was largely involved
| in airstrikes on Yemen, as the Washington Post revealed
| last year: https://www.washingtonpost.com/investigations/
| interactive/20...
|
| As a country, we ignore the Yemen war, and are told to
| only clutch pearls about taking money from Russia due to
| the Ukraine war. I imagine that YC stopped taking Yuri
| Milner's money a decade ago, partly because of his ties
| to the Kremlin, but probably it was just a natural
| parting of ways eventually:
| https://news.ycombinator.com/item?id=15631084
|
| Anyway, just saying ... ecosystems aren't always perfect.
| baffo32 wrote:
| justine worked hard on sorting out these changes and was
| attacked and experienced character assassination via
| orchestrated trolling, it's so sad
| jchw wrote:
| There's something extremely odd about the drama that occurs
| around justine in particular. It makes me feel there's some
| iceberg of things I don't know going on and I genuinely don't
| have the faintest clue what it is. I just know they make some
| cool software, like APE, cosmopolitan libc, and I believe
| landlock-make, which are inspirational projects to people who
| love clever yet practical hacks.
|
| As for these LLaMA changes, I ran it on my machine for fun,
| and it worked perfectly. I wound up re-converting my models,
| but it doesn't take terribly long to do so even for 65B.
| After that, generation starts nearly instantaneously, which
| is very impressive. I wouldn't be surprised if there are
| legitimate problems with the change. Obviously people who
| deleted their local copy of the original model to save disk
| space are probably displeased, and maybe it is a massive
| performance reduction in some cases.
|
| I wish I understood, and yet I fear I don't really want to
| know at the same time.
|
| edit: At least in this case, it seems like it's mostly drama
| around attribution and unnecessary changes. Kind of sad that
| an otherwise really useful code change wound up being marred
| by probably-avoidable drama, but such is life -\\_(tsu)_/-
| Honestly, I don't have any input, I just hope everyone can
| resolve their gripes amicably in due time.
| [deleted]
| mew12623 wrote:
| [dead]
| IshKebab wrote:
| [flagged]
| detrites wrote:
| Probably - if sadly - the right decision. Despite being a great
| feature and improvement, something simply "happened" to the
| project vibe and so the community, apparently as a direct
| response to that PR.
|
| Similarly, this GH issue response to that occurrence, despite
| having made valid points in a reasoned manner, also held some
| of the same kind of "happen", in response - which is
| understandable, but not diffusive.
|
| It ultimately doesn't matter who contributes, if someone truly
| believes in the project, they'll be just as happy to step away
| from it if they're affecting its momentum, even through no
| fault of their own or just a misunderstanding.
|
| Momentum is important - and at an early stage like this, when
| vibe is building and community is forming, it can be _very_. I
| hope ggerganov continues to make these difficult decisions
| characteristic of clear leadership.
| dasl wrote:
| > One of the downsides of the Linux cp command, is copying a file
| larger than RAM will destroy every existing entry in the file
| cache. Under normal circumstances this is a good thing, since a
| least recently used strategy usually works. However it can be
| problematic if you're just organizing your files on a production
| system where you don't want to disrupt performance. As far as I
| know, no standard command line utility offers a way to exploit
| this functionality.
|
| I think `dd` in conjunction with the `oflag=direct` has this
| functionality. See:
| https://stackoverflow.com/questions/33485108/why-is-dd-with-...
| kccqzy wrote:
| I choose to use a modern file system and use cp
| --reflink=always.
| jancsika wrote:
| I still don't understand: why was the magic number changed in
| addition to the file version number?
|
| Edit: can someone running llama.cpp ask it whether it thinks it's
| a good idea to concatenate a running list of vanity initials of
| important developers into a magic filetype constant?
| vkoskiv wrote:
| It was a significant change, and Greg - the original author of
| llama.cpp was fine with it:
|
| > Regarding the version comment - yes, the plan was to bump
| versions and no the magic. But I'm ok to change the magic to
| commemorate the significance of this update. In fact, maybe we
| can make this a thing and everybody who makes a significant
| contribution to the project will get their initials appended to
| the version. What do you think? smile
___________________________________________________________________
(page generated 2023-04-05 23:02 UTC)