[HN Gopher] Rewriting Libimagequant in Rust for Portability
___________________________________________________________________
Rewriting Libimagequant in Rust for Portability
Author : todsacerdoti
Score : 168 points
Date : 2022-01-04 13:35 UTC (9 hours ago)
(HTM) web link (pngquant.org)
(TXT) w3m dump (pngquant.org)
| bz2 wrote:
| Kornel is also behind the DSSIM library (in Rust), which I've
| found useful for comparing images before and after compression.
|
| https://github.com/kornelski/dssim
| Y-bar wrote:
| And the excellent Mac OS image compression utility ImageOptim:
| https://imageoptim.com/mac
| pornel wrote:
| Fun fact: I wrote DSSIM as a development tool specifically for
| pngquant, because I needed a way to measure improvements in
| dithering. Classic per-pixel algorithms only saw it as added
| noise.
| roblabla wrote:
| > Trying to make the library more object-oriented exposed a
| "drum-stick" design issue: does play(drum, stick) translate to
| drum.play(stick) or stick.hit(drum)?
|
| In cases where OO doesn't make sense, the solution is to simply
| provide a functional API instead. Rust stdlib has plenty of such
| APIs, it's fine to have free-standing functions!
| ReleaseCandidat wrote:
| I guess he is talking about traits. How do you 'functionize'
| them?
| pornel wrote:
| I wasn't thinking about traits when I wrote that, but you
| make a great point -- that's a whole another way to design
| things. Maybe it could have been something fancy like:
| impl<T> Sound for T where T: Percussion<Accessory=Stick>
| hedgehog wrote:
| There's also the pattern used in from+into where one
| implementation is wrapped to provide its counterpart on the
| other type. Nice to read but feels a little extra magical
| for someone without a lot of Rust background.
| tialaramex wrote:
| Blanket implementation is a very common idea in Rust. The
| standard library "owns" its own Traits and so it gets to
| provide blanket implementations for those, but you can do
| the same for your own.
|
| Here's (really, this is literally the code) how std
| provides a blanket implementation of Into given From:
| impl<T, U> Into<U> for T where U:
| From<T>, { fn into(self) -> U {
| U::from(self) } }
|
| There's actually a chain, From implies Into, Into implies
| TryFrom, then TryFrom implies TryInto.
| ape4 wrote:
| Agree, there's no rule that everything has to be a class in
| C++.
|
| I suppose drum.play() could default to using a stick.
| mikepurvis wrote:
| I think the GP's idea is to just do play(drum, stick) as its
| own function and not tie it to either class or have it be a
| passthru into internal methods for either.
| nine_k wrote:
| You can even make everything a class, and still provide a
| functional-ish interface, by using the class as a namespace
| with a common immutable context.
| lostcolony wrote:
| But...that was exactly the question the article had. Should
| drum be the enclosing class, or should stick be? What he
| cares about is the verb play, but he has to encapsulate it
| in a noun, and none of the ones actually involved dominate
| so clearly as to moot having to make a choice.
| nine_k wrote:
| musicPart.play(drum, sticks);
|
| Basicslly you can replace the module namespace with a
| class namespace if desired. What module could that be?
| lostcolony wrote:
| Right. But that's just it. "musicPart" is an object. A
| noun. What the hell is a "Music Part"? It's a completely
| synthetic object you've had to create solely to namespace
| functions. It's like how so many large Java projects have
| static "Utils" classes.
|
| What you've done is not OO; in fact, it's an anti-pattern
| in OO. You've created, as you correctly note, a namespace
| or module around static functions; there is no meaningful
| encapsulation of data.
|
| My point isn't that it can't be done, it's that trying to
| follow best OO practices, you run into this sort of
| problem (which has had a lot of digital ink spilled on
| it). The fact there is a pragmatic solution isn't in any
| doubt; it's that such a solution intentionally eschews
| the data encapsulation that OO demands (but not
| imperative or functional approaches, which instead say
| it's the right approach to do that).
| ReleaseCandidat wrote:
| But now the traits aren't connected with drums (or
| sticks) but MusicParts. Or you end up having some
| `drum.foo` too.
| enriquto wrote:
| > using the class as a namespace with a common immutable
| context
|
| This is why we can't have nice things.
| nine_k wrote:
| Pray elaborate.
| bastardoperator wrote:
| It's funny because I've been using pngquant for optimizing assets
| we display in our Rust video game extensions(by facepunch).
| steveklabnik wrote:
| We get closer and closer to getting Rust re-written in Rust
| every day... hahaha
| pdimitar wrote:
| That's the kind of analysis I want to see. Really good write-up.
| CyberRabbi wrote:
| This is one of those rare situations when monoculture is
| superior. Integrating multiple ad-hoc build systems is a
| nightmare, especially when cross compiling. Using cargo and rustc
| is a dream by comparison. It would benefit the C/C++ community to
| rally around a single build/compilation tool chain. Excellent
| write up!
| Arch-TK wrote:
| Lots of extreme generalisations and exaggerations of the
| situation with C. It's great that you like rust and want to write
| your code in it, I don't really care at the end of the day, but
| vague statements such are not very useful or productive.
|
| >There's no standard for building C programs.
|
| Is there a written standard for compiling rust programs? I don't
| think so. Rust doesn't do standards after all.
|
| Jokes aside, this is basically irrelevant. Why does there need to
| be a standard for building C programs? Making something build
| across platforms (including cross compilation to other platforms)
| has been solved by everyone and their grandmother at this point.
| This means that aside from taking the route of re-implementing it
| yourself (not particularly difficult) you have literally dozens
| of perfectly good options to chose from.
|
| In fact, most of this rant sounds like it stems from your dislike
| of OpenMP not C.
|
| >I could have kept the library in C and replaced OpenMP with some
| other solution instead, but I wasn't keen on reinventing this
| wheel. Even basic things like spawning a thread run into C's
| portability woes.
|
| Thankfully, you don't need to reinvent the wheel, the problem of
| cross platform multiprocessing in C has been solved at least a
| dozen times by now. It should be trivial to use one of these many
| options (or even write your own, even if you don't like the idea
| of "reinventing the wheel" it's certainly not particularly
| difficult either and can likely be lighter than any other more
| general option).
|
| >The platonic ideal portable C exists only as a hypothetical
| construct in the C standard. The C that exists in the real world
| is whatever Microsoft, Apple, and others have shipped. That C is
| a mess of vendor-specific toolchains, each with its own way of
| doing things, missing features, broken headers, and leaky
| abstractions. Shouting "it's not C's fault, screw <insert vendor
| name>!" doesn't solve the problem, but switching to Rust does.
|
| I mean, rust is hardly helping you here. There's one
| implementation of rust doing their own thing so far, nothing
| stops anyone from making this situation as "bad" as in C. You may
| as well just say you only support clang or only support GCC and
| base your project around that assumption. In fact, if you're
| making the mistake of trying to support a microsoft compiler then
| I would argue that you've made your life difficult for no reason
| and blamed it on C.
|
| >dreadful state of C dependency management did.
|
| Hardly any more dreadful than rust's solution to the problem.
| jcranmer wrote:
| > Thankfully, you don't need to reinvent the wheel, the problem
| of cross platform multiprocessing in C has been solved at least
| a dozen times by now. It should be trivial to use one of these
| many options (or even write your own, even if you don't like
| the idea of "reinventing the wheel" it's certainly not
| particularly difficult either and can likely be lighter than
| any other more general option).
|
| This is basically the issue he's complaining about. The author
| _wants_ to reuse a wheel, but reusing the wheel in C is
| difficult. What are your options?
|
| * OpenMP: Great, now you get the fun of figuring out how to
| cross-platform(/compiler) enable openmp in your build system
| (that you have to maintain yourself because there is no
| standard build system. Of course, you could also go with cmake
| or autotools to maintain the build system for you, but now you
| have similar issues with meta-build system stuff. Yay platform
| diversity!)
|
| * Okay, let's use another library instead. Now you _just_ have
| to have the user tell you where it 's located. And maybe have
| to provide lengthy instructions on how to find it. Or you can
| try having the build system automatically download it on the
| fly. There's a lot of potential error messages you have to deal
| with, just to get at what ought to be a core feature of any
| modern programming language.
|
| * Maybe you might want to vendor that library in your source
| tree, just to be sure you always have it. Now you get the fun
| of having your build system invoke another build system as a
| subproject.
|
| * Screw other dependencies, let's just use a newer C standard
| version, since it is in C11. How do you get your C compiler to
| compile for C11? Oh wait, that's basically the same world of
| pain as it is to try to use OpenMP.
|
| > Hardly any more dreadful than rust's solution to the problem.
|
| Contrast this with Rust's solution.
|
| * Your build system is cargo and you specify things primarily
| with Cargo.toml. This is as little work as the happiest happy
| paths of C/C++ build systems. And note you don't really choose
| build system; there is _one_ ecosystem-approved build system--
| for C /C++, you have at the very least hand-rolled makefiles,
| IDE solutions, cmake, and autotools as major candidates, and
| many major projects very happily choose "none of the above"
| (with all the pain that entails).
|
| * If you want to add a dependency in cargo, you add a line in
| Cargo.toml. _That 's it_; you're done. For the aforementioned
| C/C++ build systems, even the happiest happy paths aren't
| _that_ happy.
|
| This isn't to say that Rust's solution is perfect (it has a
| fair amount of warts), but to assert that Rust is no better
| than C is an indication to me that you've never really had to
| actually support the build system of a project before.
| sanxiyn wrote:
| I disagree about dependency management. Let's say, to avoid
| reinventing the wheel, I decided to use GLib for cross-platform
| threading. How do I add GLib as a dependency to my C project?
| Now compare it to adding Tokio as a dependency to my Rust
| project, which is entirely trivial.
| pjmlp wrote:
| You use cmake,vcpkg or conan, or better yet C11.
| rightbyte wrote:
| > How do I add GLib as a dependency to my C project?
|
| Use symbol versioning.
|
| Anyway, I feel the author is a bit unfair. He traded
| supporting multiple compilers for Rust's only and it is not
| really surprising that he felt that the maintainer burden
| decreased.
|
| As soon as Rust gets broad adaption I guess there will be
| subtly incompatible compiler vendors in Rust too.
| masklinn wrote:
| > Use symbol versioning.
|
| That doesn't mean anything.
|
| > Anyway, I feel the author is a bit unfair. He traded
| supporting multiple compilers
|
| They wanted to support multiple _platforms_ , that multiple
| platforms implied multiple compilers _was part of the
| portability issues the original goal led to_.
| rightbyte wrote:
| > That doesn't mean anything.
|
| I meant that you can specify symbol versions to link to
| glibc specifically.
|
| https://sourceware.org/binutils/docs/ld/VERSION.html
|
| > They wanted to support multiple platforms, that
| multiple platforms implied multiple compilers was part of
| the portability issues the original goal led to.
|
| He could have used gcc on different platforms too. He
| writes:
|
| "the platonic ideal portable C exists only as a
| hypothetical construct in the C standard. The C that
| exists in the real world is whatever Microsoft, Apple,
| and others have shipped. That C is a mess"
|
| I.e. his problem could have been solved by sticking to
| one compiler, be it Rust or Clang ...
| jcranmer wrote:
| > I meant that you can specify symbol versions to link to
| glibc specifically.
|
| sanxiyn was talking about GLib, not glibc, and these are
| two very different projects...
| masklinn wrote:
| > I meant that you can specify symbol versions to link to
| glibc specifically.
|
| You're confusing glibc and GLib...
|
| > He could have used gcc on different platforms too.
|
| That certainly would have solved the multiple platform
| issue, I'm sure gcc has great and easy support for
| producing VS-compatible dlls, and that there's nothing an
| ios dev likes to hear more than "you can use GCC to
| compile libimagequant".
| pornel wrote:
| I don't think that fragmentation is a given.
|
| Other implementations may never take off. Python, Node.js,
| Java, PHP, C#, Golang, Ruby are all more popular and older
| than Rust, and did not develop fragmentation like C. They
| have one dominant target, and other implementations have to
| follow it, or die. Rust's situation is closer to these
| languages than to C.
|
| But even if there were serious contenders, Rust has already
| established a strong baseline: there are over 70,000[1]
| packages that require Rust+Cargo in their current form. A
| viable alternative has to support them, or it will be niche
| like mrustc is.
|
| [1]: https://lib.rs/stats
| rightbyte wrote:
| Ye that is true (isn't Java a bad example though with
| Google-Java, OpenJdk vs BigEnterpriseJdk etc?).
| masklinn wrote:
| Java could have been a good example, but Sun had a rather
| strict validation process for calling something Java.
|
| Furthermore, there are big difference in _philosophy_
| with C:
|
| 1. IB and UB are not considered normal parts of
| specifications, meaning there's way less opportunity for
| originality in the interpretation of the specifications
|
| 2. there tends to be an ur-implementation, and notable
| divergences from that tends to be interpreted as either a
| bug in the other implementation(s) or a lack of
| specification to be resolved between all implementations
|
| Rust only has UB in unsafe (AFAIK), which greatly limits
| implementation flexibility in terms of observable
| behaviour; and the reference implementation would very
| much be considered the _reference_ implementation, so I
| expect e.g. rust-gcc will be sticking close to the
| reference implementation and behavioural divergence will
| either be fixed to match, or will lead to more precise
| specification and both implementations converging.
|
| Probably eventually with, if not a Sun-style validation
| suite, a Ruby-style Spec Suite
| (https://github.com/ruby/spec).
| lovasoa wrote:
| Let's say that, as you recommend, I only support one compiler
| and one build system in my C project. What is the "easy"
| solution to the problem stated by the author ?
|
| How do I add multi-threading to my project in a way that allows
| me to run a function on all available cores in two lines of
| code, while making the project cross-compile to every major OS
| and architecture from an M1 mac ?
|
| In rust the solution is just to edit a file named "Cargo.toml"
| and add one line in it: rayon = "*".
| jjnoakes wrote:
| I am excited about rust supporting more platforms so that this
| kind of thing can be done more widely without sacrificing another
| kind of portability.
| nicoburns wrote:
| I think this is an excellent example of why people get so excited
| about Rust. It can't do anything you can't do in C or C++, but it
| makes a whole bunch of things significantly easier and less
| stressful. Which makes maintenance easier, and in practice often
| leads to performance improvements because more ambitious designs
| can be attempted.
| stouset wrote:
| Careful now, you'll get accused of being in the Rust Hype
| Squad.
|
| Snark aside, this is what Rust proponents like myself have been
| trying to say. Rust isn't some magical panacea to all of the
| problems we face in computing, but it _does_ solve some
| frustrating and endemic problems outright while making others
| significantly more tractable. And that it manages to do so
| without sacrificing execution performance (though at a
| generally reasonable cost in compilation times) and keeping
| code complexity more manageable than equivalent C /C++ code is
| just icing on the cake.
|
| What the language adds in complexity--in my experience at
| least-- _reduces_ the complexity of my own programs and makes
| it much simpler to write them and reason about them. There 's
| definitely a learning curve as the language "wants" you to
| structure programs a certain way while it takes time to build
| up the intuition for doing so. But having done that, I've been
| able to apply those lessons to my projects in other languages
| to great success.
___________________________________________________________________
(page generated 2022-01-04 23:01 UTC)