[HN Gopher] Zigler: Zig NIFs in Elixir
___________________________________________________________________
Zigler: Zig NIFs in Elixir
Author : ksec
Score : 110 points
Date : 2024-10-24 17:53 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| lionkor wrote:
| Completely lacking a description that made it clear, but
| basically, from what I can tell, this lets you embed Zig code
| inside Elixir code
| G4BB3R wrote:
| Are sigils (~) restricted to one char? To me seems ~Zig would be
| more clear and short enough.
| Miner49er wrote:
| Erlang sigils are not, they can be any length, limited to
| characters allowed in atoms.
|
| Elixir sigils also allow multiple characters in the name, but
| chars after the first must be upper case, according to the
| docs.
|
| So for Elixir, it would have to be something like ~zIG
| throwawaymaths wrote:
| According to the docs, must be all upper case:
|
| > Custom sigils may be either a single lowercase character,
| or an uppercase character followed by more uppercase
| characters and digits.
|
| https://hexdocs.pm/elixir/sigils.html
| Miner49er wrote:
| Ah yeah, you're right.
| psychoslave wrote:
| Great! But, what is a nifs, please? :'D
| jameskilton wrote:
| Natively Implemented Functions
|
| https://www.erlang.org/doc/system/nif.html
| kuon wrote:
| I use zig a lot in elixir nif, for things like audio and video
| processing, it works great. But I do not use zigler as I prefer
| the code to live in their own codebases. But zigler is really
| nice and it provides an easy way to do computational heavy tasks
| in elixir.
| kansi wrote:
| > I use zig a lot in elixir nif, for things like audio and
| video processing
|
| Sounds interesting, is it open source? I am interested in
| seeing how the code layout looks like when mixing Zig and
| Elixir
| kuon wrote:
| I don't have open source code base to share but here it how
| it looks like: // the_nif.zig
| fn init_imp( env: ?*erl.ErlNifEnv,
| argc: c_int, argv: [*c]const
| erl.ERL_NIF_TERM, ) !erl.ERL_NIF_TERM {
| if (argc != 0) { return error.BadArg;
| } return try helpers.make("Hello
| world"); } export fn
| media_tools_init( env: ?*erl.ErlNifEnv,
| argc: c_int, argv: [*c]const
| erl.ERL_NIF_TERM, ) erl.ERL_NIF_TERM {
| return init_imp(env, argc, argv) catch |err|
| return helpers.make_error(env, err); }
| var funcs = [_]erl.ErlNifFunc{ erl.ErlNifFunc{
| .name = "init", .arity = 1,
| .fptr = media_tools_init, .flags =
| erl.ERL_NIF_DIRTY_JOB_CPU_BOUND, } };
| var entry = erl.ErlNifEntry{ .major =
| erl.ERL_NIF_MAJOR_VERSION, .minor =
| erl.ERL_NIF_MINOR_VERSION, .name =
| "Elixir.MediaTools.Stream", .num_of_funcs =
| funcs.len, .funcs = &funcs,
| .load = load, .reload = null,
| .upgrade = null, .unload = null,
| .vm_variant = "beam.vanilla", .options = 0,
| .sizeof_ErlNifResourceTypeInit =
| @sizeOf(erl.ErlNifResourceTypeInit),
| .min_erts = "erts-10.4", };
| export fn nif_init() *erl.ErlNifEntry {
| return &entry; } #
| the_exlixir_file.ex assert "Hello world" ==
| MediaTools.Stream.init()
|
| The "helpers" library is used to convert types to and from
| erlang, I plan on open sourcing it but it is not ready now.
| In the above example, the code is explicit but "entry" can be
| created with an helper comptime function. erl is simply the
| erl_nif.h header converted by zig translate-c.
|
| I wrote a piece back in 2022, but things evolved a lot since
| then: https://www.kuon.ch/post/2022-11-26-zig-nif/
| kansi wrote:
| Thanks for sharing the post, it was intriguing. The
| detailed comments mentioned in `main.zig` and `build.zig`
| towards the end helped a lot.
| ihumanable wrote:
| For anyone mystified about what a NIF is that doesn't want to go
| read the docs.
|
| The BEAM VM (which is the thing that runs erlang / elixir / gleam
| / etc) has 3 flavors of functions.
|
| - BIFs - Built-in functions, these are written in C and ship with
| the VM
|
| - NIFs - Natively implemented functions, these are written in any
| language that can speak the NIF ABI that BEAM exposes and allows
| you to provide a function that looks like a built-in function but
| that you build yourself.
|
| - User - User functions are written in the language that's
| running on BEAM, so if you write a function in erlang or elixir,
| that's a user function.
|
| NIFs allow you to drop down into a lower level language and
| extend the VM. Originally most NIFs were written in C, but now a
| lot more languages have built out nice facilities for writing
| NIFs. Rust has Rustler and Zig now has Zigler, although people
| have been writing zig nifs for a while without zigler and I'm
| sure people wrote rust nifs without rustler.
| hinkley wrote:
| It's important to note that while Erlang has protections
| against user code crashing an Erlang process and recovering, a
| faulty NIF can take down the entire virtual machine.
| alberth wrote:
| Hence why Rustler is of so much interest since it provides
| more protections against this happening.
|
| Discord is a big Erlang + Rustler user.
| drawnwren wrote:
| Is any of this code open source? As an outsider, I'm kind
| of at a loss for why anyone wants this or what you kids are
| doing over there and how offended I should be by it.
| jhgg wrote:
| https://github.com/discord/sorted_set_nif
| alberth wrote:
| Do you mean Rustler?
|
| Yes, it's Apache 2.0
|
| https://github.com/rusterlium/rustler
| depr wrote:
| Are they really? Their projects don't look so active
| sodapopcan wrote:
| It's pretty common in the Elixir ecosystem for these
| types of libraries to not change very much. Elixir itself
| doesn't change too much so these libraries stay solid
| without needing frequent updates. It doesn't mean people
| aren't using them. Some libraries even put disclaimers
| that they are actively maintained even if they haven't
| seen an update in a long time. It's something that takes
| some getting used to for some people (including myself at
| one point).
| andy_ppp wrote:
| Yeah I was trying to explain this to another developer
| that packages end up being "finished" eventually and seem
| to continue to work exceptionally well without updates
| for a really long time.
|
| Something about immutability and the structure of Elixir
| leads to surprisingly few bugs.
| sbuttgereit wrote:
| Yep. This is one reason I choose Elixir for a project.
| For a variety of use cases, long term stability is a big
| plus.
| photonthug wrote:
| > It's pretty common in the Elixir ecosystem for these
| types of libraries to not change very much.
|
| This is kind of fascinating and seems worthy of more
| detailed study. I'm sure almost anything looks stable
| compared to javascript/python ecosystems, but would be
| interesting to see how other ecosystems with venerable
| old web-frameworks or solid old compression libraries
| compare. But on further reflection.. language metrics
| like "popularity" are also in danger of just quantifying
| the churn that it takes to keep working stuff working.
| You can't even measure strictly new projects and hope
| that helps, because new projects may be a reaction to
| perceived need to replace other stuff that's annoyingly
| unstable over periods of 5-10 years, etc.
|
| Some churn is introduced by trying to keep up with a
| changing language, standard lib, or other dependencies,
| but some is just adding features forever or endlessly
| refactoring aesthetics under different management. Makes
| me wish for a project badge to indicate a commitment like
| finished-except-for-bugfixes.
| alberth wrote:
| Elixir itself is "feature complete" as of 2019 (5-years
| now).
|
| https://elixir-
| lang.org/blog/2019/06/24/elixir-v1-9-0-releas...
|
| It does get the occasional updates, but it's mainly
| related to developer tooling than language enhancements.
| ihumanable wrote:
| I wrote sorted_set_nif, the lack of activity isn't a lack
| of care about the library but more just a reflection that
| the library is done.
|
| With data structures that have some definite behavior
| unless someone finds a defect there isn't going to be
| much activity.
| johnisgood wrote:
| What kind of protections as opposed to Zigler?
| alberth wrote:
| Rust comes with memory safety.
|
| It's one less potential cause that might bring down the
| entire Erlang VM.
| hinkley wrote:
| SIGSEGV is a pretty common failure mode alright.
| kristoff_it wrote:
| There's a series of things that a NIF must do to be a good
| citizen. Not crashing is a big one, but also not starving the
| VM by never yielding (in case the NIF is long-running) is
| important, plus a few secondary things like using the BEAM
| allocator so that tooling that monitors memory consumption
| can see resources consumed by the NIF.
|
| The creator of Zigler has a talk from ElixirConf 2021 on how
| he made Zig NIFs behave nicely:
|
| https://www.youtube.com/watch?v=lDfjdGva3NE
| jlkjfuwnjalfw wrote:
| Don't be like me and do a 20ms page fault in a NIF
| bmitc wrote:
| It's also important to point out ports, because as you mention,
| NIFs are a way to integrate external code. But as someone else
| points out, NIFs can crash the entire BEAM VM. Ports are a
| safer way to integrate external code because they are just
| another BEAM process that talks to an external program. If that
| program crashes, then the port process crashes just like any
| other BEAM process but it won't crash the entire BEAM VM.
| gioazzi wrote:
| And then there are port drivers which are the worst of both
| worlds! Can crash the BEAM and need much more ceremony than
| NIF to set up but they're pretty nice to do in Zig[1] as well
|
| [1]: https://github.com/borgoat/er_zig_driver
| bmitc wrote:
| That's true. Haha!
|
| There's another option and that's setting up an Erlang node
| in the other language. The Erlang term format is relatively
| straightforward. But I'm honestly not sure of the benefit
| of a node versus just using a port.
| throwawaymaths wrote:
| Node:
|
| - can "easily" send beam terms back and forth
|
| - if you want it to be os-supervised separately (systemd,
| kubernetes, e.g.)
|
| - pain in the ass
|
| Port:
|
| - easy
|
| - usually the only choice if you're not the software
| author
|
| - really only communicates via stdio bytestreams
|
| - risk of zombies if... Iirc the stdout is not closed
| properly?
|
| - kind of crazy how it works, Erlang VM spawns a separate
| process as a middleman
| abrookewood wrote:
| Why would anyone use a NIF instead of a Port then?
| Cyph0n wrote:
| IPC/shared memory overhead?
| derefr wrote:
| Does anyone actually enjoy using these systems that encourage you
| to embed programming-language X code in programming-language Y
| heredocs?
|
| I always find actually _doing_ that -- and then maintaining the
| results over time -- to be quite painful: you don 't get syntax
| highlighting inside the string; you can no longer search your
| worktree reliably using extension-based filtering; etc.
|
| I personally find the workflow much more sane if/when you just
| have a separate file (e.g. `foo.zig`) for the guest-language
| code, and then your host-language code references it.
| systems wrote:
| I initially agree
|
| But, if all you do is write elixir wrappers around the zig
| function, to completely hide the foreign language functions,
| keeping both the wrapper and implementation in the same file,
| even if two different languages doesn't seem horrible, but
| again, keeping them in two file doesn't seem like a huge
| difference too
|
| I think its really a matter of taste, both options viable
| toast0 wrote:
| I've done some assembly in C, and for big functions, yeah, I
| want it in its own file, but smaller things often make sense to
| embed. I'm not sure if I'd like my nif code embedded into my
| erl files (assuming this works for Erlang as well), but it
| could conceivably make the nasty bit of boilerplate around
| ERL_NIF_INIT in the NIF (which I have to do in C anyway) and
| exit(nif_library_not_loaded) in the erl go away, which would be
| nice.
|
| It's certainly possible to get syntax highlighting on the
| embedded code, but you'll need to work with your syntax
| highlighter; it certainly helps if you're not the only person
| using it.
|
| But then again, I worked without syntax highlighting for years,
| so I'm happy when it works, but when it doesn't, I'm ok with
| that too.
| harrisi wrote:
| Syntax highlighting here can work correctly, actually.
|
| Also, I'm not sure why it's not better documented in Zigler,
| but you can also write the code in a separate file just fine.
| harrisi wrote:
| Zig is also used in an excellent way by burrito[0]. I've also
| used zig for compiling NIFs written in C/C++/Objective-C, since
| `zig cc` makes cross-compiling much nicer.
|
| I wish zig got more use and attention in the Erlang ecosystem,
| but rustler seems more popular.
| kansi wrote:
| [0] https://github.com/burrito-elixir/burrito
| nine_k wrote:
| (Yo dawg, we put a niche language into a niche language so
| that...)
|
| I wonder if the Zig code can be _not_ written inline, as an
| option. With anything larger than a few lines, I 'd want syntax
| highlighting, LSP support, navigation, etc. It's easier to
| achieve with one language per file.
___________________________________________________________________
(page generated 2024-10-24 23:00 UTC)