[HN Gopher] ML in Go with a Python Sidecar
___________________________________________________________________
ML in Go with a Python Sidecar
Author : zdw
Score : 57 points
Date : 2024-11-11 17:44 UTC (6 days ago)
(HTM) web link (eli.thegreenplace.net)
(TXT) w3m dump (eli.thegreenplace.net)
| calderwoodra wrote:
| I was surprised you chose http for your IPC - I was expecting
| there to be a more handy tool that Python could expose and Go
| could leverage without needing to keep a second process
| constantly running.
| herval wrote:
| Keeping models in memory is more efficient than constantly
| loading/unloading them, so a process that keeps running is the
| way to go
| daniel-thompson wrote:
| > Completely bespoke models are typically trained in Python using
| tools like TensorFlow, JAX or PyTorch that don't have real non-
| Python alternatives
|
| The article outlines some interesting ways to evade this problem.
| What's the latest thinking on robustly addressing it, e.g. are
| there any approaches for executing inference on a tf or pytorch
| model from within a golang process, no sidecar required?
| kevmo314 wrote:
| For Go specifically, there are some libraries like Gorgonia
| (https://github.com/gorgonia/gorgonia) that can do inference.
|
| Practically speaking though, the rate at which models change is
| so fast that if you opt to go this route, you'll perpetually be
| lagging behind the state of the art by just a bit. Either
| you'll be the one implementing the latest improvements or be
| waiting for the framework to catch up. This is the real value
| of the sidecar approach: when a new technique comes out (like
| speculative decoding, for example) you don't need to
| reimplement it in Go but instead can use the implementation
| that most other python users will use.
| richardjennings wrote:
| It is possible to include CPython in a CGO program - allowing
| Python to be executed from within the CGO process directly.
| This comes with some complexities - GIL and thread safety in Go
| routines, complexity of cross-compiling between architectures,
| overhead in copying values across the FFI, limitations of
| integrating as a Go module. I am hoping to see a CGO GIL'less
| Python integration show up here at some point that has all the
| answers.
| uriah wrote:
| These frameworks are C++ under the hood. A far as I know (not
| too experienced with go) you can use cgo to call any C++ code.
| So you should be able to serialize the model (torchscript) then
| run it with libtorch. Tensorflow also similarly has a C++ api
| neomantra wrote:
| Perhaps check out GoMLX ("an Accelerated ML and Math
| Framework", there's a lot of scaffolding and it JITs to various
| backends. Related to that project, I sometimes use GoNB in
| VSCode, which is Golang notebooks [2].
|
| [1] https://github.com/gomlx/gomlx
|
| [2] https://github.com/janpfeifer/gonb
| mountainriver wrote:
| I wrote https://github.com/aunum/gold as an attempt to try ML
| with Go.
|
| I came to the opinion that it's just not worth it. This isn't
| what Go was designed for and it will likely never be a good
| language to interface with ML.
|
| The Go to C FFI is too slow for native Cuda ops and Go doesn't
| have a good bridge to Python.
|
| Rust is a much much better option in these scenarios
___________________________________________________________________
(page generated 2024-11-17 23:00 UTC)