[HN Gopher] Porting SBCL to the Nintendo Switch
___________________________________________________________________
Porting SBCL to the Nintendo Switch
Author : todsacerdoti
Score : 383 points
Date : 2024-09-13 12:53 UTC (1 days ago)
(HTM) web link (reader.tymoon.eu)
(TXT) w3m dump (reader.tymoon.eu)
| colingw wrote:
| I have being using Trial[1] for the past few weeks to test out
| game development in Common Lisp, and have been having a great
| time. Being able to alter (almost) all aspects of your game while
| it's running is a blessing.
|
| I hope this port succeeds.
|
| [1]: https://github.com/Shirakumo/trial
| sinker wrote:
| Lisp languages seem well-suited for building games. The ability
| to evaluate code interactively without recompilation is a huge
| deal for feature building, incremental development, and bug-
| fixing. Retaining application state between code changes seems
| like it would be incredibly useful. Common Lisp also appears to
| be a much faster language than I would have blindly assumed.
|
| The main downside for me (in general, not just for game
| programming) is the clunkiness in using data structures - maps
| especially. But the tradeoff seems worth it.
| pjmlp wrote:
| There are 1980's papers about Lisp compilers competing with
| Fortran compilers, unfortunately with the AI Winter, and the
| high costs of such systems, people lost sight of it.
| coliveira wrote:
| Well, I imagine at the time they had some LISP
| implementations that were very well tuned for specific high
| end machines, which essentially duplicated Fortran
| functionality. This is difficult to do for general purpose
| Lisps like SBCL. It was also probably very expensive.
| pjmlp wrote:
| What is difficult is having Apple, Google, IBM,
| Microsoft, Intel, NVidia, AMD,.... compiler teams budget.
| lispm wrote:
| One of the downsides is that implementations like SBCL have a
| deep integration and need things like a well performing GC
| implementation - to get this running on specialized game
| hardware is challenging. The article describes that. Getting
| over the hurdle of the low-level integration is difficult.
| The reward comes, when one gets to the point, where the rapid
| incremental development cycles of Common Lisp, even with
| connected devices, kicks in.
|
| For the old historic Naughty Dog use case, it was a
| development system written in Common Lisp on an SGI and a C++
| runtime with low-level Scheme code on the Playstation.
|
| > Common Lisp also appears to be a much faster language than
| I would have blindly assumed.
|
| There are two modes:
|
| 1) fast optimized code which allows for some low-level stuff
| to stay with Common Lisp
|
| 2) unoptimized, but natively compiled code, which enables
| safe (-> the runtime does not crash) interactive and
| incremental development -> this mode is where much of the
| software can run nowadays and which is still "fast enough"
| for many use cases
| koito17 wrote:
| > The ability to evaluate code interactively without
| recompilation
|
| SBCL and other implementations compile code to machine code
| _then_ execute it. That is to say, when a form is submitted
| to the REPL, the form is not interpreted, but first compiled
| then executed. The reason execution finishes quickly is
| because compilation finishes quickly.
|
| There are some implementations, like CCL, with a special
| interpreter mode exclusively for REPL-usage.[1] However, at
| least SBCL and ECL will compile code, not interpret.
|
| [1] https://github.com/Clozure/ccl/blob/v1.13/level-1/l1-read
| loo...
| Shinmera wrote:
| I specifically talk about the fast evaluator for SBCL. But
| even without that contrib, SBCL does have another evaluator
| as well that's used in very specific circumstances.
| taeric wrote:
| I think a lot of this is confusion between online versus
| batch compilation? Most of us have only ever seen/used
| batch compilation. To that end, many people assume that JIT
| in an interpreter is how online compilation is done.
|
| I probably am more guilty of that than I should be.
| igouy wrote:
| > online compilation
|
| ? incremental compilation
| akira2501 wrote:
| > the form is not interpreted, but first compiled then
| executed
|
| That's TempleOS technology right there.
| Filligree wrote:
| Other way around.
| darby_nine wrote:
| Do either CCL or SBCL have any kind of partial evaluation
| or tracing compilation?
| Jach wrote:
| If that's your main downside, that's pretty good, since
| clunkiness is in many ways fixable. Personally with standard
| CL I like to use property lists with keywords, so a "map
| literal" is just (list :a 3 :b 'other). It's fine when the
| map is small. The getter is just getf, setting is the usual
| setf around the getter. There's a cute way to loop by #'cddr
| for a key-and-value loop, though Alexandria (a very common
| utility library) has some useful utils for
| looping/removing/converting plists as well.
|
| If typing out "(list ...)" is annoying, it's a few lines of
| code to let you type {:a 3 :b 4} instead, like Clojure. And
| the result of that can be a plist, or a hash table, or again
| like Clojure one of the handful of immutable map structures
| available. You can also easily make the native hash tables
| print themselves out with the curly bracket syntax.
|
| (On the speed front, you might be amused by
| https://renato.athaydes.com/posts/how-to-write-slow-rust-
| cod... But separately, when you want to speed up Lisp (with
| SBCL) even more than default, it's rather fun to be able to
| run disassemble on your function and see what it's doing at
| the assembly level, and turn up optimization hints and have
| the compiler start telling you (even on the individual
| function level) about where it has to use e.g. generic
| addition instead of a faster assembly instruction because it
| can't prove type info and you'll have to tell it/fix your
| code. It can tell you about dead code it removed. You can
| define stack-allocation if needed. Simple benchmarking that
| also includes processor cycles and memory allocated is
| available immediately with the built-in time macro...)
| phlakaton wrote:
| The cost of a macro is not measured in lines of code. It's
| measured in things like adoption, clarity, and
| debuggability.
| spsesk117 wrote:
| Thanks to the author for the fascinating and detailed write up.
| It feels like a lot of the time this level of detail around the
| specifics of 'blessed' (not homebrew) console porting are only
| revealed years after the end of the consoles lifetime.
|
| As an aside, reading about this kind of deeply interesting work
| always makes me envious when I think about the rote software I
| spend all day writing :)
| cschep wrote:
| well said!
|
| as I was just sitting down to another day of ruby on rails
| (that I am grateful for!) I was thinking.. I wonder what
| hobby/open source projects could use some of my attention
| later..
|
| .. what projects my attention could use later .. :D
| whizzter wrote:
| At least back when I was working these "blessed" tools were
| usually a tad hacked together, modern homebrew toolchains for
| many older platforms are better except for debugging support
| (since the devkits for the machines usually had better hooks
| available but also avoiding the entire GDB focus).
|
| Having been in both worlds, i'm not entirely sure there's that
| much to be envious of.
| fouric wrote:
| This is super neat - SBCL is an awesome language implementation,
| and I've always wanted to do CL development for a "real" game
| console.
|
| I'm also surprised (in a good way) that Shinmera is working on
| this - I've seen him a few times before on #lispgames and in the
| Lisp Discord, and I didn't know that he was into this kind of
| low-level development. I've looked at the guts of SBCL briefly
| and was frightened away, so kudos to him.
|
| I wonder if SBCL (+ threading/SDL2) works on the Raspberry Pi
| now...
| f1shy wrote:
| - Is it not her?
| anisoco wrote:
| It is
|
| https://reader.tymoon.eu/article/436
| f1shy wrote:
| Oh I did not know that she transitioned. I just remebered
| it was the author of portacle, that was a woman.
|
| What a world of pain must be to have to go through it.
|
| Kudos to her. Also what she does for lisp is amazing.
| Shinmera wrote:
| I'm not doing the SBCL parts, that's all Charles' work that I
| hired him for. My work is the portability bits that Trial
| relies on to do whatever and the general build architecture for
| this, along with the initial runtime stubbing.
|
| And, as mentioned, *her :)
| cellularmitosis wrote:
| Holy cow, Kandria looks amazing. Is it also developed using
| Trial? https://www.youtube.com/watch?v=usc0Znm-gbA
| Shinmera wrote:
| Yes, it's also open source:
| https://github.com/shirakumo/kandria
|
| My current unannounced project is a lot more ambitious
| still, being a 3D hack & slash action game. I post updates
| about that on the Patreon if you're interested.
| ducktective wrote:
| I wish the likes of Nintendo and Sony themselves finance such
| efforts. I mean it's one another way to create games (IP) for
| your console, what possibly could be the downside of starting
| something similar to Github Accelerator for your platform?
| jsheard wrote:
| Because it's well established that game developers can and will
| jump through whatever hoops the platform holder demands at
| their own expense, they don't have the leverage to be picky
| about the technical details when deciding which platforms to
| ship on. Nintendo doesn't need to create new incentives to
| release on the Switch when they already have the biggest
| incentive of all: 140+ million units sold, and a high attach
| rate.
|
| At least there isn't as much hoop jumping as there used to be,
| since the systems have all converged on using commodity CPU and
| GPU architectures with at most minor embellishments.
| Arelius wrote:
| Yeah, also whatever they would build, the other platform
| vendors won't choose the same thing, and it wont be the exact
| variant of lisp or w/e that even the few nice developers
| would want.
|
| I wish vendors would be just more supportive of different
| llvm tool chains. Rust isn't even well supported.
| pjmlp wrote:
| Rust isn't even available on Android NDK, even though is
| now used on Android.
|
| Same applies to Rust on Windows, and whatever Microsoft is
| doing, windows-rs isn't that great, and after what the team
| did with C++/WinRT I don't have high expectations.
|
| So consoles have even less reasons to support Rust.
| pjmlp wrote:
| They did in the past and the only thing people did was to port
| MAME and other emulators, or copies from old 8 and 16 bit days.
|
| That is why we cannot have nice things.
| Retr0id wrote:
| > The answer to that is that while I would desperately like to
| share it all publicly, the NDA prevents us from doing so.
|
| I'm curious what the rationale here was for using the official
| SDK, rather than the unencumbered "homebrew" ones[0].
|
| As a complete guess, maybe Nintendo doesn't let you officially
| publish games built using 3rd party SDKs?
|
| [0]
| https://switchbrew.org/wiki/Setting_up_Development_Environme...
| Shinmera wrote:
| You cannot publish games with homebrew, it has to use the
| official SDK. Besides that, almost nobody has a jailbroken
| Switch, so it would make it extremely hard to play any games on
| anything but an emulator.
| Retr0id wrote:
| > You cannot publish games with homebrew, it has to use the
| official SDK
|
| This doesn't surprise me much, but does Nintendo state this
| explicitly anywhere public?
|
| If Nintendo chose to sign an application developed using a
| 3rd party toolchain, there's no _technical_ reason why it
| couldn 't run on retail consoles.
| Arelius wrote:
| Yeah, maybe. But also the official SDKs are pretty good and
| you get support from Nintendo. It seems like a pretty big
| risk to use an unsupported SDK... for what benefit?
| Retr0id wrote:
| I don't have access to Nintendo's SDK so I can't compare
| directly, but the article cites an inability to map
| executable pages. libnx supports this (but of course,
| this is moot if Nintendo wouldn't let you ship it). But
| the main benefit is being able to talk about and share
| your work without worrying about violating an NDA.
|
| https://switchbrew.github.io/libnx/jit_8h.html
|
| https://switchbrew.org/wiki/JIT_services
| Shinmera wrote:
| The OS can do it, and some Nintendo titles on the Switch
| do use this capability, but I have talked to Nintendo
| directly about using it, and it's a hard No. I can't even
| use the JIT feature purely for dev.
| nightpool wrote:
| Really? What is the justification for allowing Nintendo
| titles to use it but not third parties? Security
| concerns?
| Shinmera wrote:
| That's what they claim, but ultimately it's their thing,
| so they do with it whatever they like.
| favorited wrote:
| > almost nobody has a jailbroken Switch
|
| This isn't really my scene so I don't know the details, but I
| remember reading that the first 10+ million Switches produced
| have an unpatchable bootloader exploit. I'm sure you're
| correct that almost nobody actually has a hacked console, but
| my understanding is that they're readily available for people
| who want one.
| ArneBab wrote:
| You don't let your kids jailbreak their Switch. Because it's
| a damn online system, so any leaked info and Nintendo can
| brick the Switch. And their game states are far too valuable
| for the kids for that.
| opan wrote:
| Trey can ban the Switch, but offline games will continue to
| work. Also the account doesn't get banned, so you can buy a
| new one. (Speaking from experience, unfortunately) You can
| still play the new Zelda, just can't play Splatoon, Mario
| Kart, or Smash online then on the banned Switch. It's
| possible but arduous to rescue the saves off the banned
| Switch if you have access to a second modded Switch that is
| not banned (also speaking from experience) and use homebrew
| to back up and restore your saves, then launch them all
| from sysMMC with legitimately owned versions of those games
| and let the cloud save feature kick in. Animal Crossing has
| a separate dedicated save tool.
|
| Block Nintendo servers, disable auto updates, use separate
| sysMMC and emuMMC with no unauthorized games or DLC run on
| the sysMMC. If you follow the main guide everyone uses now,
| it's pretty safe. But updating becomes a more difficult and
| manual process. Have to grab a zip of the new firmware from
| the 'net on your PC and copy it to the SD card to be
| installed via a homebrew method. Installing games, game
| updates, and DLC is similarly manual. It's not like the
| PS3, Vita, and 3DS(?) where you can pull it all off of
| official servers easily.
|
| Oh yeah, and we're stuck with a "tethered jailbreak",
| that's perhaps the worst part. Any time you turn off the
| hacked Switch it needs to be sent a payload from your PC or
| phone to boot up again then.
|
| Whether it's all worth it depends on your needs I suppose.
| You could get a bunch of tournament setups going with Smash
| (or another fighting game) + all DLC for your LAN party and
| save a bit of money. You can try out new singleplayer games
| before buying them physically. You can mod games and run
| emulators. Honestly the Switch scene seems largely less
| cool than what we had with the 3DS or Wii (Wii U was a
| little disappointing as well). I barely touch my Switch(es)
| since getting a Steam Deck.
| noobermin wrote:
| This is what I come to HN for. Kudos to OP and their colleague. I
| know it's impossible but what a blessing it would be if Nintendo
| could be a little more open about their system.
| whitten wrote:
| I probably missed it, but since the switch doesn't have a
| keyboard how do you handle text input ?
| jsheard wrote:
| The Switch can support a USB keyboard, which would be the nice
| way to do it. There's already a fair number of officially
| published games with keyboard and/or mouse support, including a
| couple of programming ones. It has an on-screen keyboard too of
| course but you wouldn't want to rely on that more than
| absolutely necessary.
| Arelius wrote:
| Over the network most likely?
| Shinmera wrote:
| It does have a HID keyboard API, but typically you're expected
| to present an on-screen keyboard.
| not_your_vase wrote:
| Somewhat offtopic, just flashed through my mind: you know what
| would be amazing and absolutely useless at the same time?
|
| Porting Yuzu to Nintendo Switch
| laidoffamazon wrote:
| This has been done, and it's been possible to run Switch on
| Switch for several months [0] (about 39 minutes into the
| video).
|
| [0] https://youtu.be/H1gveQUBIKk
| not_your_vase wrote:
| Shoot, it was a suspiciously genius idea* for a Friday 13th.
|
| *: Compared to my usual ideas
| jfim wrote:
| Since I had to look it up, Yuzu is a Nintendo switch emulator.
| sleepycatgirl wrote:
| Her work is just amazing. And makes me incredibly happy, as I
| like to write some CL here and there.
| Shinmera wrote:
| Aw, thank you. Happy hacking!
| trenchgun wrote:
| Does anybody know what is the status of Trial on Mac OS?
| Specifically Apple Silicon
| Shinmera wrote:
| I don't have a mac, let alone a silicon one, so I can't test on
| it (I also have no patience for Apple's BS). However, it should
| work. At least SBCL itself runs, and Trial is mostly portable
| code, so it should, too, modulo some regressions.
| nanna wrote:
| I've just bought Kandria. I'm not much of a game player so I
| probably won't get much play out of it, but Shinmera is clearly
| pushing the bounds of the Lisp world, and that's something to
| support.
| hoten wrote:
| b/c it isn't described anywhere...
|
| SBCL - "Steel Bank Common Lisp"
|
| > Steel Bank Common Lisp (SBCL) is a high performance Common Lisp
| compiler. It is open source / free software, with a permissive
| license. In addition to the compiler and runtime system for ANSI
| Common Lisp, it provides an interactive environment including a
| debugger, a statistical profiler, a code coverage tool, and many
| other extensions.
|
| https://www.sbcl.org/
| varjag wrote:
| "Steel Bank" in turn is a homage to "Carnegie Mellon" of CMUCL.
| ArneBab wrote:
| According to the Benchmarks Game, SBCL is roughly as fast as
| Node.
|
| https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
| tmtvl wrote:
| See also https://github.com/attractivechaos/plb2 ...where I
| provided the SBCL solutions, so there's probably still a
| significant chunk of performance to be squeezed out.
| drmeister wrote:
| Awe-inspiring work Shinmera and Charles.
| dasyatidprime wrote:
| How's CL's GC performance for games nowadays? I've been slightly
| eyeing the upcoming Autumn Lisp Game Jam myself, but last I
| checked all the major libre CL impls, including SBCL, still used
| a full stop-the-world collector, which feels like a recipe for
| latency spikes. I saw flashes of stuff on sbcl-devel about
| someone working on a lower-latency one, but I don't know whether
| it got anywhere.
| vindarel wrote:
| see this detailed report:
| https://raw.githubusercontent.com/Shinmera/talks/master/els2...
|
| > Overall we have needed to do surprisingly little actual
| performance analysis and optimisation work to make Kandria run
| well.
| superdisk wrote:
| Great article. One question I had, not to diminish this hard
| work, is why not use a different implementation like ECL which is
| pretty portable already and can compile to static C code which
| can just be compiled traditionally for the target? I've been
| doing that for a Wasm + SDL2 game in Lisp and it's been
| (relatively) straightforward. I suppose performance might be a
| issue.
| Shinmera wrote:
| Because, as you guessed, no implementation other than SBCL
| comes close to the performance needed.
| aktau wrote:
| Related: https://opengoal.dev.
|
| Context: Naughty Dog used a custom Lisp-alike (GOAL) to build the
| Jak & Daxter series on PS2. They left enough debugging
| information in that it was possible to reverse engineer. The
| OpenGOAL project has done so, and these games can now be run on
| all platforms that their GOAL compiler gets ported to (x86 for
| now AFAIK). Would be cool to port this to the Switch.
___________________________________________________________________
(page generated 2024-09-14 23:01 UTC)