[HN Gopher] 3D + 2D: Testing out my cross-platform WASM graphics...
___________________________________________________________________
3D + 2D: Testing out my cross-platform WASM graphics engine
I used to work at Adobe on the infrastructure powering big
applications like Photoshop and Acrobat. One of our worst headaches
was making these really powerful codebases work on desktop, web,
mobile, and the cloud without having to completely rewrite them.
For example, to get Lightroom and Photoshop working on the web we
took a winding path through JavaScript, Google's PNaCl, asm.js, and
finally WebAssembly, all while having to rethink our GPU
architecture around these devices. We even had to get single-
threaded builds working and rebuild the UI around Web Components.
Today the web builds work great, but it was a decade-long journey
to get there! The graphics stack continues to be one of the
biggest bottlenecks in portability. One day I realized that
WebAssembly (Wasm) actually held the solution to the madness. It's
runnable anywhere, embeddable into anything, and performant enough
for real-time graphics. So I quit my job and dove into the
adventure of creating a portable, embeddable WASM-based graphics
framework from the ground up: high-level enough for app developers
to easily make whatever graphics they want, and low-level enough to
take full advantage of the GPU and everything else needed for a
high-performance application. I call it Renderlet to emphasize the
embeddable aspect -- you can make self-contained graphics modules
that do just what you want, connect them together, and make them
run _on_ anything or _in_ anything with trivial interop. If you
think of how Unity made it easy for devs to build cross-platform
games, the idea is to do the same thing for all visual
applications. Somewhere along the way I got into YC as a solo
founder (!) but mostly I've been heads-down building this thing for
the last 6 months. It's not _quite_ ready for an open alpha
release, but it's close--close enough that I'm ready to write about
it, show it off, and start getting feedback. This is the thing I
dreamed of as an application developer, and I want to know what you
think! When Rive open-sourced their 2D vector engine and made a
splash on HN a couple weeks ago
(https://news.ycombinator.com/item?id=39766893), I was intrigued.
Rive's renderer is built as a higher-level 2D API similar to SVG,
whereas the Wander renderer (the open-source runtime part of
Renderlet) exposes a lower-level 3D API over the GPU. Could
Renderlet use its GPU backend to run the Rive Renderer library,
enabling any 3D app to have a 2D vector backend? Yes it can - I
implemented it! You can see it working here:
https://vimeo.com/929416955 and there's a deep technical dive here:
https://github.com/renderlet/wander/wiki/Using-renderlet-wit....
The code for my runtime Wasm Renderer (a.k.a. Wander) is here:
https://github.com/renderlet/wander. I'll come back and do a
proper Show HN or Launch HN when the compiler is ready for anyone
to use and I have the integration working on all platforms, but I
hope this is interesting enough to take a look at now. I want to
hear what you think of this!
Author : seanisom
Score : 194 points
Date : 2024-04-02 16:44 UTC (6 hours ago)
| dsp_person wrote:
| What do you think of sokol in comparison?
| seanisom wrote:
| sokol is great - I think of it as more of an "STB for apps".
| With renderlet/wander the goal is more to express graphics
| using higher level constructs and automatically generate the
| runtime code behind it. For example, with the Wasm build of
| sokol you could build a canvas-style app that directly runs in
| a browser, whereas with renderlet you can build a function that
| can parametrically render grids that can be run in (any) app
| like that.
| flohofwoe wrote:
| TBH I would love an extended WASI standard with 'media apis'
| (window, 3D, audio, input) to run sokol code compiled to WASM
| in without having to compile/distribute per-platform native
| apps.
|
| Deno seems to work on that idea [0], but having a WASI like
| standard would be better of course.
|
| [0] https://github.com/deno-windowing
|
| PS: How much work was it to "port" the Rive renderer? Would
| be great to see a blog post or similar about how you
| approached that and about any difficulties on the way :)
| mendyberger wrote:
| This? https://github.com/WebAssembly/wasi-webgpu
| flohofwoe wrote:
| Yeah, but the 'wasi-canvas' part is really important to
| get something on screen (as opposed to "just" using
| WebGPU for compute-tasks).
|
| Also a simple wasi-audio API is needed (preferrably
| something less overengineered than WebAudio, just a
| simple sample-streaming API would be perfect).
| seanisom wrote:
| +1 for Audio and lots of other APIs necessary to make
| WASI more like a true OS. With Preview 2 / Component
| model, hoping the pace of contributions rapidly
| increases.
| seanisom wrote:
| Also, with the solid foundation and simple API footprint
| you've built for APIs like sokol_audio, would be
| interesting to see if they could be expressed in WIT and
| used as a basis for something like a wasi-audio.
| boomskats wrote:
| Are you involved at all with the Bytecode Alliance or the
| decisions around the WASI proposals/standards? It feels
| like your take on these things would be super valuable
| given all the work you've done. They're a very open
| minded group.
| seanisom wrote:
| Yes! wasi-webgpu is coming, as well as more APIs with wasi-
| gfx.
|
| Getting rive-renderer working was not hard because in the
| demo its running on the host side, and not in Wasm yet,
| although compiling for Windows/DX11 took some minor
| changes. Getting it fully working in Wasm outside of the
| browser looks to be non-trivial, but doable, but will
| likely require upstream changes.
| dartos wrote:
| So kind of like a graphics transpiler?
| seanisom wrote:
| Yes, that's a good mental model. Input - high-level
| description of graphics, output low-level graphics code
| running in Wasm.
| billconan wrote:
| do I still need to write platform specific shaders using this
| library?
| seanisom wrote:
| The long-term goal is no - wander should handle shader
| compilation automatically. In the current version, the host app
| has to attach a shader using the host's shader API to render
| arbitrary geometry and textures.
|
| Was looking at several different approaches to this, one of
| which would be cross-compiling wasm to spir-v. Most likely will
| expose a higher-level shader API (think shadertoy) and have
| wander compile to the platform backend. Also will be able to
| run WGSL shaders directly through Wasm with wasi-gfx support.
| billconan wrote:
| or maybe support https://github.com/shader-slang/slang
| seanisom wrote:
| Yes! Will look into that
| mendyberger wrote:
| Cool project!
|
| Looks like it supports geometry and textures now, any plans to
| support shaders?
| seanisom wrote:
| Yes! There are a few different approaches to making that work -
| one would be to have an intermediate shader representation
| generated from Wasm compile to native platform shaders on the
| host graphics API. Longer-term, will likely expose WebGPU WGSL
| shaders to Wasm directly.
| mendyberger wrote:
| Readme says it's a C++ library. Any plans to support higher level
| languages such as Go or even Python?
| seanisom wrote:
| Yes! It's kind of a pain to build now, so will probably shift
| to shipping as a .so/dll with a raw C api in a future version.
| With that, should be easy to generate bindings for any language
| - the host API footprint is minimal.
| billconan wrote:
| I don't follow this part. if the lib is shipped as .so/dll,
| how can it be compiled into wasm?
| seanisom wrote:
| There is the host API - wander, which contains the Wasm
| runtime and interfaces with the GPU. The actual graphics
| code is always compiled to Wasm.
| nerpderp82 wrote:
| Can one use with WebGPU in the browser? I see you answer
| that here https://news.ycombinator.com/item?id=39909440
|
| The primary issue with things that include their own Wasm
| env, that then moving that system to the web doesn't work
| because you can't run wasm in wasm.
| seanisom wrote:
| Yes! Not in the open-source repo yet (because it's
| currently broken :) ) but you can see it in the video,
| and will have it working again soon.
|
| That's exactly the goal - one wasm binary with defined
| input/outputs that can be loaded either in a browser or
| running in any app outside of a browser.
| nerpderp82 wrote:
| Then effectively your non-browser container is a browser
| subset just for your application.
|
| You could run a browser and record all the page faults,
| then remove all the code you didn't run.
|
| https://fgiesen.wordpress.com/2012/04/08/metaprogramming-
| for...
| boomskats wrote:
| Not much to add, just wanted to say I thought your presentation
| at wasm I/O in Barca a couple of weeks ago was amazing and it's
| great to see this work getting some attention!
| mambru wrote:
| Barca -> rowing boat
|
| Barca -> the football club
|
| Barna -> cute form of Barcelona
| boomskats wrote:
| I guess a lot of the English-speaking world has
| (mis)appropriated the anglicised name of the football club?
|
| Still, for what it's worth, b7a is my favourite city so far.
| ingen0s wrote:
| how about 1D?
| virtualritz wrote:
| Isn't that what https://github.com/gfx-rs/wgpu is providing?
| seanisom wrote:
| Partially. wgpu can translate graphics api calls like WebGPU to
| platform specific APIs - this is something I had to implement
| on the backend. In the future, will likely tool on top of wgpu
| on the backend as wasi-gfx (WebGPU) becomes a reality.
|
| What we do on top of that is compile the graphics code to wasm
| and provide a well-defined interface around it, so it can
| run/work inside any application.
| netbioserror wrote:
| Saved. This is the sort of project that would be an amazing
| canvas for a nice widget kit and interaction model to make cross-
| platform GUIs with. The C/C++ backend and WASM target means
| people could build FFIs in almost any language. I'm sure I'm
| saying nothing new, but this is promising.
| doctorpangloss wrote:
| > If you think of how Unity made it easy for devs to build cross-
| platform games, the idea is to do the same thing for all visual
| applications.
|
| But why wouldn't I "just" use Unity?
|
| I agree with you. Nobody cares about the platform specific
| details anymore, and people are willing to pay a little bit of
| money for an end-all-be-all middleware. I have gone my whole life
| not paying attention to a single Apple-specific API, and every
| single time, someone has written a better, more robust, cross-
| platform abstraction.
|
| But Unity is already this middleware. I already can make a whole
| art application on top of Unity (or Unreal). People do. Sometimes
| people build whole platforms on top of Unity and are successful
| (Niantic) and some are not (Improbable). You're one guy. You are
| promising creating a whole game engine - you're going to get hung
| up on not using the word game engine, but that is intellectually
| honest, it is a game engine - which a lot of people 1,000x better
| capitalized than you have promised, and those people have been
| unable to reach parity with Unity after years of product
| development. So while I want you to succeed, I feel like a lot of
| Y Combinator guys have this, "We make no mistakes, especially we
| do not make strategic mistakes." It's going to be a long 3 years!
| dartos wrote:
| Unity is so much larger and more complex than a graphics
| middleware.
|
| It comes with physics engines, telemetry, networking, a c#
| runtime and probably even more.
|
| I don't think that any of the adobe suite would ever be built
| in unity bc why do they need to ship a physics engine with
| their photo editor.
|
| Not to mention that unity is backed by an, imo, untrustworthy
| company who's obviously willing to change pricing structure on
| a dime and retroactively.
| pphysch wrote:
| I agree, I think "game engine" is a misnomer for what are
| better termed "game (dev) studio", like Unity. They include a
| sophisticated game engine but also a lot of supporting tools
| and GUIs.
| doctorpangloss wrote:
| > ever be built in unity bc why do they need to ship a
| physics engine with their photo editor.
|
| I know you narrowly mean "rigidbody physics for the purpose
| of videogames." But Adobe did ship a physics engine with
| their photo editor! They discontinued their "3D" support, and
| raytracing is most definitely physics with a capital P, but
| they were shipping it for a long time. If you have an even
| more normal definition of physics, to include optical
| physics, well they have a CV solution for many features like
| camera RAW processing, removing distortion, etc.
|
| > It comes with physics engines, telemetry, networking, a c#
| runtime and probably even more.
|
| Because that is what people need to make multimedia
| applications.
| seanisom wrote:
| Without going into the motivations for building a startup and
| doing Y Combinator, I do agree with many of your points.
|
| People can use Unity to build games and non-games. I personally
| don't think it fits a lot of different use-cases or application
| models and that it tends to be most successful in specific
| gaming verticals, but if it works well for you, by all means
| use it!
|
| I'm strategically betting both on the lines between what is
| viewed as a game and not blurring, as well as developers
| needing a friendlier, more flexible way of building this kind
| of interactive content. I'm by no means under the illusion that
| strategic mistakes won't be made, or that this won't be a
| 10-year+ journey - realistically many (most?) successful
| companies have a very nonlinear path, including Unity
| themselves.
| mentos wrote:
| I agree Unreal and Unity are not appropriate but I do wonder
| about Godot. Its early enough where it doesn't have the
| strong connotations of being a game engine yet. I've seen
| some cool applications made in it too
| (https://www.youtube.com/watch?v=9kKp0oguzr8). So I wonder if
| you could apply your energy to making it more cross platform
| using WASM (if that's even necessary) and extend it with your
| own UI language instead of rolling your own?
| password4321 wrote:
| When you launch include WASM in the title for more traction.
| dang wrote:
| I put it in this one too. Thanks!
| et1337 wrote:
| What's the story with threading on the web these days? My
| impression was that the browsers have purposely and permanently
| handicapped some things necessary for performance in order to
| prevent things like rowhammer and speculative execution exploits.
| But I haven't paid super close attention.
| seanisom wrote:
| It works well now! You use a SharedArrayBuffer to communicate
| between Web Workers. Was indeed a dark couple of years where
| the meltdown stuff disabled that, but there are now specific
| cross-origin headers that are used to isolate the
| SharedArrayBuffer from outside access.
| speps wrote:
| I understand the appeal of Rive. However, even if their renderer
| is open source now, their editor isn't and their free tier is
| quite limited.
|
| Have a look into supporting Ruffle/SWF content, Lottie, etc.
|
| Also, for a renderer there is one by Mozilla called Pathfinder:
| https://github.com/servo/pathfinder
| seanisom wrote:
| Thanks! Lottie should be straightforward. SWF is a much higher
| bar, but would be useful.
| neurowave wrote:
| Can you share what you find limiting in the free tier? Would
| love to know more! I'm one of the founders btw
| seanisom wrote:
| Interested as well - I think you've built an incredibly
| productive editor with Rive - a spiritual successor to Flash!
| ronyeh wrote:
| Awesome project. What are you planning for text and font support?
| Some graphics engines don't support all the ways you might want
| to display text. Will we be able to load OTF or WOFF2 files and
| display arbitrary strings? :-)
| seanisom wrote:
| Thanks! I haven't looked deeply into font yet, but I've always
| been partial to HarfBuzz for shaping, so will probably build on
| top of that. It also has an experimental Wasm shaper which
| certainly served as a bit of inspiration for the design of
| this.
| ronyeh wrote:
| Nice! Looking forward to your alpha release. (And eventual
| HarfBuzz integration.)
| iFire wrote:
| We've been doing work in Godot Engine trying to get wasm working.
|
| How did you overcome the shared array buffer accessibility
| problem on safari vs access to ad networks which is important for
| online games?
|
| I called it single threads vs regular builds.
|
| Hope to help make sure there's a diverse set of rendering kernels
| for everyone.
|
| Edited: Link to our work at making portable 3d graphics on the
| web with an editor.
| https://editor.godotengine.org/releases/latest/
| iFire wrote:
| We also collaborate with https://github.com/thorvg/thorvg.
|
| We were impressed by your work, https://github.com/rive-app and
| https://graphite.rs/
| Keavon wrote:
| Thanks! If perhaps we spoke at GDC (taking a guess given the
| context), it was nice meeting you! (Keavon from Graphite
| here.)
| seanisom wrote:
| Hey Keavon, I'm also a big fan, been lurking in your
| Discord for years! The design of the render tree of Wasm
| nodes certainly took inspiration from Graphite's node
| system.
| seanisom wrote:
| Big fan of Godot! I think it has done wonders to make graphics
| more accessible.
|
| From an Adobe perspective - it doesn't. If you go to
| photoshop.adobe.com in Safari, you will see the answer. Things
| can work in a single-threaded build, but that is not production
| code.
|
| I can't speak for the Safari team, but I do see this getting
| traction soon with the current priorities for Wasm. Seems like
| now the most common answer is just to use Chrome.
| spxneo wrote:
| whew game engine running on WASM + WebGPU would finally be what
| it takes to power browser based AAA titles. We shouldn't have
| to download executables via garden walled ecosystems that take
| a huge chunk of devs revenues
| seanisom wrote:
| This right here. The web is the OS of the future - the
| standards are getting there, the tools are just starting to
| catch up.
| iFire wrote:
| For cad kernels I highly recommend manifold
| https://github.com/elalish/manifold to embed in your app.
| seanisom wrote:
| Manifold is awesome! Would love to get that integration going.
| I've implemented a lot of procedural geometry functions, but
| that is a long way from an actual CAD kernel.
| astlouis44 wrote:
| Great to see more projects in the 3D graphics/WASM space! Any
| tips for getting into YC?
|
| For context, my team has spent the past few years porting Unreal
| Engine 5 to WebGPU and WebAssembly - we have a multi-threaded
| renderer as well as an asset streaming system that fetches in at
| runtime asynchronously (as needed) so users don't need to
| download an entire game/app upfront. This also frees up needing
| to have the whole application in memory at once. We've also built
| out a whole hosting platform and backend for developers to deploy
| their projects to online.
|
| You can learn more about SimplyStream here:
|
| Website: https://simplystream.com/
|
| Blog post: https://simplystream.com/create/blog/latest
|
| Demos: https://simplystream.com/demos
| seanisom wrote:
| I'm probably the worst person to ask for advice about applying
| to YC - it just kind of happened.
|
| I was sad when UE4 sunset HTML5 support, and glad to see a
| spiritual successor! There are a lot of parallels to other
| large in-browser apps in terms of load time for games - not
| just for the content but the size of game code itself. Are you
| able to use streaming compilation or some sort of plugin model?
| astlouis44 wrote:
| Unreal Engine 5 just got a WebGPU/WASM port:
|
| https://news.ycombinator.com/item?id=39911041
| yacine_ wrote:
| i've been using sokol.h and it just works :)
| zengid wrote:
| This is super neat and I am very interested!
|
| I'm in a rush so I can't look to closely now but I have a few
| questions (and please forgive any stupid questions, I'm not a
| graphics dev, just a hobbyist):
|
| What's the runtime like? Is there an event loop driving the
| rendering? (who calls the `render` on each frame? are there hooks
| into that? ) FFI story? Who owns the window pointer?
|
| I'm interested in audio plugins, and VSTs (etc) have a lot of
| constrains on what can be done around event loops and window
| management. JUCE is pretty much the de-facto solution there, but
| it's pretty old and feels crufty.
| seanisom wrote:
| Great questions!
|
| The host app owns the event loop. I don't foresee that changing
| even once we re-architect around WebGPU (allowing the Wasm
| guest to control shaders), as the host app is responsible for
| "driving" the render tree, including passing in state (like a
| timer used for animations). The host app owns the window
| pointer, as renderlets are always designed to be hosted in an
| environment (either an app or a browser). Open to feedback on
| this, though.
|
| FFI is coming with the C API soon!
|
| I don't know much about audio but I see a ton of parallels -
| well-defined data flow across a set of components running real-
| time, arbitrary code. Simulations also come to mind.
| qingcharles wrote:
| Totally off-topic, but in that video you have some preview view
| on the right-side of your code window in VS where you can scroll
| visually through a large file. How the hell do you switch that
| on? I've been using VS for 30 years and never seen that, but it
| would be really helpful for the shitty apps that I write for my
| own use which are single-file monsters.
| morder wrote:
| Right click the scroll bar and choose "scroll bar options" then
| under "behavoior" select the 2nd option with the source
| overview set to "wide"
| seanisom wrote:
| This. I think it used to be a productivity power tool that
| they since incorporated into the core editor.
___________________________________________________________________
(page generated 2024-04-02 23:00 UTC)