[HN Gopher] GPU-Backed User Interfaces
___________________________________________________________________
GPU-Backed User Interfaces
Author : paulgb
Score : 135 points
Date : 2023-02-07 13:48 UTC (9 hours ago)
(HTM) web link (digest.browsertech.com)
(TXT) w3m dump (digest.browsertech.com)
| spankalee wrote:
| The browser _is_ a GPU-backed renderer.
|
| At some point with your own renderer you have to implement a
| large part of the DOM, styling and rendering layers. And why
| should a text editor team have to do that?
|
| I think a lot of teams would be better off by learning how to
| make faster web apps, where there's a subset of HTML/CSS that
| approaches a tree of GPU textures with layout and style on top.
| Do things like: use fewer elements, very simple CSS selectors,
| `contain: content`, `contain: strict`, or `content-visibility:
| auto` everywhere possible, use shadow roots, use a fast vdom-less
| template system, and a framework where each component is its own
| render root, use virtualization for large lists...
|
| If you do that you can also ship a fast web app. And as VS Code
| shows by popping up in more and more sites, being an actual web
| app has some very nice distribution and reuse advantages.
| outworlder wrote:
| > The browser is a GPU-backed renderer.
|
| That matters very little. By that logic, _everything_ is GPU
| backed these days and rendered by a compositor, on all
| platforms.
|
| > At some point with your own renderer you have to implement a
| large part of the DOM
|
| No, not really. Unless you are saying "DOM" to mean any tree
| structure. And you might not even need that for your app,
| depending on its layout.
|
| When people say "GPU rendering" in this context, think user
| interfaces found in games. They are directly rendered by the
| game code without any intermediate layers. And very often,
| without anything that one would even remotely recognize as a
| DOM (3D games have a scene graph, but that's still different).
|
| > And why should a text editor team have to do that?
|
| Ask Sublime Text and iTerm. They already do that to render
| their content (and implement native OS dialog controls for the
| rest).
|
| > I think a lot of teams would be better off by learning how to
| make faster web apps
|
| True, just because of the amount of effort. _Users_, on the
| other hand...
| aconbere wrote:
| Being a webapp and adding GPU rendering are not orthogonal.
| Figma for instance famously renders it's UI in WebGL.
|
| While I mostly agree that creators of today's web forget that
| there are a number of simple techniques to render fast pages,
| there are also a bunch of places where the HTML+CSS+JS web
| technologies serve us poorly.
|
| * Content Editable is a terrible editing canvas
|
| * Dom and CSS are pretty terrible core animation primitives
|
| * DOM and CSS make for pretty poor interactive graphics
| capabilities
|
| And in these cases WebGL offers some interesting alternatives
| to explore.
| spankalee wrote:
| Figma renders its _documents_ with WASM /WebGL, but the UI of
| the app is just HTML.
|
| A fully GPU rendered web app is almost the worst of both
| worlds. You'll generally have massive code bundles and
| terrible a11y.
| DaiPlusPlus wrote:
| I remember when I first tried Flutter+Dart and was really
| put-off by how it rendered to <canvas>, then later versions
| did use the DOM but looked like the worst case of divitis
| I've ever seen, it even made React.js's output look
| handwritten.
|
| But why do people do this? I note that a big problem with
| the web, as-is today, is how anemic and inflexible HTML's
| standard <input> elements are, which forces devs to build
| their own widgets (often poorly), maybe use a framework
| like Angular, and then eventually go all-in on web
| abstractions like Flutter/Blazor/etc. If HTML were better
| then many (but far from all) of these cases could be
| avoided - for instance, I don't understand why HTML still
| doesn't have absolutely basic widgets like a combobox, a
| single-line textbox that wraps, a date-range input,
| hierarchical drop-downs (optgroup is single-level), defined
| styling and non-text content of <select>, and so on...
| (though we did recently get color and range/sliders -
| though there's still cross-browser quirks which hamper
| adoption).
|
| I hated the days of ActiveX like everyone, but when you're
| building an internal-only (I.e. IE-only) webpage it was
| neat that we could pull-in any of the standard Windows
| controls - and plenty others - into a page and it would
| just-work. And you got accessibility for free too, which is
| more than what you get with <canvas>-based frameworks still
| - or a multi-megabyte JS/WASM bundle download either (or an
| npm-based build system that breaks every time you update
| your dependencies...) ]. Okay yes I'm jaded.
| auggierose wrote:
| Content Editable seems to be quite good these days, if you
| don't use it as is, but are willing to layer a framework on
| top of it. I tried Lexical, but am now going to roll my own,
| but still entirely based on HTML/CSS and ContentEditable. No
| WebGL required, although I might very well use it for a
| particular editing component.
| audidude wrote:
| > And as VS Code by popping up in more and more sites,
|
| On Linux, VS Code can't even resize without glitching and
| showing unrendered portions of the window like it's 1999. Nor
| can it kinetic scroll. Those are two expected highlights of
| quality GPU-backed software.
|
| Hopefully running in an actual browser would make this better
| simply because you can choose a browser that isn't Electron.
| spankalee wrote:
| afaict, VS Code doesn't employ most of these modern
| techniques for faster layout and rendering. Last I checked
| they didn't have CSS containment or shadow roots anywhere,
| and they have very, very large DOM trees.
|
| They're the classic example of jumping from unoptimized
| HTML/CSS to GPU to try to fix performance as they moved the
| terminal to be GPU-based, and it doesn't seemed to have
| helped because that introduced rendering bugs that would have
| been impossible with HTML.
|
| I only use VS Code as an example of how many places you app
| can be used if it's web-based.
| teucris wrote:
| DIY accessibility is harder than it looks, even if you plan to
| hook into each platform's native stack for it. For all its flaws,
| the web stack standardizes the structure of UI, greatly reducing
| the dev cost of making accessible apps.
| c-smile wrote:
| > Unlike the UIs provided by the operating system, where you can
| say "draw a button with the label 'delete'", you have to tell
| OpenGL how to make a button out of triangles.
|
| Just use Sciter, by default for rendering it uses
| Direct2d/DirectX (+ Vulkan||OpenGL) on Windows, Metal||OpenGL on
| Mac and Vulkan||OpenGL on Linux. With support of a11y by the way.
| fassssst wrote:
| But the UIs provided by the operating system are making
| DirectX/Metal calls for you... they are GPU accelerated. And
| they have been for like 16 years now.
| c-smile wrote:
| > UIs provided by the operating system ... GPU accelerated
|
| That depends on what you mean by "system UI".
|
| Most system utilities on Windows are still using GDI. GDI is
| not GPU accelerated (other than BitBlt).
|
| Same apply to MacOS - CoreGraphics there is pure CPU
| rasterization library.
|
| And the same situation on Linux/GTK, system uses Cairo - pure
| CPU rasterization library.
|
| Usually, on modern OSes, window/desktop manager is the only
| GPU accelerated thing - it composes window bitmaps on screen.
| audidude wrote:
| > And the same situation on Linux/GTK, system uses Cairo -
| pure CPU rasterization library.
|
| GTK 4 renders by default using your GPU. The only part that
| is CPU rasterized is if we don't have an accelerated path
| for it (rare) and glyphs are first rendered on CPU,
| uploaded to a texture atlas, and then GPU colored/blit from
| there on.
| fassssst wrote:
| WPF and WinUI on Windows are both GPU accelerated. They
| render with Direct3D calls under the hood.
| MrGilbert wrote:
| While WPF is hardware accelerated, it is not very
| efficient[1]. I'm not sure if this has changed lately.
| Always had the impression that WPF was not Microsoft's
| favorite child.
|
| [1]: https://jeremiahmorrill.wordpress.com/2011/02/14/a-c
| ritical-...
| c-smile wrote:
| Sciter greatly overlaps with those.
|
| It is just that instead of XAML it uses HTML/CSS,
| JS/C/C++ instead of C# and works on all desktop
| platforms.
|
| For the note: historically Sciter appeared before WPF and
| WinUI/UWP : https://sciter.com/sciter/sciter-vs-wpf/
|
| By any means WPF and WinUI are not UIs provided by the
| OS. These are userland UI libraries. Same thing as
| Sciter.
| jakearmitage wrote:
| Closed source.
| c-smile wrote:
| Yes, and what? How that is related to the article we are
| discussing?
| th0ma5 wrote:
| What is the accessibility story for GPU backed interface
| software? On the web?
| alex_suzuki wrote:
| As mentioned in TFA, it's not there yet.
| FpUser wrote:
| I just briefly looked at makepad and the underlying framework.
| Seems like very nice project.
| neilv wrote:
| One of the things I liked about Blender UI implementation, eons
| ago, is that I could fire it up on a _very_ modest old Linux
| laptop, and it started up in an instant.
|
| What I'd think was the most demanding GUI application on my
| laptop also seemed the most responsive.
| aequitas wrote:
| Also their save file format was in a way that allowed super
| fast saves of really complex projects.
| noobermin wrote:
| This article doesn't answer why. I can come up with some uses of
| a GPU backed interface, but you know people will start using this
| to render their six page web app in due time. Why not just use
| the already existing GUI frameworks???
| alokedesai wrote:
| Disclaimer: I work at Warp (warp.dev) and wrote the initial How
| Warp Works blog post (https://www.warp.dev/blog/how-warp-works)
| that discusses why we chose to write our own UI framework.
|
| This is a great article! Writing a custom UI framework to render
| on the GPU is a big cost--but it has been worth it for us. It has
| let us build a UI-heavy terminal while continuing to match the
| performance of some of the fastest terminals on the market (such
| as Alacritty). There is certainly an initial velocity hit for new
| engineers as they have to learn the semantics of our framework,
| but overall I wouldn't say it has a big detriment on our velocity
| as a consumer of the framework.
|
| The callout to a11y is a good one--we have basic a11y support but
| it is by no means robust. There are additional downsides to
| building your own GPU-rendered UI framework other than a11y that
| I want to call out: you have to build most of the interactions
| with the platform yourself instead of getting it for free when
| rendering using system APIs.
|
| One example of this is properly supporting i18n. A robust
| framework should support RTL text, positioning IME / the emoji
| picker, so forth. While implementing these yourself is doable (we
| have decent support for IME at Warp, for example) it requires an
| engineer or two to spend a few weeks to implement.
|
| Text rendering is also a beast, see
| https://www.warp.dev/blog/adventures-text-rendering-kerning-....
|
| For Warp, this has definitely been worth it, but it hints that
| the community needs better cross-platform GPU-rendered UI
| frameworks so people don't need to consistently build them
| themselves.
| efficax wrote:
| Would be great for the rust community and also for warp's rep
| among developers if y'all opensourced the UI framework. The
| rust ecosystem just doesn't have a good answer when it comes to
| choosing a UI lib yet.
| capableweb wrote:
| Interesting terminal, sadly not yet available for other
| platforms than Mac.
|
| I went to your GitHub (https://github.com/warpdotdev/Warp) in
| order to see if I couldn't figure out how to build it for
| Linux, but seems that's like a "issues" repository and I cannot
| find the source code anywhere, nor is it linked on the main
| website (https://warp.dev).
|
| I'm guessing at this point that it's on purpose? The source
| won't be available and it's only a closed source product?
| smoldesu wrote:
| They've talked about it being on the roadmap for a while, but
| I'm not aware of any recent advancement in getting it to work
| cross-platform (much less publishing the source).
|
| Indeed, "at this point" they've demonstrated that cross-
| platform development is a low priority for them. Even if they
| do ship a Linux client, I feel like it will be a hard-sell
| when there are so many other terminals treating Linux like a
| first-class target.
| buserror wrote:
| I don't want to brag, but I wrote GLterm (for OSX) in 2002 or
| so, first ever terminal emulator using OpenGL... It was a
| shareware tho (it was all the rage back then)...
|
| In fact I probably reported and had ATI and nVidia fix bugs
| that even _made_ it possible to have pixel aligned textures,
| partial refreshes of GL surfaces etc. It was all basic stuff,
| but it was always overlooked because "gaming"...
|
| I find it hilarious that the GL UIs are all trendy now, I've
| done many of them professionally for the last 20+ years. One of
| first one was Sibelius v2, a desktop software for writing
| (engraving) musical scores, I converted the windows GD code to
| a GL backend all the way back then, rendering in textures etc
| etc.
| aconbere wrote:
| I was shocked to find that there wasn't a fully fleshed out
| text rendering engine for WebGL.
|
| On the one hand I think you could argue that if you have a lot
| of text maybe html would suit your needs better. On the other
| hand a lot of UIs just need some text and it's so east for it
| to look like junk.
| flohofwoe wrote:
| There's an entire layer of mid-level web APIs missing to
| better connect WebGL and WebGPU to all the other browser
| systems (like text and DOM rendering).
|
| Ideally canvas with WebGL and WebGPU would sit at the bottom
| of the browser API layer cake, the DOM at the top and
| squeezed inbetween public APIs for text, shape, CSS+HTML
| rendering. But instead we got this weird canvas thing
| dangling off as another high-level DOM element.
| aconbere wrote:
| Totally. It would be /amazing/ if I could leverage the
| browser's existing (and largely excellent) document and
| text layout engine from /within/ WebGL. Let me build up my
| UI as I want, and inject a DOM into that.
| californical wrote:
| First off: thanks for sharing! Warp is building some real
| innovation in a Terminal, and it's been a blast to follow the
| project.
|
| It's interesting that you mention this now - I've been a Warp
| user for a few months but just yesterday started seeking out
| another option because the UI has been feeling sluggish in
| Warp.
|
| I installed alacritty and kitty, and they both feel insanely
| fast, comparatively. iTerm is approximately the same as Warp.
|
| I think the difference is significantly smaller than most
| people would intentionally target with UI paints. But when I
| press enter, it feels like warp just has a minuscule delay,
| where kitty has instantaneous printed the results within the
| same frame that my finger hit enter.
|
| Do you know if there's a specific # of ms that Warp is
| targeting for a UI response from user input? I'm not at all an
| expert in this domain, but am curious if it's "all in my head",
| or if there really is a difference here. Do you have similar
| stats for other terminals?
|
| Edit:
|
| Only asking because I know it's hard to get the right balance
| between extra feature & speed. The autocomplete & syntax
| highlighting are really awesome features that I love about
| Warp, and those likely take many more CPU cycles. Warp is _far_
| more powerful than kitty in that way. But how many ms does it
| cost to have those features?
|
| I know the answer isn't the same for everyone about what the
| right trade-offs are between features and speed, and I haven't
| made up my mind yet which is more important for me, to be
| honest. Warp is crazy impressive and has been a step up in
| functionality. Again, really love the work you're doing!
| madeofpalk wrote:
| I find it strange how, in an industry where people will eke out
| minor percentage improvements on conversation rates, "we" are so
| happy to just intentionally slice off a portion of the potential
| user base to never be able to use the software.
| password11 wrote:
| 2.3% of the population have a visual impairment
| (https://ux.stackexchange.com/questions/57340/percentage-
| of-s...).
|
| But because conversion rates are typically low and because of
| how the math works out for a funnel product, there's a larger
| business impact depending on which 2.3% is cut out. (Ignoring
| the negative business impact of being non inclusive/a11y)
|
| 12% vs. 10% is a 20% decrease in performance.
|
| 100% vs. 98% is a 2% decrease in performance.
|
| I'm not arguing against a11y. I'm just saying, that's why ADA
| exists. It's not in the business interest without the law. Also
| it's just good ethics to be accessible.
| madeofpalk wrote:
| > 2.3% of the population have a visual impairment
|
| I'm not sure exactly what this 2.3% is (for "blindness"?),
| but significantly more than 2.3% of the population wear
| perscription glasses. Those people have visual impairment and
| use assistive technologies
|
| Screen readers are not the only assistive technology.
| Especially on phones, many people use assistive tech to
| increase font weight or size. Ignoring these options will
| remove customers.
| snickmy wrote:
| I feel there is something wrong with this approach but I struggle
| in putting my finger on it.
|
| Something about using a level of abstraction at the wrong layer
| of the stack. I would expect GPU rendering to be used way deeper
| in the stack (OS level).
| jillesvangurp wrote:
| As others are pointing out, most UIs are hardware accelerated
| directly or indirectly.
|
| Most "native" apps, are native in the sense that they use some
| some UI toolkit that renders using the platform specific and
| typically hardware accelerated ui infrastructure.
|
| With webgl and wasm, those same toolkits can now work in a
| browser.
|
| As for accessibility, that is indeed a challenge. You would need
| to use some ui toolkit that supports accessibility features.
| Those of course do exist, browsers don't have a monopoly on
| accessibility.
|
| A related challenge is the integrating that with the browser
| properly so that the native accessibility facilities in your OS
| can do whatever it needs to do (screen reading, enhancing
| contrast, etc.). This is not a solved problem. But it sounds like
| it could be a solvable problem that would require some changes to
| wasm and webgl.
| actionfromafar wrote:
| When you have made all these changes to the browser, is it even
| a browser then? More like a Javascript shim on top of native
| controls.
| necessary wrote:
| Maybe not compared to what browsers were a decade or two ago,
| but I think the definition has changed. These days browsers
| are just sandboxes with convenient tooling for finding,
| downloading, and running arbitrary code. And I don't think
| that's a bad thing: that's what allows Photopea and Figma to
| exist in the browser at a moment's notice.
|
| Once you generalize the definition of browser, both ends of
| the spectrum coexist: from basic HTML pages to fully WebGL
| rendered WebAssembly apps.
| actionfromafar wrote:
| It's already that, but what I got from the OP was to add
| native controls, like what is necessary to (re-)implement
| accessability. (That builtin accessability mechanism of the
| browser itself won't do for various reasons.)
| jillesvangurp wrote:
| Technically, it's a bit of a bait and switch. But one that
| has largely already happened. It's just that web developers
| have yet to notice. Javascript at this point is a legacy
| compilation target for stuff that hasn't been ported to WASM
| yet.
|
| WASM is getting better with every browser release. Likewise
| for webgl. It was perfectly fine for doing 3d games many
| years ago. It's just that we are stuck doing the same
| react/js/whatever crap that we've been using for the last
| decade or so. It's just awful how low the standards are for
| web UI. But it was never good enough for game consoles or
| mobile.
|
| At this point, it's probably easier to run a browser as a
| wasm program inside of a modern browser just to render some
| legacy web app from way back than it is to get that app
| working with modern browsers. Why even bother? Just run IE 6
| in an emulated browser on an emulated windows 95. And
| probably it would be fine in terms of performance too.
|
| Javascript + css has always been a bit of a cringe fest in
| terms of capabilities. If you develop a UI that matters for a
| gaming console, a desktop OS, or anything else, you wouldn't
| use any of that. It's why native development is a thing for
| IOS and Android. The phones are fast enough to do web
| development. It's just that the whole space is too
| competitive for that to be even worth considering.
|
| So, maybe it's becomes a better browser without that?
| Strom wrote:
| > _most UIs are hardware accelerated directly or indirectly._
|
| To widely varying degrees. Some do a lot of work on the GPU,
| others use only the rasterization features, some only
| accelerate the compositing. This unfortunately means that using
| simple surface level terminology like _GPU accelerated_ doesn
| 't sufficiently describe what is actually happening so that we
| could make meaningful comparisons between toolkits.
|
| Ultimately I think what's needed is proper benchmarking of
| performance and measuring of battery life. A toolkit that uses
| GPU acceleration for a simple step might actually be slower
| than a full CPU implementation, because there is a cost to
| syncing your data between CPU and GPU. Also if the whole UI is
| rather simple, then even a solution that is built completely on
| the GPU might be slower than CPU, again because of the
| associated costs. If it takes 2ms to get everything set up on
| the GPU for a frame, then it doesn't matter if the GPU renders
| it in only 1ms vs 2ms on the CPU, as total time is now 3ms with
| the GPU vs 2ms with CPU only. It's a contrived example, but one
| that could easily happen without a focus on measuring.
| _jezell_ wrote:
| Flutter is quickly shaping up to be the best way to accomplish
| this. The new renderers landing on all platforms including web
| are really nice. I've ignored Flutter until now, but it's getting
| too good to ignore for much longer.
| nzoschke wrote:
| Came here to say this.
|
| Flutter draws to a canvas at 60 FPS and uses the GPU.
|
| https://buildflutter.com/how-flutter-works/
|
| Anecdotally, I see it getting faster and faster on new engine
| versions without app changes.
| von_lohengramm wrote:
| I absolutely hate how 60 FPS is the goal for these canvas-
| based solutions. I can render DOM at 240 FPS (and much higher
| if my monitor had a better refresh rate) no problem, but
| things like Flutter barely ever even make it to the triple
| digits. It's an embarrassing case of contentment. 60 FPS
| should not be considered ideal. It should be considered the
| absolute bare minimum and really only applicable for
| resource-constrained systems. If you're not doing absurd
| amounts of number crunching like raytracing, then you should
| aiming for 1000 FPS.
| Kukumber wrote:
| That's weird that the author completely ignores popular projects
| like ImGui [1] or lvgl, maybe the author wanted to talk about
| W.I.P. GPU based UIs written in Rust only? In that case, there
| doesn't seems to be a lot of choice
|
| [1] - https://github.com/ocornut/imgui/issues/5886
| paulgb wrote:
| I had some notes on Dear ImGui (and the prevalence of
| immediate-mode across other frameworks that it inspired), but
| ultimately cut it for length because I wanted to focus on
| applications rather than frameworks/GUI toolkits.
| Kukumber wrote:
| https://github.com/ocornut/imgui/issues/3075#issuecomment-60.
| ..
| lionkor wrote:
| ImGui, by default, is run at like 60+fps. No GUI app that
| doesnt display real time changes needs to render at 60+ fps
| seanw444 wrote:
| Maybe not a constant 60+ FPS, but if I scroll, and I can tell
| it's rendering at 20-30 FPS max on my 144 Hz monitor, I'm
| going to be annoyed. I want smoothness.
| jakelazaroff wrote:
| Dear ImGui doesn't actually include a draw loop, does it?
| AFAIK you're in control of how often that happens.
| lionkor wrote:
| Yes, and the example and sample code does it with max
| framerate.
| jakelazaroff wrote:
| Okay, but that's not the same thing as "Dear ImGui runs
| at 60fps by default".
| mastax wrote:
| > It takes a lot of work to make a usable UI out of triangles, so
| most applications did not go this path. One big exception was
| games, which were already heavy GPU users
|
| In the past, Scaleform was close to ubiquitous, at least for
| large games. Scaleform is Flash-based, so it was a vector
| graphics engine with a JavaScript-like language. It was
| presumably faster and less bloated than a browser engine would
| be, as a game middleware from the Xbox 360 era. But from a
| developer experience point of view, it closer to a browser engine
| than to "[making] a usable UI out of triangles."
|
| Scaleform is still pretty popular, but it has become popular to
| create game UIs with web technologies. You can find libCEF.dll in
| pretty much every game nowadays. Sometimes it's just used to
| render the update news or in-game store, but often the entire UI
| is HTML.
| aconbere wrote:
| I've spent the last week playing around with running WASM apps
| executing WebGL.
|
| I'm exhausted by the modern web. There are so many layers and
| layers and layers, and none of it works very well!
|
| But WebGL rendering on the web offers a unique and compelling
| alternative. I can write my frontend and backend in Rust. I can
| use gui libraries like egui directly if I want or write my own
| UIs if I prefer.
|
| It's almost like going back to the bad flash days, but with open
| web standards.
|
| There are a bunch of downsides (some the article mentions.
| Accessibility isn't there yet, the platform is immature so a lot
| of basic stuff you have to write yourself (form handling, etc),
| and a big one for me is: text rendering! There appears to be few
| if any mature text rendering libraries that you can drop into a
| webgl project and get good crisp text, shaping, and kerning.
| traverseda wrote:
| My ideal stack is plain html, enhanced with HTMX for things
| like form validation.
|
| Then for more complicated stuff I use self contained web
| components. That web component might be a terminal, or it might
| be a text editor, or it might be some kind of video calling
| widget.
|
| One example, you don't need javascript for a chat interface
| (aside from htmx). You can use htmx to subscribe to a server-
| sent-event stream for receiving messages, and for posting them
| you just use a normal form element. It lets you write a lot
| more app-like functionality as standard html, and for when you
| can't do that a custom element is often the answer.
|
| Don't know why web components haven't caught on more.
| martin_drapeau wrote:
| 15 years ago I was at the same place you are. Believe it or
| not, I built an entire web-like UI using Tcl/Tk!
|
| But then I hit limits and started re-implementing the wheel. So
| I bit the bullet and learned all those layers of web
| development. Never looked back.
|
| When you get to that point, may I suggest you master HTML and
| submitting forms. You can do a lot without Javascript. Look at
| Laravel for an easy to comprehend framework (PHP). It is just
| perfect.
| aconbere wrote:
| I think you might be confused about where I'm coming from and
| starting at. I've been writing rich web applications
| professionally since 2004.
| whitehexagon wrote:
| I have also been developing a Wasm UI framework first with
| canvas2D and Go, but the wasm was too big, and now using Zig
| and WebGL. But yes, fonts/text is a pain with GPU, all that
| power and you'd think by now you could upload a font to the GPU
| and call drawText(x,y). I thought about a shader for that, or
| custom svg renderer, but probably months of work, and now
| thinking to layer a Canvas2D over the WebGL and keep the font
| rendering on the browser side and let it do its z compositing,
| but probably lose a lot of the performance benefits of a pure
| WebGL UI... And I already found a bottleneck trying to render
| animated SVG onto a texture map that results in a copy back
| each frame, and yet 'video' can be rendered to texture directly
| by blit... frustrating. Almost thinking bare metal PI fb
| development would be more fun :)
| wolframhempel wrote:
| Having built a few GPU/HTML hybrids in the past, (e.g.
| https://arcentry.com/) I found that it teaches you a lot about
| the things you take for granted when developing for browsers and
| that you now have to manage yourself. This includes for example:
|
| - figuring out when to redraw the screen or sections of it
|
| - organizing your UI as a hierarchical tree that can be traversed
|
| - handling events in general. what did the user click on? What Z
| order are things in? Can we find out using 2D boxes or do we need
| to do 3D ray intersection?
|
| - combining UI hierarchies and events into event propagation
|
| - rendering text and images crisply is surprisingly tricky
|
| - blending layers over each other requires care.
|
| - layout flows (margins, paddings, things pushing against other
| things) need managing.
|
| ...and a host of other things. For anyone interested, I wrote a
| summary in my early days of development here:
| https://medium.com/@WolframHempel_82303/seven-things-ive-lea... -
| though this misses a lot of what I figured out later.
| dekhn wrote:
| With some exceptions, the Qt Graphics View Framework satisfies
| all these goals with very little additional code from the user.
|
| It won't automatically render things as a tree (but the widget
| system does that for you) or doing layout flow (it's not a
| layout manager). But it does all the other things so well it's
| amazing it hasn't been cloned into every other graphics
| framework.
|
| https://doc.qt.io/qt-6/graphicsview.html
| outworlder wrote:
| > figuring out when to redraw (...) sections of it
|
| What happens if you just brute-force and re-render everything
| when there's a change? I'm all for optimizing but is this
| really required these days? Even considering laptops and mobile
| power requirements, rendering off-screen and flipping is
| lightning fast.
|
| Layout flows are a different matter and can be expensive to
| compute.
| kaba0 wrote:
| > Even considering laptops and mobile power requirements,
| rendering off-screen and flipping is lightning fast
|
| I believe it is not speed, but battery life. Most of the time
| nothing happens on the screen besides some small widget
| changing like once per second (e.g. clock), doing 59 full
| rerenders every second for that (or even more with high
| refresh rate screens) is significantly higher drain.
|
| But I am absolutely not an expert on this.
| Strom wrote:
| Your point is correct and there is another changing thing
| beyond the clock that's common - the blinking caret in a
| textbox. Firefox used to have really bad battery usage on
| macOS for a long time because in order to only change the
| caret you have to do a lot of things the Apple way.
| kaba0 wrote:
| Oh didn't think of that! When you are more in control of
| the hardware then hardware planes could help with that -
| at least cursors in display managers are handled that
| way.
|
| Does native OS frameworks have some escape hatch for
| these slightly changing, constantly damaging elements?
| wolframhempel wrote:
| I think that's right. I started of just rendering
| everything on every requestAnimationFrame callback - and
| things rendered smoothly, especially when nothing needed
| copying to the GPU. But users complained that their laptop
| fans sprang into action and their battery life plummeted.
| seba_dos1 wrote:
| > I'm all for optimizing but is this really required these
| days?
|
| _Of course_ it is. Even if it doesn 't matter on your
| machine, your users will launch your code:
|
| * on a 10 years old PC
|
| * in train on a laptop with 10% battery left
|
| * on a computer with so many things running in the background
| that you'll be on the edge of thrashing
|
| * on a thermally throttled ultrabook
|
| * on an underpowered SBC
|
| * through foreign CPU arch emulation
|
| * [insert more cases here]
|
| In each of those scenarios, your code will either do its job
| well or be the source of frustration.
| russellendicott wrote:
| figuring out when to redraw the screen or sections of it
|
| Yes. I did a fun side project where I wrote a client-server
| protocol and browser for terminal user interfaces and I had to
| learn all of this. It was a lot of fun though and I learned a
| lot about browser design and how it manages cookies.
|
| Info here: http://uggly.bytester.net/
| outworlder wrote:
| That's something that I have always wondered. And, having built a
| few of these myself on my free time for personal projects
| (starting with a university homework), I have my $.02
|
| It is really difficult to do this correctly. The simplest systems
| to build with are the ones where you have a large visualization
| window and you only need some knobs to control it.
|
| The things that you think are going to be tricky (say,
| skeuomorphic knobs) are actually almost trivial to implement,
| even if compared to standard GUI toolkits(but not browsers).
| Things that you have taken for granted (textboxes) require months
| to implement correctly. And you will still find corner cases, be
| it keystrokes(even without taking different operating systems
| into account).
|
| It is easy to make it look 'flashy'. It is surprisingly difficult
| to make it look 'good'. We see some beautiful UIs in games and we
| think our applications can look the same. They really can't, not
| without great graphic designers. And no matter how great it
| looks, it will look 'alien' compared to other applications in the
| system.
|
| If you need any kind of dynamic layout (which you normally do to
| handle resizing), you are now looking into an entire system and
| months of development.
|
| Native OS controls usually have ways to hook up to them from
| external apps. Not so when you are drawing yourself. So that
| breaks accessibility but also any chance of automation.
|
| All those drawbacks (and more I haven't touched) aside, there are
| benefits.
|
| They can be incredibly fast, faster than native OS controls.
| Rendering a few quads is no problem for any GPU. You can have
| incredible interactivity(this needs to be explored more by non-
| Games), since rendering is so fast, faster than the older days of
| blitting stuff in framebuffers(and a lot of it can be handled by
| the GPU).
|
| You can also implement transitions that are impossible to do in
| anything else (maybe except for web browsers). You can transition
| between 2D and 3D seamlessly.
|
| For color blindness you can just apply a very simple shader over
| the whole thing.
|
| I think there is a lot of potential hidden by this approach.
| Think of some the of GUIs imagined by works of science fiction.
| The main issue is that there's a lot of work to be done before
| you have anything functional and if you use something out of the
| box, it will be probably difficult to extend.
|
| That said, I've considered taking this idea to an absurd level
| and write, say, a text editor on something like Unreal Engine and
| see how far into absurdity we can extend this idea.
| SpartanJ wrote:
| This is an ongoing trend that will continue until some of the new
| GUI-backed UIs gain some popularity. What we need right now is a
| real alternative to Electron based applications, and probably
| carrying new technologies. I know that we already have
| alternatives, like Qt, which is an amazing framework, but it
| doesn't fit all developer's needs and sometimes looks also "too
| big". I think that many devs are saturated of the HTML+CSS+JS
| ecosystem and want something new, maybe something less bloated in
| the sense of the tech specs. I celebrate the numerous projects
| that are appearing these days, to name a few: egui, iced, imgui,
| GuiLite, RmlUI, elements. They need more exposure, more
| collaborators, and more development, they need to grow.
|
| Having said that, I'm working on an alternative GPU accelerated
| and retained-mode GUI called eepp
| (https://github.com/SpartanJ/eepp/) while I work on a new code
| editor similar to Zed called ecode
| (https://github.com/SpartanJ/ecode). The framework still needs a
| lot of work but presents some ideas that might be interesting to
| some. Retained-mode GUIs are a complex beast and it's hard to
| achieve simplicity (I think immediate-mode GUIs can't be beaten
| in that regard, but they have their own downsides). So I invite
| anyone interested in GPU accelerated GUIs to take a look at my
| projects (online demos are available), I'm looking for
| collaborators! (sadly C++ isn't very appealing these days, but my
| code-base is huge and I can't write everything from scratch in a
| new fancy and cool language).
| thewebcount wrote:
| Aren't most native UI toolboxes also GPU-backed? For example
| Quartz on macOS is. I assume that Windows has something similar,
| too. Someone else mentioned GTK4 is on Linux. If you want a fast,
| accessible, user-friendly GPU-back UI, just use the native one.
|
| I keep hearing that learning 2 or 3 native UI frameworks is too
| much work and too costly, and the answer keeps being to learn yet
| another new cross-platform UI framework that creates apps that
| are some combination of slow, lack useful features, and don't
| look right to the users. At this point, wouldn't it be less
| costly to just learn the native UI frameworks?
| [deleted]
| audidude wrote:
| Does Cocoa draw controls like checkboxes and what not with the
| GPU?
|
| In GTK 4's case, drawing the widgets, CSS transitions, shadows,
| etc is all on the GPU. Video from GStreamer arrives as GL
| textures, same for WebKit integration and therefore can be used
| with all of the other high-level render-tree snapshotting
| features.
| kaba0 wrote:
| I'm way out if my depth here, but isn't certain things faster
| to render on the CPU still? I believe it is mostly vector
| graphics - I have seen a few papers showing that it can be
| done fast(er?) on the GPU but that may not be that widespread
| yet.
| audidude wrote:
| > but isn't certain things faster to render on the CPU
| still?
|
| It really depends on the context.
|
| If you look at it in a certain light, fonts are just
| "vector graphics" and there are ways to do that without
| using SDF (signed distance fields) and get really high
| quality precisely positioned fonts fully rendered on the
| GPU.
|
| An example of that is the GLyphy project and we have a
| branch which can render fonts using that with GTK 4. It's a
| bit slower, but better quality and still easily able to
| handle 144hz full screen text scrolling. It requires some
| preprocessing before uploading glyph curves to a data atlas
| on the GPU.
|
| For other shapes, there are options, but given GTK has so
| few upstream developers, they aren't quite ready for prime-
| time so they'll likely be part of GTK 5 in the form of
| "GskPath" if you feel like browsing the branches with that.
| vbezhenar wrote:
| Windows 95 rendered its UI on CPU and it was fast. You need
| GPU for big bitmaps and useless effects. For square with
| border and text CPU is more than enough. Then you send this
| small square bitmap to GPU to add shadows and whatnot to
| it.
| Strom wrote:
| Windows 95 doesn't even support Unicode. The level of UI
| you can draw there is a very small subset of what you can
| do in say Chrome.
|
| Indeed correct unicode text rendering is one of the
| difficult problems. It's the same thing that is missing
| from basically all games. So when people say stuff like
| _games can run at 120fps, how hard can UI be?_ it 's
| basically the same Windows 95 argument. The UI work being
| done is a small subset.
| ihatepython wrote:
| Not supporting Unicode is a feature for me. I have
| absolutely no use for Unicode, but it is a constant
| source of security vulnerabilities.
| kaba0 wrote:
| Yeah, let's ditch all those other pesky languages and
| just stick with English /s
| pjc50 wrote:
| > At this point, wouldn't it be less costly to just learn the
| native UI frameworks?
|
| People seem to be really unwilling to do this, I don't know
| why. Maybe it's a language issue. Mind you, the official
| Microsoft story for how to write a native app is .. messy. But
| its XAML-based renderers do use the GPU.
| outworlder wrote:
| > People seem to be really unwilling to do this, I don't know
| why.
|
| They are usually _horrible_ to write in. Win32 is an archaic
| mess. Linux doesn't even _have_ a native UI framework, X and
| the like don't handle widgets. GTK is the least unpleasant
| but crafting UIs with it is very tedious. KDE is C++ based
| and interoperates poorly with anything that's not C++.
| Wrappers exist but they vary in quality. The Apple equivalent
| is almost pleasant to work with in comparison even though it
| has its warts (iOS you can deal with it directly in
| Objective-C no problem, OSX is more idiosyncratic)
|
| And then you have to repeat for all supported platform and
| rewrite everything.
|
| You could use something like wxWidgets, but it's not pleasant
| to use either. Language bindings are still problematic.
| guhidalg wrote:
| > They are usually _horrible_ to write in.
|
| No? Basically all the GUI frameworks are horrible to write
| in, and if you haven't reached a point where you're pulling
| your hair out then your application just isn't complex
| enough. There is an unavoidable amount of complexity in
| writing GUIs and using a cross-platform framework means you
| get to wade through that unavoidable complexity N+1 times:
| once for each platform and then once for your framework!
| yread wrote:
| You can still use that 90s visualbasic 6, it still works
| Kukumber wrote:
| Microsoft sticking to XAML is one of their biggest mistake
| ever WRT native UIs, no wonder it's a dreaded target
|
| Then you have both Google and Apple who invested massively
| into new languages and framework to facilitate the
| developments of apps and therefore native UIs
|
| - Compose for Google in partnership with Jetbrains for Kotlin
|
| - SwiftUI with Swift for Apple
|
| Flutter in the way to make things cross-platform, all of them
| declarative, all of them native, all of them doesn't require
| learning a dedicated markup language
| cal85 wrote:
| Off topic, but I wish more tech articles were written like this.
| The writing style is really easy to follow, succinct, plain-
| spoken, but with enough technical detail. Difficult balance to
| get right.
| iTokio wrote:
| a11y There are efforts to support a cross platform accessibility
| library:
|
| https://github.com/AccessKit/accesskit
| xlazom00 wrote:
| On windows 8 and newer, microsoft use new framework to create UI
| for apps. They are all GPU accelerated. Calculator, Settings,..
| And it is up to the framework if it support all feature like
| accesiblity or system spellcheck or whatever.
|
| And on Mac OS Cocoa use GPU acceleration as well.
|
| As if you want to show something nice and fency you will need GPU
| acceleration. But still you can fall back to CPU rendering. There
| is Open GL on CPU in MESA. same for DirectX. It is shower for
| some fency stuff but good enought for basic UI
| naillo wrote:
| Regarding the last point, does this mean developing UIs like this
| is illegal in some parts of the world? I have no idea asking if
| someone is more knowledgable than me (e.g. europe).
| CGamesPlay wrote:
| I don't think developing a UI that isn't accessible is ever
| illegal? I believe the illegal thing is providing a _service_
| that doesn't have "reasonable affordances". For example,
| providing a "m.coolco.io" where that subdomain has
| fundamentally the same service in an accessible fashion is
| allowed.
| alex_suzuki wrote:
| For government services, it can be a legal requirement for
| them to be accessible.
| PaulHoule wrote:
| Isn't Windows fundamentally GPU based since WDDM came with
| Windows Vista?
| incrudible wrote:
| _Fundamentally?_ No. It 's really just compositing whatever
| buffers represent the content of the windows, but those may
| well be wholly or partially rendered on the CPU. Almost nobody
| does text rendering on the GPU, vector graphics are often CPU-
| based as well.
| audidude wrote:
| GTK 4 renders with OpenGL, has native a11y built-in, styled with
| a comprehensive CSS engine, runs on Linux/bsd/Mac/windows, has
| bindings for almost every noteworthy language...
| traverseda wrote:
| > has native a11y built-in
|
| What does this mean? `a11y` isn't some standard for making your
| app accessible, in fact it mostly seems to be guidelines for
| how to make your website more accessible. Is there an `a11y`
| native app checklist somewhere that Gnome is compliant with
| (and presumably QT/kde isn't?). Am I missing some protocol
| Gnome is implementing that QT/kde isn't?
|
| Legitimate question, not snark, I've looked into in the past
| and I really don't see what Gnome's doing to comply with this
| that everyone else isn't.
| audidude wrote:
| GTK 4 has built-in support for at-spi in it's native controls
| and provides GtkAccessible interfaces which you can use to
| make custom widgetry accessible.
|
| > I really don't see what Gnome's doing to comply with this
| that everyone else isn't.
|
| That's because Sun and GNOME created those layers two decades
| ago for Xorg.
| aninteger wrote:
| I've heard that native a11y support only works on GNOME,
| although I don't know for sure since Orca is just another GTK
| application. I believe a11y support was available starting back
| in GTK 2.x but I don't know ATK's full history.
| traverseda wrote:
| QT also works alright with screen readers for what it's
| worth.
| c-smile wrote:
| > GTK 4 renders with OpenGL
|
| It may do that but generally it still uses Cairo.
|
| BTW, I would not rely long run on OpenGL existence on Windows
| and Mac (especially).
|
| In any case GTK working on Windows and Mac is an artefact
| rather than something really usable.
|
| I suspect that efforts to support such multiplatform feature
| makes Linux native GUI development less developed.
|
| Generally Linux cannot be considered as a desktop OS as it has
| no stable and uniform system UI (window manager, graphics,
| etc.)
| audidude wrote:
| > It may do that but generally it still uses Cairo.
|
| Uh, no it doesn't. I literally wrote the GL renderer, I know
| what I'm talking about.
|
| > In any case GTK working on Windows and Mac is an artefact
| rather than something really usable.
|
| Maybe pre-4, which is why it got redesigned.
|
| > I suspect that efforts to support such multiplatform
| feature makes Linux native GUI development less developed.
|
| What?
|
| > Generally Linux cannot be considered as a desktop OS as it
| has no stable and uniform system UI (window manager,
| graphics, etc.)
|
| Huh? You're out of your mind.
| c-smile wrote:
| > Huh? You're out of your mind.
|
| Do we have such basic thing as `MoveWindow(wnd,x,y,w,h)` on
| GTK?
|
| Seems like still not:
|
| - https://stackoverflow.com/questions/58103333/set-frame-
| posit...
|
| - https://stackoverflow.com/questions/65508569/how-to-move-
| my-...
| nsajko wrote:
| This is a part of Wayland fanaticism:
|
| https://discourse.gnome.org/t/how-to-center-gtkwindows-
| in-gt... As I've said above, not all
| windowing systems have a concept of global
| coordinates or allow direct positioning of the windowing
| from an application. Wayland, for instance, does
| not.
|
| The purpose of the Wayland project was clearly to impose
| a set of weird limitations like this one on everyone. The
| best explanation I can think of is that it's all part of
| some cult at Red Hat.
| smoldesu wrote:
| Are they wrong? SwiftUI doesn't have a concept of window
| positioning coordinates, but that doesn't mean that MacOS
| isn't a desktop OS. They just implemented their windowing
| stack differently, and honestly I think the Quartz method
| is a lot nicer.
|
| There's a lot to complain about with Wayland, but this is
| one of the few changes that seems rooted in logic rather
| than laziness.
| c-smile wrote:
| SwiftUI is not a desktop UI by its concept. It is very
| close to Web Fronted UI - _document_ UIs running inside a
| sandbox.
| smoldesu wrote:
| Alright, let's go up a layer of abstraction then. Where
| does the Quartz system expose windowing coordinates the
| the developer?
| audidude wrote:
| How does that make it "not desktop"?
|
| This is literally the dumbest argument ever. It's one
| extra line of code. If you have the GDK X11 backend, get
| the XID and call MoveWindow() yourself. If you have a
| macOS Window, call [NSWindow setFrame]. I don't see why
| you expect GTK to be your abstraction for insecure UI
| practices.
|
| The compositor should be in control of this policy, not
| applications. Which is why that API isn't there any more.
| c-smile wrote:
| Glad you've mentioned "GDK X11 backend".
|
| It means that GTK is not a desktop API.
|
| X11 is not a desktop API either as you cannot build with
| it GUI consistent with other applications on target
| platform.
|
| Linux is not a desktop OS. By any means. Dot.
|
| Until we have linux_move_window() function.
| smoldesu wrote:
| You're right, Linux isn't a desktop OS. Neither is Darwin
| or NT.
|
| To make an OS you need a system of desktop-providing
| tools. _Windows_ is an operating system that provides a
| desktop stack to the NT kernel. MacOS is an operating
| system built on Darwin and Mach. Following this line of
| logic, there are dozens of opinionated Linux
| distributions that qualify for your definition of desktop
| OS. (RHEL, Fedora, Ubuntu, KDE Neon, et al.)
|
| Linux systems are no different than Darwin or NT ones if
| you compare like systems. If you spend all day comparing
| kernel features to OS features, I don't think you're
| going to make any meaningful discoveries.
| c-smile wrote:
| Neither of RHEL, Fedora, Ubuntu, KDE Neon have APIs that
| GUI developers can rely on.
|
| To be short:
|
| First Linux company that will decide to provide stable
| API similar to `WndProc(window, params)` (Windows) or
| `class_addMethod(window,@selector)` (Mac/iOS) will win
| Linux Desktop war - will make the Linux GUI for years to
| come.
|
| If someone knows such company - please let us know, at
| least I am willing to participate.
| smoldesu wrote:
| WNDPROC is part of Win32, not the kernel. You seem to
| have mixed something up.
|
| If we're talking about kernels with windowing
| capabilities, there are none. If you're asking about OSes
| with a reliable, documented graphics stack, RHEL is
| literally what you're looking for. It's the shitty,
| "Windows-ified" model of desktop development, and almost
| everyone ignores it for desktop use. You might argue that
| it doesn't qualify, but you can't argue that nobody has
| tried it before.
| c-smile wrote:
| > WNDPROC is part of Win32,
|
| Users and UI developers don't care. It is part of OS and
| it is always there.
|
| The only things I know for sure are:
|
| 1. Desktop OS is not a distribution but set of popular
| GUI applications that work on that OS natively including
| tested specifically on it.
|
| 2. Lack of stable window and graphics API leads to
| fractured GUI developers community. Keeping in mind that
| number of Linux GUI developers is in magnitude of times
| less than for Windows/MacOS, makes Linux GUI perspectives
| very grim.
|
| In general "desktop Linux" is not fair to its most loyal
| users because of these. Users are forced to pay extra
| price for hardware just to be able to run those Electron
| applications - no real native options for similar
| functionality.
| smoldesu wrote:
| > Desktop OS is not a distribution but set of popular GUI
| applications that work on that OS natively including
| tested specifically on it.
|
| That's what a Linux distribution is.
|
| > Keeping in mind that number of Linux GUI developers is
| in magnitude of times less
|
| It's roughly proportional to the number of people using
| Windows or MacOS for serious networking applications,
| yes.
|
| > Users are forced to pay extra price for hardware just
| to be able to run those Electron applications
|
| What are you even talking about anymore? Linux is not
| responsible for shitty Electron apps, Windows and MacOS
| is. People wanted a univeral runtime that worked across
| both operating systems - neither Microsoft nor Apple
| budged. So the browser it is - it just came with the
| side-effect that other browser-enabled platforms can also
| run Electron. Linux is not forcing you to use bad
| software because Apple and Microsoft can't work together
| to fix desktop computing.
|
| If you don't like the Linux desktop, fine. Go use 9front
| or something. It still exists though, you cannot deny
| it's usability because you don't like the way it does
| something. I dislike the way MacOS handles it's desktop,
| but that doesn't mean it's not a desktop OS.
| simion314 wrote:
| All the news about GTK were about big projects dumping it in
| favor of Qt or other, did anything changed significantly? I
| only seen it used cross platform for extremely basic GUIs for
| Python apps. It also looked like shit==GNOME on Mac and
| Windows.
| dekhn wrote:
| Qt also supports GPU rendering. Ultimately Qt is based around
| a software rasterizer that can also be implemented using
| hardware primitives. It handles the abstraction better than
| any other framework I've used.
| audidude wrote:
| GTK 4 allows widget to snapshot into a render-tree, which
| has OpenGL, Vulkan, and Cairo for software fallback
| rendering. It also provides diff'ing for minimal damage and
| all the other important bits.
| traverseda wrote:
| Is there anything you specifically think QT is missing?
| traverseda wrote:
| Well new GTK is claiming that they're making it easier to
| write non-gnome GTK apps. What that looks like to me is that
| a lot of functionality is moving into a gnome-specific
| library (libadwaita) and you'll need to find another provider
| for those gnome specific widgets and functions and stuff. Pop
| OS makes another add-on library like that.
|
| Meanwhile QT continues to work great and they've largely
| solved the licensing issues that were a problem in the past.
| It's also a comparatively stable project and supports targets
| like android, iOS, rendering directly on to buffers (you can
| embed it into games). If you're starting a new project I
| think QT should be your default unless you have a good reason
| to use GTK.
|
| I'm not qualified to assess the code quality of either
| project, but I have been impressed by QT for the few things
| I've needed to build a gui for. Things just work but if you
| need really specific functionality (like rendering directly
| to a framebuffer with no display server on embedded linux)
| it's not too hard to make it work. I'm honestly a bit
| mystified as to why anyone would choose GTK for a new
| project, unless they were specifically targeting the Gnome
| desktop and only the Gnome desktop.
| simion314 wrote:
| True, I seen also a lot of Qt usage in proprietary Desktop
| apps and I was surprised. In my work Deskop apps are no
| longer popular, everything is a website now, SPA.
| audidude wrote:
| > a lot of functionality is moving into a gnome-specific
| library (libadwaita)
|
| Sort of?
|
| Mostly just design patterns that are explicit to GNOME. And
| that is primarily a response to people being (rightfully or
| not) unhappy that too much GNOME was ending up in GTK
| itself.
| audidude wrote:
| I have no idea which project's you're alluding to, but I'm
| certain they weren't on GTK 4, which is what this thread is
| about.
| simion314 wrote:
| I was thinking at Wireshark, LXDE and a very popular
| Youtube Video where Linux Torvalds explains that GTK is so
| bad that the GUI app he was creating for diving switched to
| Qt because even his RedHat friends were unable to help him
| to do some ncie things in GTK. I never seen some cross
| platform app switching from Qt to GTK because GTK is
| better.
| zokier wrote:
| > Traditionally, if you wanted to write desktop software, you
| would use the operating system's APIs to create a user interface.
| That's why Windows apps looked like Windows apps, and Mac apps
| looked like Mac apps
|
| That is oft repeated but having lived through those times that
| seems just untrue, at least on Windows which was far more
| dominant than it is today. So many applications, big and small,
| were using completely custom UIs instead of native things. From
| WinAmp to Microsoft Office to all of Adobe/Macromedia suites to
| all cross-platform apps using GTK/Qt (or god forbid, Java). One
| particular example is the pre-electron Spotify app, which was
| wonderful small single-exe thing.
| danielvaughn wrote:
| Considering how GPU-backed UI is viewed as a performance win, I
| wonder why Flutter web performs so poorly.
| dahart wrote:
| Using GPUs only _enables_ high performance. It definitely
| doesn't guarantee it. ;)
| mhoad wrote:
| Flutter web is in the process of moving to a new rendering
| pipeline that is C++ compiled to WASM and backed by WebGPU.
|
| I don't think they are going to have troubles performance wise
| for long.
|
| This is actually an interesting read I came across by someone
| from their team recently which poses some hypothetical
| directions they could take things in the future depending on
| how the web platform evolves over time.
|
| https://docs.google.com/document/d/1peUSMsvFGvqD5yKh3GprskLC...
| HellDunkel wrote:
| Independence of screen resolution and vector graphics do not
| align very well with rendering via OpenGL. And what if your app
| makes heavy use of rendering 3D on the GPU? You would better stay
| away from creating too much workload for the UI.
| graiz wrote:
| The thing that holds many web UI's back is the web/html DOM and
| the respective CSS recalculations. You get get close to 1000x
| performance on web apps if you make sure the DOM will never cause
| a UI block or re-computation on the layout. The web is great for
| simple apps but if you're building anything complex you have to
| play a bunch of tricks to get web UI to work. Going full GPU and
| building your own controls is certainly possible but I'm hoping
| there's a middle ground.
| The5thElephant wrote:
| This would be way easier if everyone didn't default to using
| React or even worse React-Native-Web. Not only do you have to
| deal with handling CSS recalcs (not so bad on its own), you
| also have to deal with React rerendering the entire DOM when
| you don't want it to and then the layers of div-soup and absurd
| style abstractions React-Native-Web uses.
|
| Tauri + Vue/Svelte/Solid/Etc seems like a great way to build a
| decent desktop and web app, but you just don't have teams
| picking anything but React React React React.
| pjc50 wrote:
| Do we have "transactional" DOM yet? The ability to batch up a
| bunch of stuff and _then_ hit the UI update.
| bartq wrote:
| We could have amazing HTML/CSS performance if we had
| programmable "HTML to bitmap" caching. But: a) it's not secure
| if we can render HTML to bitmap and send over network, b) cache
| invalidation problem - we'd have to invent mechanisms to lock
| HTML against further changes or track changes and rerun "HTML
| to bitmap" procedure.
|
| All this is not suitable for web, but for thing like Elector -
| hell yeah. HTML to bitmap extension would be really welcome.
| Then you could render UI in WebGL or WebGPU. Adobe Flash had
| something similar if I recall correctly.
| AshleysBrain wrote:
| AFAIK modern browsers all use GPU accelerated rendering, so using
| HTML/CSS does give you a GPU-backed user interface.
|
| Also desktop PWAs are IMO underrated and a really good way to
| distribute a desktop app without having to bundle a whole browser
| engine with your app. We do this for Construct
| (www.construct.net) - a complete game development IDE - and I
| think it's worked out great.
| starburst wrote:
| Saying HTML/CSS is "GPU-backed" gives the wrong impression.
|
| What's the point of saying a UI is GPU-backed (which usually
| means a UI hand-crafted using mesh, texture atlas, shaders,
| minimal draw calls, etc.) if it's also grouped together with
| one of the most bloated and inefficient means to output pixels
| on the screen (HTML/CSS).
|
| I love the web, not hating at all, but this is comparing apple
| and oranges.
| dahart wrote:
| The point is that GPU-backed is faster than not, which is why
| your browser (and your OS) is doing all those things except
| maybe hand-crafted meshes (and I don't agree that hand-
| crafted meshes is the usual definition of a GPU-backed UI).
| The GPU-backed UI does benefit from simple things like faster
| scrolling, especially on high DPI displays, in addition to
| speeding up the rendering of new pixels for
| HTML/CSS/SVG/image/video elements. The point is the browsers
| are actively trying to fix the inefficiency you're referring
| to, and it might now be further along than you think.
|
| BTW, the GPU-backed browser UI and GPU-backed OS UI, these
| come with a bunch of things that someone's hacked UI doesn't,
| namely transparent software fallback when a GPU isn't
| available, full compatibility with the UI spec, accessibility
| support, and tolerance for a wide range of GPUs. People
| making their own UIs directly almost never do these things
| (I'm guilty as charged), and as such their UIs are often some
| combination of more brittle, device dependent, and limited to
| a narrower audience.
| von_lohengramm wrote:
| > GPU-backed is faster than not
|
| Not necessarily. I've seen far too many cases where GPU
| acceleration actually made things "slower." Throughput !=
| latency
| starburst wrote:
| Sure hand-crafted mesh might push the envelop too much (I
| work in game dev) but GPU-Backed UI referred to a
| specialized form of UI.
|
| Have you read the article? They specifically mention how
| nowadays app are just a web page instead of being a GPU-
| Backed UI.
|
| Trying to push the narrative that HTML / CSS is somehow a
| GPU-Backed UI is almost comical.
|
| The browser is GPU-accelerated, it is not what anyone in
| the industry would consider a GPU-Backed UI.
| master-lincoln wrote:
| > Trying to push the narrative that HTML / CSS is somehow
| a GPU-Backed UI is almost comical.
|
| To me it rather seems like there are different
| definitions of what GPU-backed means.
|
| >The browser is GPU-accelerated, it is not what anyone in
| the industry would consider a GPU-Backed UI.
|
| In which industry?
|
| As a former web dev I would consider any calculation GPU-
| backed if it is utilizing a GPU.
|
| With that definition and considering html/css is used to
| build user interfaces, rendering in modern browsers could
| certainly be called GPU-Backed UI.
|
| If this is a term that already has a different meaning to
| just the words it is composed of, that's fair to mention.
| But it doesn't help a discussion to escalate using
| wording such as "Trying to push the narrative ... is
| almost comical"
| starburst wrote:
| It is stated right there in the article that "GPU-Backed
| UI" is one of the alternative to the mainstream way of
| building app nowadays that are basically just web page.
|
| Now people argue that web page are "GPU-Backed UI".
|
| I get the word might be misleading, but the article is
| talking about a specific way of building UI using the
| GPU, which does not include web page, so trying to fit
| the web into that category makes no sense.
|
| In some loose sense every UI is "GPU-Backed", so how do
| we name that specific way of building UI that the article
| is mentioning? They choose "GPU-Backed UI" which if
| anyone would have told me those words I would immediately
| know what they are talking about and I would've never
| included HTML/CSS in that category (even if in some broad
| loose sense of the words it does fit).
| master-lincoln wrote:
| That's a good point. So up to which abstraction level
| would you consider "GPU-Backed UI" to be the right term?
| Looks like it doesn't end with low level APIs like OpenGL
| as the article also mentions e.g. the rust framework
| https://github.com/emilk/egui
|
| Where to draw the line?
| starburst wrote:
| Drawing a line is always hard (when did in our evolution
| we become sentient?), but HTML / CSS is probably the
| furthest you can be from that line in modern UI tooling.
|
| As long as the UI or it's underlying engine was built
| with GPU in mind and not as an after-thought to
| accelerate it, it can be safe to say it probably fit the
| "GPU-Backed UI" definition.
|
| If you can easily attach a Shader (https://github.com/emi
| lk/egui/blob/master/examples/custom_3d...), it is also a
| dead giveaway that it is a "GPU-Backed UI".
|
| WebGL is different from HTML / CSS. You could have a
| "GPU-Backed UI" running in WebGL.
| sebzim4500 wrote:
| >Trying to push the narrative that HTML / CSS is somehow
| a GPU-Backed UI is almost comical.
|
| The fact that Electron apps are GPU backed is
| indisputable. Whether they are _good_ GPU-Backed UIs is
| an entirely different question.
| starburst wrote:
| All I'm saying is that the article refer to "GPU-Backed
| UI" as a very specific way of building UI which doesn't
| include HTML / CSS.
|
| Which is very different than saying that it doesn't use
| the GPU. Virtually every UI solution is "GPU-Backed", but
| when I hear the term "GPU-Backed UI", I know exactly what
| type of UI architecture they are talking about.
| dahart wrote:
| Yes sir, I did in fact read the article. I used to work
| in game dev. After that I built a WebGL-based whiteboard
| a lot of people used. Now I help make GPUs. I think it's
| great if you agree with the author.
|
| > The browser is GPU-accelerated, it is not what anyone
| in the industry would consider a GPU-Backed UI.
|
| Now this is a different argument than what you said
| above. I agree that browsers are what make it GPU-backed,
| not the HTML/CSS spec per se. But HTML/CSS is becoming
| _effectively_ a GPU-backed UI in practice, by virtue of
| the fact that all the browsers use GPUs to render HTML &
| CSS. This is also true of the UIs provided by Windows and
| MacOS and Linux - the UI isn't required to use the GPU,
| but the OSes will use GPUs for performance when possible.
| This is all anyone here actually means when they call
| HTML & CSS a GPU-backed UI.
|
| Is there a meaningful difference once there doesn't exist
| a modern browser that doesn't use a GPU to significantly
| render web apps?
|
| What I reject is the notion that "GPU-backed" is limited
| to more than what the words mean literally. If the
| implementation is using a GPU to render the UI elements,
| then it's GPU backed. It doesn't need to be hand-coded,
| or a one-off, or using shaders, or guaranteed by the UI
| spec.
| jakear wrote:
| With that definition, could you give an example of
| something that _isn 't_ GPU-backed? As far as I'm aware
| the GPU is always there at the bottom of the stack,
| pushing bits to the monitor. Heck in many desktops the
| monitor plugs directly into the GPU!
|
| Not saying I disagree HTML can be fast, in fact I think
| it's quite a bit faster than the vast majority of
| homecooked solutions. But tautological categorization
| isn't very helpful.
| dahart wrote:
| Now this is a distinction worth discussing, you're right.
| I do assume that "GPU backed" means you're interfacing
| with the hardware acceleration features of the machine,
| and not only using the frame buffer as pixel storage.
| This is what I mean when I say all the major browsers and
| OSes are "GPU backed", and I'm not trying to sneak in a
| silly tautological straw man about the GPU being the only
| connection to the monitor. The browsers and OSes are
| making increasing use of GPU APIs for hardware
| acceleration like Vulkan, Metal, and DirectX, and that is
| what I think of when I read or say "GPU backed".
| starburst wrote:
| > What I reject is the notion that "GPU-backed" is
| limited to more than what the words mean literally.
|
| This is exactly what I'm arguing against. We name stuff
| to represent something. The article very explicitly
| mention "GPU-Backed UI" as one of the alternative to app
| that are made with web tech (HTML / CSS). Because it is
| talking about a very specific way to build UI. And the
| name they choose is "GPU-Backed UI", which is what anyone
| would use too.
|
| But you are arguing that a Web Page / Browser / HTML /
| etc is "GPU Backed UI" and should be group together. But
| then we have the issue of how we name that type of UI
| that they are talking in the article which is very
| obviously something different than what a UI built using
| a browser would be.
| dahart wrote:
| Why do you say it's a "name"; why are you assuming &
| asserting the words "GPU backed" are a term of art? Do
| you have any references to support this assertion? Why do
| you disagree that UI implemented with a GPU can be called
| "GPU backed"; aren't overloaded meanings common enough to
| consider them possible and reasonable? What, exactly, is
| your definition of "GPU backed" then? Does it make room
| for the fact that OpenGL can run on a CPU? I would also
| argue that Scaleform on the Wii is GPU backed UI, for
| example.
|
| This thread is now hung on a gate-keeping technicality,
| and has lost the actual point, which is that, regardless
| of what you want to call it, html & css rendering have
| been getting faster due to the increasing use of GPUs,
| which is narrowing the gap between bespoke UIs and
| interfaces made with more standard toolkits.
| pjmlp wrote:
| Not really it is a bit more subtle than that, unless you do
| some magic incantations with CSS, like z-ordering or
| transforms, it may bet that software rendering is used instead.
|
| There are a couple of talks regarding CSS tricks to force GPU
| redrawing.
| AshleysBrain wrote:
| I think that was the case several years ago but I thought
| these days much more of the browser rendering pipeline was
| automatically GPU accelerated. It's hard to find up-to-date
| references (would appreciate any links) but for example going
| to chrome://gpu shows it using GPU acceleration for both
| compositing and rasterization (including "multiple raster
| threads"), as well as the usual suspects like video decoding,
| canvas and WebGL.
| pjmlp wrote:
| Don't be fooled by canvas/WebGL support.
|
| I remember seeing this kind of content on Google IO and
| WebDev talks.
|
| Would need to also go treasure hunting to find them.
| AshleysBrain wrote:
| Treasure hunting to find an API widely supported on 95%+
| of devices?
| pjmlp wrote:
| Treasure hunting the Google IO/WebDev talks about the CSS
| magic incantations that force GPU rendering.
|
| Here is an old article,
|
| https://www.chromium.org/developers/design-documents/gpu-
| acc...
| esperent wrote:
| Do we know for sure if this is still true in modern browsers?
|
| Browser makers are constantly updating their rendering
| engines and these pieces of wisdom do go out of date so it's
| important to verify them every few years.
| pjmlp wrote:
| As answered on sibling comment I remember seeing this on
| Google talks, not sure about latest versions without going
| hunting for this information.
| airstrike wrote:
| > We do this for Construct (www.construct.net) - a complete
| game development IDE - and I think it's worked out great.
|
| Mind sharing what you used for backend / frontend of your
| desktop PWA? I'm planning to work on a similarly complex UI and
| have been searching around for best-in-class (or best-trade-
| offs) technologies today
|
| Any books, tips, tutorials, whatever you can share that
| inspired your architecture decisions would be ___immensely___
| appreciated.
| AshleysBrain wrote:
| We have an in-house UI library, mainly because we started
| writing code in the mid-2010s and there was nothing good out
| there at the time. Maybe that's changed but while I haven't
| researched it thoroughly, I also haven't come across anything
| that looks like it does everything ours does. But our UI lib
| aims to be as simple and straightforward as possible,
| standard model-view-controller, no virtual DOM, making full
| use of HTML and CSS features. Personally I view a lot of
| modern CSS frameworks with suspicion as they all seem to
| massively overengineer this stuff. We have a whole desktop
| IDE running on basically old fashioned desktop app
| engineering style but on top of HTML, and it works great.
|
| Apart from a few extra services on the side, the main app is
| all static resources so no real backend to speak of. Again,
| engineered like a traditional desktop app with everything
| running locally, which also means the UI always responds
| promptly (no waiting for network) and it all works offline.
|
| Maybe I should write up a blog post about how it all works,
| but not sure how many people out there want to hear more!
| runlaszlorun wrote:
| I'd love to see a writeup if yyou had a chance to write
| one!
| matlin wrote:
| This is pretty much what Flutter is. Except Flutter has system-
| specific UI's created by Google so each app can have that native
| look depending on where it's running.
| ArtWomb wrote:
| Impeller. New shader compilation backend. Heroic coding.
| Metal/Vulkan/WebGPU translators. The bar is pixel perfect
| across gpus which makes it so challanging. My current favorite
| open source project: Android & Win ARM ports ;)
|
| https://github.com/flutter/flutter/wiki/Impeller
| actionfromafar wrote:
| _Almost_ that native look. Diluting what a native app feels
| like, but that price is something Google is willing for you to
| pay.
| [deleted]
| ognarb wrote:
| Another example of a GPU-Backed user interface is QtQuick.
| ClumsyPilot wrote:
| No mention of Unity or games in General?
|
| Thats what every game does - draws a UI directly on the GPU. Game
| engines like Unity have in-built frameworks for creating UI and
| then can package the same application for Windows, Linux,
| playstation, etc.
|
| Many people have (ab)used it to create cross-platform software.
| It's very high-performance, you just get large applications and
| load times.
| paulgb wrote:
| > Many people have (ab)used it to create cross-platform
| software. It's very high-performance, you just get large
| applications and load times.
|
| (author here) Part of the appeal of posting here is that I
| learn stuff like this :) Are there some good examples of these
| that come to mind?
| jb1991 wrote:
| A lot of cross-platform desktop applications have also been
| written in Godot.
| arminiusreturns wrote:
| This is one more reason I am loving godot (nightly compile
| from main). I see this as a real opportunity to learn an
| eco system I can do more than just make games with.
| emaro wrote:
| > It takes a lot of work to make a usable UI out of triangles,
| so most applications did not go this path. One big exception
| was games, which were already heavy GPU users, and usually had
| minimal UIs. The 3D renderer Blender is another example.
|
| Only briefly, but seems covered to me.
|
| Edit: now I get it -- they mention games, but not game engines
| as a means of creating other applications.
___________________________________________________________________
(page generated 2023-02-07 23:02 UTC)