https://bkardell.com/blog/OffscreenCanvas.html

Author Information

Brian Kardell
[profile]
Betterifying the Web

  * Developer Advocate at Igalia
  * Original Co-author/Co-signer of The Extensible Web Manifesto
  * Co-Founder/Chair, W3C Extensible Web CG
  * Member, W3C (OpenJS Foundation)
  * Co-author of HitchJS
  * Blogger
  * Art, Science & History Lover
  * Standards Geek

Follow Me On...

  * Wordpress
  * Medium
  * Twitter
  * Github
  * Codepen
  * LinkedIn
  * Instagram (for art)
  * Fine Art America (art for sale)

Posted on 09/30/2021

Making the whole web better, one canvas at a time.

One can have an entire career on the web and never write a single
canvas.getContext('2d'), so "Why should I care about this new
OffscreenCanvas thing?" is a decent question for many. In this post,
I'll tell you why I'm certain that it will matter to you, in real
ways.

How relevant is canvas?

As a user, you know from lived experience that <video> on the web is
pretty popular. It isn't remotely niche. However, many developers I
talk to think that <canvas> is. The sentiment seems to be something
like...

    I can see how it is useful if you want to make a photo editor or
    something, but... It's not really a thing I've ever added to a
    site or think I experience much... It's kind of niche, right?

What's interesting though, is that in reality, <canvas>'s prevalence
in the the HTTPArchive isn't so far behind <video> (63rd/70th most
popular elements respectively). It's considerably more widely used
than many other standard HTML elements.

Amazing, right? I mean, how could that even be?!

The short answer is, it's just harder to recognize. A great example
of this is maps. As a user, you recognize maps. You know they are
common and popular. But what perhaps you don't recognize that it's on
a canvas.

As a developer, there is a fair chance you have included a <canvas>
somewhere without even realizing it. But again, since it is harder to
recognize "ah this is a canvas" we don't idenitfy it the way we do
video. Think about it: We include videos similarly all the time - not
by directly including a <video> but via an abtraction - maybe it is a
custom element or an iframe. Still, as a user you still clearly
idenitfy it, so in your mind, as a developer you count it.

If canvas is niche, it is only so in the sense of who has to worry
about those details. So let's talk about why you'll care, even if you
don't directly use the API...

The trouble with canvas...

Unfortunately, <canvas> itself has a fundamental flaw. Let me show
you...

Canvas (old)

This video is made by Andreas Hocevar using a common mapping library,
on some fairly powerful hardware. You'll note how janky it gets -
what you also can't tell from the video is that user interactions are
temporarily interrupted on and off as rendering tries to keep up. The
interface feels a little broken and frustrating.

For whom the bell tolls

For as bad as the video above is, as is the case on all performance
related things, it's tempting to kind of shrug it off and think
"Well, I don't know.. it's pretty usable, still - and hardware will
catch up".

For all of the various appeals that have been made over the years to
get us to care more about performance ("What about the fact that the
majority of people use hardware less powerful than yours?" or "What
about the fact that you're losing potential customers and users?"
etc), we haven't moved that ball as meaningfully as we'd like. But,W
I'd like to add one more to the list of things to think about here...

    Ask not for whom the performance bell tolls, because
    increasingly: It tolls for you.

While we've been busy talking about phones and computers, something
interesting happened: Billions of new devices using embedded web
rendering engines appeared. TVs, game consoles, GPS systems, audio
systems, infotainment systems in cars, planes and trains, kiosks,
point of sale, digital signage, refridgerators, cooking appliances,
ereaders, etc.. They're all using web engines.

Interstingly, if you own a high-end computer or phone, you're
similarly more likely to enounter even more of these, as a user.

Embedded systems are generally way less powered than the universal
devices we talk about often when they're brand new -- and their
replacement rate is way slower.

So, while that moderately uncomfortable jank on your new iPhone still
seems pretty bearable, it might translate to just a few (or even 1)
FPS on your embedded device. Zoiks!

In other words, increasingly, that person that all of the other talks
ask you to consider and empathize with... is you.

Enter: OffscreenCanvas

OffscreenCanvas is a solution to this. It's API surface is really
small: It has a constructor, and a getContext('2d') method. Unlike
the canvas element itself, however, it is neatly decoupled from the
DOM. It can be used in a worker - in fact, they are tranferrable -
you can pass them between windows and workers via postMessage. The
existing DOM <canvas> API itself adds a .transferControlToOffscreen
which will (explcitly) give you one back, and is in charge of
painting in this rectangle.

If you are one of the many people who don't program against canvases
yourself, don't worry about the details... Instead, let me show you
what that means. The practical upshot of simply decoupling this is
pretty clear, even on good hardware, as you can see in this demo...

OffscreenCanvas based maps

Using OffscreenCanvas, user interactions are not blocked - the
rendering is way more fluid and the interface is able to feel smooth
and responsive.

A Unique Opportunity

Canvas is also pretty unique in the history of the web because it
began as unusually low level. That has its pros and its cons - but
one positive thing is that the fact that most people use it by
abstraction presents an intersting opportunity. We can radically
improve things for pretty much all real users, through the actions of
comparatively group of people who directly write things against the
actual canvas APIs. Your own work can realize this, in most cases,
without any changes to your code. Potentially without you even
knowing. Nice.

New super powers, same great taste

There's a knock on effect here too that might be hard to notice at
first. OffscreenCanvas doesn't create a whole new API to do its work
- it's basically the same canvas context. And so are Houdini Custom
Paint worklets. In fact, it's pretty hard to not see the relationship
between painting on a canvas in a worker, and painting on a canvas in
a worklet - right? They are effectively the same idea. There is
minimal new platform "stuff" but we gain whole new superpowers and a
clearer architecture. To me, this seems great.

What's more, while breaking off control and decoupling the main
thread is a kind of easy win for performance and an intersting super
power on it's own, we actually get more than that: In the case of
Houdini we are suddenly able to tap into all of the rest of the CSS
infrastructure and use this to brainstorm, explore and test and
polyfill interesting new paint ideas before we talk about
standardizing them. Amazing! That's really good for both standards
and users.

Really interestingly though: In the case of OffscreenCanvas, we now
suddenly have the ability to parallelize tasks and throw more
hardware at highly parallelizable problems. Maps are also an example
of that, but they aren't the only one.

My colleague Chris Lord recently gave a talk in which he gave a great
demo visualizing an interactive and animated Mandlebrot Set (below).
If you're unfamilliar with why this is impressive: A fractal is a
self repeating geometric pattern, and they can be pretty intense to
visualize. Even harder to make explorable in a UI. At 1080p
resolution, and 250 iterations, that's about half a billion complex
equations per rendered frame. Fortunately, they are also an example
of a highly parallelizable problem, so they make for a nice demo of a
thing that was just totally impossible with web technology yesterday,
suddenly becomming possible with this new superpower.

OffscreenCanvas super powers!

A video of a talk from a recent WebKit Contributors meeting, showing
impressive rendering. It should be time jumped, but on the chance
that that fails, you can skip to about the 5 minute mark to see the
demo.

What other doors will this open, and what will we see come from it?
It will be super exciting to see!