https://moxie.org/2024/09/23/a-good-engineer.html

  * ABOUT

  * LIFE

  * PROJECTS

  * BLOG

  * 

     
  * 

     

The magic of software; or, what makes a good engineer also makes a
good engineering organization

Sep 23, 2024

The people who create software generally refer to themselves as
software engineers, and yet if they graduate from university, it is
typically with a degree in computer science. That has always felt a
little strange to me, because science and engineering are two pretty
different disciplines - yet we for the most part seem to take such an
obvious contradiction for granted. However, I think there is
something uniquely magical about software, and part of that magic
might stem from this tension in how we define it.

At first glance, software seems like a straightforward engineering
practice. After all, it exists within a fully known universe - the
computer - of our own making. This is in stark contrast to
disciplines that are more obviously science, like biology or physics,
which exist within domains that we did not create and do not fully
understand. Unlike computers, which are fully understood, those
disciplines are in large part about the search for understanding.
Science, in other words, is often occupied with the process of
discovering how something works - which we don't need to do with
computers.

Since we understand everything about the computer and don't need to
discover its workings, software seems much more straightforward. With
nothing to discover, we might say that software is the engineering
practice of combining and assembling what is available from the
complex system of computing in order to manifest a given vision.

Start with vision, then do engineering Start with a vision, and then
doing the engineering to manifest that vision.

However, I don't believe that is the full story. I want to suggest
that the relationship between vision and engineering in software is
often intertwined and bidirectional rather than linear, and that the
whole of software development is actually full of discovery even
though it exists within a completely known universe.

An intertwined and bidirectional relationship between vision and
engineering Vision and engineering as a more intertwined and
bidirectional relationship.

Consider these computer graphics animations from the era when I was
first getting interested in software:

Plasma animation
Falls animation

These are from an extremely primitive era, and yet they are beautiful
and compelling. The interesting thing to me about these animations is
that they are not actually "animating" - at least not in the
conventional sense, where the value of each pixel is calculated and
rendered 30 times a second. It certainly looks like that's what is
happening, given all the dynamic movement, but that would not have
been feasible based on the hardware constraints at the time.

Instead, these were actually images that made use of a technique
known as color cycling. Color cycling exploited the fact that
graphics systems like VGA used "indexed" color. Indexed color systems
were a memory optimization technique for representing each pixel on
the screen with a single 1-byte color value rather than a full 3-byte
RGB value, along with a separate "color pallet" that mapped these
1-byte color values to a full 3-byte RGB value.

Plasma animation Each pixel can be represented with a 1-byte color
value.
Falls animation A separate table maps the 1-byte color indexes to
3-byte RGB values.
Falls animation The resulting colors are rendered for each pixel.


Someone who understood how indexed color systems worked realized that
it would be possible to create impressive (for the time) video
effects by carefully specifying color values for an image and then
intelligently cycling the 256 colors in the color pallet to create
the appearance of animation.

Ocean animation

Combining tricks like this are what enabled the famous Amiga boing
ball demo:

Amiga boing ball animation

I think this was a somewhat non-obvious result. Indexed color systems
weren't created for this purpose, but software developers who
understood them were able to use them to create something
unexpectedly great.

Computing in general is a long history of non-obvious results. I
mean, if I had been there at the beginning of it all, I'm not sure I
would have predicted that by quickly turning electricity on and off,
we could get... all this?

Much like with pure science disciplines, software's history has been
a history of discovery. However, while understanding is the object of
discovery in biology or physics, understanding and engineering is
often the basis for discovery in software. While software development
is indeed the process of combining and assembling resources to
manifest a given vision, deep understanding of the resources and all
the ways to use them also inform the vision in a bidirectional
relationship. It's not always as simple as starting with a vision and
then using the resources available to build it. They often emerge
together or in an intertwined way, and things continually leap
forward over and over.

If we consider the animations above to be a software product, it is
clear that the product did not begin with a vision that someone had
for animations which looked this way. It was an understanding of the
engineering that informed the vision, and vice versa.

I'm using color cycling as an example because it's visual and so old
as to be uncontroversial, but it's a pattern that I've seen appear
again and again throughout the course of my involvement with
software: deep understanding of tools and technology yields a new
vision, unexpected results, insight, or a solution of some kind (big
or small!). That doesn't always mean a new product or demo, and more
often than not is simply an advance within a project that only a
small group of people will ever take note of, but it almost always
starts with the understanding part.

I feel like we might be losing sight of the understanding part.

Abstractions: Building More, Creating Less

Software development has the benefit of relying on abstraction
layers. It would be incredibly arduous to have to implement all of
the logic associated with storing data to disk every time one needs
to store data to disk, so abstracting that out into a reusable
storage interface like an RDBMS or a KV store is, of course, greatly
valuable. The same can be said for the value of an HTTP library,
sockets interface, C compiler, a game engine, or any other
abstraction layer.

However, there are two ways of interacting with an abstraction layer:
as shorthand for an understanding of what it is doing on your behalf,
or as a black box. I think engineers are most capable and most
effective when abstraction layers serve as shorthand rather than a
black box. Computing is so complex that these interfaces are never
perfectly clean (and never will be), so to use them as a black box
without understanding what happens within them can lead to negative
outcomes which compound with the size/complexity/scale of the
application.

Most people who work with software generally have a sense of this.
But to me, there also seems to be something more going on, which is
related to that intertwined relationship between engineering and
vision: when we see everything as black boxes, it can be difficult to
have any vision at all.

For instance, sometimes we create abstraction layers that allow
people to create things on top of them explicitly without having to
understand anything beneath them. We call those "platforms." The
expectation is that when we create abstraction layers like that, we
should see an explosion of creativity, since now people can focus
only on the creative aspects of what to build rather than the
technical aspects of how to build it.

This seems obviously true for some endeavors outside of computing: a
filmmaker can use a camera to make great films without understanding
in detail how to build the camera, and there is probably not much of
a predictable relationship between knowledge of how the camera works
and quality of the resulting film. We may be tempted to extrapolate
from this into the world of computing and say that by creating
something like a game engine, we should expect to see an explosion of
creativity in games, because now game makers can focus on the
creative aspects of building games without having to understand the
technical aspects of how to build a game - in the same way that a
filmmaker doesn't have to build a camera.

However, I think the problem with metaphors like these is that a
camera is not actually an abstraction layer for a film. A story does
not simply sit on top of a camera. There is a lot of other really
complicated stuff that a filmmaker really does have to understand
about making a film that is necessary in order to bring a story to
life.

When we apply this logic in the realm of computing by creating
explicitly black-box abstraction layers, I think we do actually see
an explosion of more stuff, but it is often a lot more mediocre stuff
.

A graph of games vs critically acclaimed games over time

As we've made it easier to build games, we have certainly seen more
of them. But the number of highly rated games (in this case as
recorded by metacritic) does not seem to be increasing over time.

To me, in complex ecosystems like computing, there just seems to be
some kind of ongoing relationship between deep understanding of how
the tools we use to create things function and the quality or
creativity of the output we get as a result.

I think we can only truly create when we truly understand what we
have to create with.

When Teams Become Tupperware: Sealed Tight and Stale Inside

Maybe that much is obvious, but I think this applies in all kinds of
meta ways to the software industry as a whole. For instance, I think
many of the dynamics that make a good engineer are also the dynamics
that make a good engineering organization.

Engineering organizations today have ballooned to huge numbers of
people, but these huge engineering organizations don't exactly have a
reputation for high velocity output. Some of this is the result of
what happens with products at scale: it is just fundamentally faster
and easier to iterate, improve, or change a product with 100 users
than it is to do the same with a product of 100 million users.

However, having seen inside some large engineering organizations, I
don't think this explains everything.

Most people managing or building engineering teams today have a
familiar management philosophy. The general thesis being that
hierarchical decision making - where information flows upwards in the
form of reports, summaries, and executive summaries, and then flows
back downwards in the form of budgets, headcount, and priorities - is
too slow.

Instead, it is now more common to think that organizations should be
composed of autonomous teams, and the function of leadership in an
organization like that is to create "alignment" across all these
teams by ensuring that they are pulling in the same direction towards
the same vision:

Autonomous unaligned teams

--->

Autonomous aligned teams

These two ways of thinking about organizations (autonomous vs
hierarchical) seem really different at first glance. However, in one
respect, they are both essentially the same:

Start with vision, then do engineering

Both assume a relatively unidirectional relationship between vision
and engineering. The substantial difference is the way execution is
organized, but neither leave a lot of room for that bidirectional
relationship between engineering and vision to emerge.

To exacerbate this, organizing engineers into silos is very similar
to creating abstraction layers for an organization, almost like a
microservices architecture. Unfortunately these abstractions are
almost always treated as black boxes, both by peer teams and by
leadership teams, and much like in software, living in a world of
black box abstractions can make it difficult for anyone to have the
insights that are necessary for truly excellent outcomes.

A thought experiment I like to apply to an engineering organization
when considering its efficacy is to imagine that it is Skype at the
dawn of "mobile." Imagine that you're running Skype, that you have
all these people siloed into autonomous teams, and that you have the
foresight to see how important mobile is going to be. So you make
sure that everyone is aligned on mobile as the key, number one,
absolutely existential priority for the organization.

Even with everyone aligned, adapting to mobile is a big lift. The
existing product is 100% organized at every level around a desktop
model. The UX is centered around the concept of being "online" or
"offline," which no longer exists in a mobile environment. The
communication protocol is a semi-distributed mesh that is dependent
on being able to run in the "background" on a desktop OS, which won't
work at all on a mobile device. The code base is written in a way
that is not deployable on mobile OSes, etc etc.

Clearly things need to change. Almost everything needs to change. But
the change that every team needs to make is dependent on the change
that every other team needs to make. The product vision itself is
intertwined and bidirectionally informed from the engineering, but if
everyone in the organization sees every other part of the
organization as a black-box abstraction layer, I don't think a change
like that is going to happen.

While seismic shifts like the mobile revolution may seem rare, I
think organizations ill-equipped for radical change are likely
underperforming even in stable times. Just like with software, where
deep understanding is often the basis for discovery, an organization
has to truly understand itself to be primed for innovation. When
teams operate as impenetrable black boxes, vision becomes myopic, and
potential withers. Despite having immense talent, the large
engineering organizations of SV often structure themselves in ways
that isolate expertise. As a result, they're probably sitting on a
huge amount of untapped potential - creative and technical ideas that
never connect.

Fruit loops and false idols

I've noticed that aspiring entrepreneurs often fixate on studying the
current habits of successful entrepreneurs, hoping to mimic their
achievements. The problem with this logic is that, right now, whether
Jeff Bezos starts every day with a cold plunge at 5am or a bowl of
fruit loops at 10am probably has minimal bearing on the outcome of
his companies. If imitation is really the goal, people should
probably be looking at what he was doing in 1994 - which was probably
just working obsessively to the detriment of all else.

I think there is a similar fallacy for how we consider engineering
organizations. Many of today's "best practices" have been drawn from
long-established internet companies like Google. However, the problem
with copying their current practices on the basis of their success is
that most of those companies found near-invincible business models
that basically printed money, and so almost any organizational or
management practice developed or selected at random likely would
continue to be "successful" to some degree. Without much in the way
of selective pressure, I think the practices and cultures that have
developed at these companies more often reflect optimizations for the
benefit of various decision makers rather than optimizations for
actually creating something great. The most obvious example being the
size of the organizations themselves, which I think is much more
often the result of the benefits that accrue to the hiring manager
for having more "reports" than to any actual relationship with
capacity, ability, or velocity.

The people from those companies became the management hires at new
companies, and the people who learned there have become the
management hires at other new companies, and eventually we have come
to forget that there was any other way of doing things before all
this.

If you're building a company or an engineering team today, I think
the real opportunity lies in understanding that you're not bound by
the conventions that have been passed down as "standard."

The qualities that make a good engineer are often the same ones that
create a good engineering organization. Both start with deep
understanding as the basis for innovation - cultivating the curiosity
to look inside the black boxes. Both recognize that vision and
engineering are often deeply intertwined and mutually informative
rather than linear. The magic of both software and software
organizations comes from those moments where insight into how
something works sparks entirely new ideas about what it could become.

(c) 2012 Moxie Marlinspike

C