https://rtpg.co/2021/06/07/changes-checklist.html

Raphael  
About Feed

The pedantic checklist for changing your data model in a web
application

2021-06-07 * Raphael

Let's say you have a web app with some database. In your database you
have an Invoice model, where you store things you are charging your
customers.

Your billing system is flexible, so you support setting up tax
inclusive or exclusive invoices, as well as tax free invoices! In
this model you store it as:

class Invoice:
   ...
    is_taxable: bool
    is_tax_inclusive: bool

You use this for a couple of years, write a bunch of instances to
your database. The one day you wake up and decide that you no longer
like this model! It allows for representing invalid state (what would
be a non-taxable, but tax-inclusive invoice?). So you decide that you
want to go for a new data model:

class TaxType(enum.Enum):
    no_tax = enum.auto()
    tax_inclusive = enum.auto()
    tax_exclusive = enum.auto()

class Invoice:
    tax_type: TaxType

You write up this new code, but there's now a couple problems:

  * Your database's data is all in the old shape! You'll need to move
    over all your data to this new data model.
  * You're a highly available web app! You're not gonna do any
    downtime (well, planned, anyways) unless you can't avoid it.
  * Your system is spread across multiple machines (high availability
    right?) so you have to deal with multiple versions of your
    backend code running at the same time during a deployment.

There's a whole list of steps you have to do in order to roll out
this change. There's a whole thing about "double writing",
"backfilling" etc. But there's actually a lot of steps when you end
up actually needing to make a backwards-incompatible change!

I feel like I know this list, but every once in a while I end up
missing one step when I go off the beaten path, so here's the list,
with every little step required, in all the pedantry.

An important detail here is that you need to roll out each version
one-by-one. You can have some parts of your system on Version 3 and
others on Version 4. But if you have some of your system on Version 3
and others on Version 5, you will hit issues with data stability.
This makes for a lot of steps! Depending on your data model, you
might be able to fuse multiple versions into one (especially when you
have a flexible system).

Version 0: Read/Write Old Representation

The initial version of your application is using the old model, of
course. This is your starting point, and it might not actually be
ready for us to start introducing a new model (especially if your
application <-> DB layer is particularly strict about what it
receives)

Version 1: Can Accept The New Representation

This version will be able to read your data and not blow up if the
new representation is present. This doesn't mean you are using the
new representation for anything! Just that you can handle it.

A lot of systems don't actually require this as a distinct step. You
can add a new column to a database and have existing queries continue
to work just fiine. But there are a couple places where you need to
be more careful. Some examples:

  * Adding a new value to an enumeration. If I only have
    tax_inclusive and tax_exclusive, I need to put into place
    no_tax-handling code before I start migrating data over to this
    (or having new rows use it).
  * systems with strict validation. A system might have error paths
    when a new key starts appearing in some JSON dictionary, so you
    might need to add preparation code for this.

Migration 1: Add The New Representation To The Database

For an SQL database, this usually is about adding a new column to
your database. Some databases might not need this step, and some data
model changes might not need this (for example if you are just adding
a new value into an enumeration, but the underlying data was stored
as a string)

Version 2: Write To Old + New Representation

This version of your application will start filling in both
representations on writes to your database. You still continue to
read from the old representation (so that writes that happened with
V1 of the application still make sense), and writing to the old
representation means that during your V1 -> V2 deploy, V1 reads don't
get stale.

Migration 2: Backfill The New Representation In Existing Data

For any rows that haven't been written to since you rolled out
Version 2, you won't have filled in the new representation of your
data (maybe a user just hasn't logged in for a while!). In order to
make sure the new representation is ready to be read, this migration
should go through all existing data and fill in the new
representation.

You need to do this after Version 2 is deployed, because if a Version
1 write happens during the migration, then the backfilled value will
actually be stale. And you need to do this migration before you begin
reading the new representation, so that old records can be properly
read.

Version 3: Read From The New Representation

You have now filled out the new field, so you can read from it!
However, you still need to be writing to both representations. Why?
Because Version 2 of your application is still reading from the old
field! During a deployment you'll still have machines on previous
versions, so you need to be compatible with coexisting, at least for
the duration of a deploy.

Migration 3: Remove Any Mandatory Constraints For Old Representation

This is sometimes not needed, but if you are removing an old field
that was once required, you'll want to remove those constraints at
this point. If you don't do this, then once a version of the app is
deployed which removes references to the old version, you will likely
hit database constraint failures or the like.

Version 4: Read/Write From The New Representation Only, Remove
References To Old Representation

At this point, the previous version was only writing to the old
representation for backwards-compatibility reasons. So you can stop
writing to the old representation, and have all read/write paths just
hit the new one.

At this point you also want to remove references to the old
representation (in particular stuff like model fields), in
preparation for the final migration.

Migration 4: Drop The Old Representation Entirely

Once you have version 4 out, queries should no longer be referencing
the old representation at all. You should be good to go for just
dropping this stuff entirely!

This one you gotta be sure though, really hard to roll back this
change.

Once you've done that you're good to close out that work!

The Checklist

  * Deploy Version 1 (Accept New Representation)
  * Add New Representation To The Database
  * Deploy Version 2 (Read Old/Write Old + New)
  * Backfill The New Representation In Existing Data
  * Deploy Version 3 (Read New/Write Old + New)
  * Remove Any Mandatory Constraints For The Old Representation
  * Deploy Version 4 (Read/Write New, Remove References To Old)
  * Drop The Old Representation Entirely

In your specific case it could be that some of these can be merged.
I've found that these steps are general enough to cover the most
scenarios safely. Though really, the best thing is to really
understand why so many steps are needed and whether the
characteristics of your system impose different constraints.

If you have had a problem with this style of rollout in practice or
otherwise think this game plan is missing some crucial steps, please
send me an e-mail with those details! I'm very curious about covering
all bases.

Previously: Livereload, the simple tool that can carry you very far

  * rtpg
  * rtpg_